This tutorial and the assets can be downloaded as part of the Wallaroo Tutorials repository.

In-Line Model Updates at the Edge Tutorial

Wallaroo pipeline publishes are containerized versions of the pipeline, models and model steps, and the inference engine published to an Open Container Initiative (OCI) Registry. Once published, they are used to deploy models to edge devices and used for inference requests.

Edge locations added to a pipeline published allow deployments on edge devices to connect with the Wallaroo Ops instance and transmit their inference results as part of the pipeline logs.

Pipeline publishes can be replaced by new pipeline publishes from the following sources:

• A new version of the same pipeline that creates a new pipeline publish. In these instances, the edge locations associated with the pipeline publish are preserved with the original pipeline along with their pipeline inference logs. Any input or output schema changes from the new models or models steps are reflected in the pipeline logs.
• A separate pipeline with its own models and model steps. In this scenario, the edge locations associated with the original pipeline publish are assigned to the new pipeline. The inference logs are stored with the original pipeline, and new inference logs for the edge locations are stored as part of the new pipeline.

For more information on pipeline inference logs, see Inference Logs.

• IMPORTANT NOTE: Assays and other log data is based on the pipeline and model name. If these values change, assays that are currently running will no longer generate analyses. These should be paused, and new assays based on the current pipeline and model name should be created. For more information, see Model Drift Detection with Model Insights.

Goal

• Package a model in Wallaroo and perform sample inferences. Publish the pipeline and create an edge location, then deploy the model to an edge location.
• Perform sample inferences on the edge location and verify the inference logs are added to the pipeline associated with the edge location.
• Create a new pipeline version with a new model step and perform sample inferences. Update the publish with the new pipeline version, and verify that the model is deployed to the edge location deployment.
• Perform sample inferences on the edge location to verify the new model is being used, and verify the inference logs are added to the pipeline associated with the edge location.
• Create a new pipeline with a different model and perform sample inferences. Publish the pipeline, then replace the publish used for the edge location. Verify that the edge location is transferred to the new pipeline.
• Perform sample inferences on the edge location to verify model is updated from the new pipeline is being used, and verify the inference logs are added to the new pipeline associated with the edge location.

Resources

This tutorial provides the following:

• Models:
  • models/rf_model.onnx: The champion model that predicts house prices.
  • models/xgb_model.onnx: A new challenger model that takes the same inputs as models/rf_model.onnx and outputs house prices with a field of the same name.
  • models/gbr_model.onnx: A new challenger model that takes the same inputs as models/rf_model.onnx and outputs house prices with a field of the same name.

Prerequisites

• A deployed Wallaroo instance with Edge Registry Services and Edge Observability enabled.
• The following Python libraries installed:
  • wallaroo: The Wallaroo SDK. Included with the Wallaroo JupyterHub service by default.
  • pandas: Pandas, mainly used for Pandas DataFrame
• A X64 Docker deployment to deploy the model on an edge location.

Edge Deployment Preparation Steps

The following is used to set up the environment that is used for the pipeline publish replacement and in-line model updates on an edge tutorial.

In these steps, we will:

• Deploying a sample ML model used to determine house prices based on a set of input parameters.
• Publish the model deployment configuration to an OCI registry.
• Use the publish and set edge locations.
• Deploy the model in an edge location and perform sample inferences.

Import Libraries

The first step will be to import our libraries, and set variables used through this tutorial.

import wallaroo
from wallaroo.object import EntityNotFoundError
from wallaroo.framework import Framework

from IPython.display import display

# used to display DataFrame information without truncating
from IPython.display import display
import pandas as pd
pd.set_option('display.max_colwidth', None)

import json
import time

workspace_name = 'in-line-edge-replacement-demo'

# ignoring warnings for demonstration
import warnings
warnings.filterwarnings('ignore')

# used to display DataFrame information without truncating
from IPython.display import display
import pandas as pd
pd.set_option('display.max_colwidth', None)

Connect to the Wallaroo Instance

The first step is to connect to Wallaroo through the Wallaroo client. The Python library is included in the Wallaroo install and available through the Jupyter Hub interface provided with your Wallaroo environment.

This is accomplished using the wallaroo.Client() command, which provides a URL to grant the SDK permission to your specific Wallaroo environment. When displayed, enter the URL into a browser and confirm permissions. Store the connection into a variable that can be referenced later.

If logging into the Wallaroo instance through the internal JupyterHub service, use wl = wallaroo.Client(). For more information on Wallaroo Client settings, see the Client Connection guide.

# Login through local Wallaroo instance

wl = wallaroo.Client()

Create Workspace

We will create a workspace to manage our pipeline and models. The following variables will set the name of our sample workspace then set it as the current workspace.

Workspace names are unique across the Wallaroo instance. If a user requests a workspace they have not been granted access to, an error message is returned.

workspace = wl.get_workspace(name=workspace_name, create_if_not_exist=True)

wl.set_current_workspace(workspace)

{'name': 'in-line-edge-replacement-demo', 'id': 7, 'archived': False, 'created_by': '738d05e5-425c-4a3e-8870-1beb652db894', 'created_at': '2024-03-27T16:40:02.4306+00:00', 'models': [{'name': 'rf-house-price-estimator', 'versions': 4, 'owner_id': '""', 'last_update_time': datetime.datetime(2024, 3, 27, 17, 15, 50, 488392, tzinfo=tzutc()), 'created_at': datetime.datetime(2024, 3, 27, 16, 40, 6, 63908, tzinfo=tzutc())}, {'name': 'xgb-house-price-estimator', 'versions': 4, 'owner_id': '""', 'last_update_time': datetime.datetime(2024, 3, 27, 17, 18, 14, 596059, tzinfo=tzutc()), 'created_at': datetime.datetime(2024, 3, 27, 16, 51, 7, 706652, tzinfo=tzutc())}, {'name': 'gbr-house-price-estimator', 'versions': 4, 'owner_id': '""', 'last_update_time': datetime.datetime(2024, 3, 27, 17, 18, 26, 994434, tzinfo=tzutc()), 'created_at': datetime.datetime(2024, 3, 27, 16, 51, 41, 86395, tzinfo=tzutc())}], 'pipelines': [{'name': 'edge-inline-replacement-demo', 'create_time': datetime.datetime(2024, 3, 27, 16, 40, 8, 60605, tzinfo=tzutc()), 'definition': '[]'}, {'name': 'edge-inline-replacement-demonstration', 'create_time': datetime.datetime(2024, 3, 27, 16, 47, 37, 180104, tzinfo=tzutc()), 'definition': '[]'}, {'name': 'new-edge-inline-replacement-demo', 'create_time': datetime.datetime(2024, 3, 27, 16, 51, 41, 632386, tzinfo=tzutc()), 'definition': '[]'}]}

Upload The Champion Model

For our example, we will upload the champion model that has been trained to derive house prices from a variety of inputs. The model file is rf_model.onnx, and is uploaded with the name rf-house-price-estimator.

housing_model_control = (wl.upload_model("rf-house-price-estimator", 
                                        './models/rf_model.onnx', 
                                        framework=Framework.ONNX)
                                        .configure(tensor_fields=["tensor"])
                        )

Build and Deploy the Pipeline

This pipeline is made to be an example of an existing situation where a model is deployed and being used for inferences in a production environment. We’ll call it edge-inline-replacement-demonstration, set housing_model_control as a pipeline step, then run a sample inference.

mainpipeline = wl.build_pipeline('edge-inline-replacement-demon')

# undeploy if already created earlier
mainpipeline.undeploy()
# clear the steps if used before
mainpipeline.clear()

mainpipeline.add_model_step(housing_model_control)

#minimum deployment config
deploy_config = wallaroo.DeploymentConfigBuilder().replica_count(1).cpus(0.5).memory("1Gi").build()

mainpipeline.deploy(deployment_config = deploy_config)

Inference Request Testing

We’ll use a single row inference request that should return a result of around $700k.

normal_input = pd.DataFrame.from_records({"tensor": [[4.0, 
                                                      2.5, 
                                                      2900.0, 
                                                      5505.0, 
                                                      2.0, 
                                                      0.0, 
                                                      0.0, 
                                                      3.0, 
                                                      8.0, 
                                                      2900.0, 
                                                      0.0, 
                                                      47.6063, 
                                                      -122.02, 
                                                      2970.0, 
                                                      5251.0, 
                                                      12.0, 
                                                      0.0, 
                                                      0.0]]})
result = mainpipeline.infer(normal_input)
display(result)

Edge Deployment

We can now deploy the original pipeline to an edge device. This will require the following steps:

• Publish the pipeline: Publishes the pipeline to the OCI registry.
• Add Edge: Add the edge location to the pipeline publish.
• Deploy Edge: Deploy the edge device with the edge location settings.

Publish Pipeline

Publishing the pipeline uses the pipeline wallaroo.pipeline.publish() command. This requires that the Wallaroo Ops instance have Edge Registry Services enabled.

The following publishes the pipeline to the OCI registry and displays the container details. For more information, see Wallaroo SDK Essentials Guide: Pipeline Edge Publication.

pub = mainpipeline.publish()

Waiting for pipeline publish... It may take up to 600 sec.
Pipeline is publishing..... Published.

Add Edge Location

The edge location is added with the wallaroo.pipeline_publish.add_edge(name) method. This returns the OCI registration information, and the EDGE_BUNDLE information. The EDGE_BUNDLE data is a base64 encoded set of parameters for the pipeline that the edge device is associated with, the workspace, and other data.

For full details, see Wallaroo SDK Essentials Guide: Pipeline Edge Publication: Edge Observability.

For this example, we will add two locations:

• houseprice-edge-inline-demo

These will be used in later steps for demonstrating inferences through different locations.

edge_name_01 = "houseprice-edge-inline-demonstration"
edge_publish_01 = pub.add_edge(edge_name_01)
display(edge_publish_01)

docker run -v $PERSISTENT_VOLUME_DIR:/persist \
    -e OCI_USERNAME=$OCI_USERNAME \
    -e OCI_PASSWORD=$OCI_PASSWORD \
    -e EDGE_BUNDLE=ZXhwb3J0IEJVTkRMRV9WRVJTSU9OPTEKZXhwb3J0IENPTkZJR19DUFVTPTQKZXhwb3J0IEVER0VfTkFNRT1ob3VzZXByaWNlLWVkZ2UtaW5saW5lLWRlbW9uc3RyYXRpb24KZXhwb3J0IE9QU0NFTlRFUl9IT1NUPWRvYy10ZXN0LmVkZ2Uud2FsbGFyb29jb21tdW5pdHkubmluamEKZXhwb3J0IFBJUEVMSU5FX1VSTD1naGNyLmlvL3dhbGxhcm9vbGFicy9kb2Mtc2FtcGxlcy9waXBlbGluZXMvZWRnZS1pbmxpbmUtcmVwbGFjZW1lbnQtZGVtb246ZWRiMGMzMjktNWI1Zi00YzRjLTgwMWEtMDdiNjIwMzIyYTdiCmV4cG9ydCBKT0lOX1RPS0VOPWIwOTM2MTE3LWY2YjktNDdmMy04ODI5LTYzMWM3ZmUwYmVkMgpleHBvcnQgT0NJX1JFR0lTVFJZPWdoY3IuaW8=\
    -e PIPELINE_URL=ghcr.io/wallaroolabs/doc-samples/pipelines/edge-inline-replacement-demon:edb0c329-5b5f-4c4c-801a-07b620322a7b \
    -e CONFIG_CPUS=4 ghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/fitzroy-mini:v2024.1.0-main-4806

helm install --atomic $HELM_INSTALL_NAME \
    oci://ghcr.io/wallaroolabs/doc-samples/charts/edge-inline-replacement-demon \
    --namespace $HELM_INSTALL_NAMESPACE \
    --version 0.0.1-edb0c329-5b5f-4c4c-801a-07b620322a7b \
    --set ociRegistry.username=$OCI_USERNAME \
    --set ociRegistry.password=$OCI_PASSWORD \
    --set edgeBundle=ZXhwb3J0IEJVTkRMRV9WRVJTSU9OPTEKZXhwb3J0IENPTkZJR19DUFVTPTQKZXhwb3J0IEVER0VfTkFNRT1ob3VzZXByaWNlLWVkZ2UtaW5saW5lLWRlbW9uc3RyYXRpb24KZXhwb3J0IE9QU0NFTlRFUl9IT1NUPWRvYy10ZXN0LmVkZ2Uud2FsbGFyb29jb21tdW5pdHkubmluamEKZXhwb3J0IFBJUEVMSU5FX1VSTD1naGNyLmlvL3dhbGxhcm9vbGFicy9kb2Mtc2FtcGxlcy9waXBlbGluZXMvZWRnZS1pbmxpbmUtcmVwbGFjZW1lbnQtZGVtb246ZWRiMGMzMjktNWI1Zi00YzRjLTgwMWEtMDdiNjIwMzIyYTdiCmV4cG9ydCBKT0lOX1RPS0VOPWIwOTM2MTE3LWY2YjktNDdmMy04ODI5LTYzMWM3ZmUwYmVkMgpleHBvcnQgT0NJX1JFR0lTVFJZPWdoY3IuaW8=

DevOps Deployment

The following shows two methods of model deployments on edge devices. For more details on model edge deployments with Wallaroo, see Model Operations: Run Anywhere.

Docker Run Based Deployment

For the model deployment to the edge location via docker run, use the Docker Run Command displayed with the pipeline publish. The following variables must be set first.

• $REGISTRYURL: The URL of the OCI registry service hosting the pipeline publish.
• $OCI_USERNAME: The username for the OCI registry service.
• $OCI_PASSWORD: The password or token for the OCI registry service.
• $HELM_INSTALL_NAME: The name of the Helm based installation for the edge deployment.
• $HELM_INSTALL_NAMESPACE: The name of the namespace to set the edge based deployment to.

Before deploying the pipeline, verify that Docker is able to connect to the OCI registry with the following command.

docker login -u $OCI_USERNAME --password $OCI_PASSWORD $REGISTRYURL

To add a specific port mapping, add the -p {External Port}:8080 option to map the edge devices {External Port} to the Wallaroo edge deployment port 8080. For example, to map the external port 8080 to the internal port 8080:

docker run -v $PERSISTENT_VOLUME_DIR:/persist \
    -p 8080:8080 \
    -e OCI_USERNAME=$OCI_USERNAME \
    -e OCI_PASSWORD=$OCI_PASSWORD \
    -e EDGE_BUNDLE=abcdefg\
    -e PIPELINE_URL=ghcr.io/wallaroolabs/doc-samples/pipelines/edge-inline-replacement-demonstration:96a926bf-bd64-4e88-ae27-9d5773fdc007 \
    -e CONFIG_CPUS=4 ghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/fitzroy-mini:v2024.1.0-main-4806

Helm Based Deployment

For the model deployment to the edge location via helm, use the Helm Install Command displayed with the pipeline publish. The following variables must be set first.

• $REGISTRYURL: The URL of the OCI registry service hosting the pipeline publish.
• $OCI_USERNAME: The username for the OCI registry service.
• $OCI_PASSWORD: The password or token for the OCI registry service.
• $HELM_INSTALL_NAME: The name of the Helm based installation for the edge deployment.
• $HELM_INSTALL_NAMESPACE: The name of the namespace to set the edge based deployment to.

Before deploying, verify helm has access to the target registry. Using the variable above, this command is:

helm registry login $REGISTRYURL --username $OCI_USERNAME --password $OCI_PASSWORD

If the command succeeds, proceed. If there are any issues, verify the settings above before continuing.

Once the helm registry service is set, deploy using the Helm Install Command from edge_publish_01.

Once deployed, use the following to port forward to the helm based deployment.

kubectl port-forward svc/engine-svc -n $HELM_INSTALL_NAMESPACE 8080 --address 0.0.0.0

Edge Inference Example

We will perform sample inferences on the current model edge deployment to demonstrate how the inferences change as the models are updated through the in-line edge deployments.

With the model deployed on the edge location, we will perform sample inferences on the edge deployment. For this example, the hostname is HOSTNAME. Adjust the hostname to fit your edge location deployment.

The following endpoints are available.

For more details on model edge deployments with Wallaroo, see Model Operations: Run Anywhere.

First we’ll retrieve the model. We should see the same name we set when we uploaded it to the Wallaroo instance.

!curl HOSTNAME:8080/models

{"models":[{"model_version":{"name":"rf-house-price-estimator","visibility":"private","workspace_id":7,"conversion":{"python_version":"3.8","requirements":[],"framework":"onnx"},"id":16,"image_path":null,"status":"ready","task_id":null,"file_info":{"version":"5a610669-9287-4dd8-8273-0cd21cccb39a","sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6","file_name":"rf_model.onnx"},"created_on_version":"2024.1.0"},"config":{"id":32,"model_version_id":16,"runtime":"onnx","filter_threshold":null,"tensor_fields":["tensor"],"input_schema":null,"output_schema":null,"batch_config":null,"sidekick_uri":null},"status":"Running"}]}

Now we’ll retrieve the pipeline details, and verify the name is the same as set earlier and it is running.

!curl HOSTNAME:8080/pipelines

{"pipelines":[{"id":"edge-inline-replacement-demon","status":"Running"}]}

Now do the infer with the same DataFrame we used for our sample inference when the model was deployed in the Wallaroo instance.

import json

df = pd.DataFrame.from_records({"tensor": [[4.0, 
                                            2.5, 
                                            2900.0, 
                                            5505.0, 
                                            2.0, 
                                            0.0, 
                                            0.0, 
                                            3.0, 
                                            8.0, 
                                            2900.0, 
                                            0.0, 
                                            47.6063, 
                                            -122.02, 
                                            2970.0, 
                                            5251.0, 
                                            12.0, 
                                            0.0, 
                                            0.0]]})

data = df.to_dict(orient="records")

!curl -X POST HOSTNAME:8080/infer \
    -H "Content-Type: Content-Type: application/json; format=pandas-records" \
    --data '{json.dumps(data)}' > ./inferenceoutput.df.json

# get the dataframe from what we just did

df_result = pd.read_json('./inferenceoutput.df.json', orient="records")
df_result.loc[:, ['time', 'out', 'metadata']]

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   623  100   495  100   128   7061   1825 --:--:-- --:--:-- --:--:--  8900

How to Replace a Pipeline Publish

Pipeline publishes are replaced with the wallaroo.pipeline.Pipeline.publish() method. This creates a new publish from the most recent pipeline version and takes the following parameters.

When a pipeline published is replaced with a new one, the edge locations are transferred to the pipeline that the publish came from. In this example, this is the same pipeline. Inference results from the edge location deployments are stored with the pipeline that generated the publish. Note that if the model or pipeline steps have changed from one pipeline version to the next, the pipeline log schema will change with it. For more information, see Wallaroo SDK Essentials Guide: Pipeline Log Management.

Verify Pipeline Publishes and Edge Locations

We verify that the pipeline is published and is assigned edge locations via the wallaroo.pipeline.Pipeline.publishes() method and wallaroo.pipeline.Pipeline.list_edges() method.

display('Publishes:')
display(mainpipeline.publishes())
display('Edge Locations')
display(mainpipeline.list_edges())

'Publishes:'

'Edge Locations'

How to Replace a Pipeline Publish from a New Pipeline Version

The following demonstrates replacing a pipeline publish with a new publish from a pipeline version. In this procedure, the edge location has an existing model edge deployment. The publish that is used for that deployment is replaced, and an in-line update of the model deployment is performed on the edge device.

Create New Pipeline Version

For the next demonstration, we will use the same pipeline and update the pipeline steps with a new model. This time with the model ./models/xgb_model.onnx, which takes the same input as ./models/rf_model.onnx. We will also set a different name for this model to distinguish it from the one previously uploaded and used as the pipeline step.

Once set, we will deploy the pipeline with the new model, and perform a new inference request. From this, we will see that this new model outputs a slightly different prediction that the previous one - closer to $650k.

housing_model_challenger01 = (wl.upload_model("xgb-house-price-estimator", 
                                        './models/xgb_model.onnx', 
                                        framework=Framework.ONNX)
                                        .configure(tensor_fields=["tensor"])
                        )

# undeploy if already created earlier
mainpipeline.undeploy()
# clear the steps if used before
mainpipeline.clear()

mainpipeline.add_model_step(housing_model_challenger01)

#minimum deployment config
deploy_config = wallaroo.DeploymentConfigBuilder().replica_count(1).cpus(0.5).memory("1Gi").build()

mainpipeline.deploy(deployment_config = deploy_config)
mainpipeline.steps()

[{'ModelInference': {'models': [{'name': 'xgb-house-price-estimator', 'version': '5363f0eb-852e-4883-a2c8-6195b36f8829', 'sha': '31e92d6ccb27b041a324a7ac22cf95d9d6cc3aa7e8263a229f7c4aec4938657c'}]}}]

normal_input = pd.DataFrame.from_records({"tensor": [[4.0, 
                                                      2.5, 
                                                      2900.0, 
                                                      5505.0, 
                                                      2.0, 
                                                      0.0, 
                                                      0.0, 
                                                      3.0, 
                                                      8.0, 
                                                      2900.0, 
                                                      0.0, 
                                                      47.6063, 
                                                      -122.02, 
                                                      2970.0, 
                                                      5251.0, 
                                                      12.0, 
                                                      0.0, 
                                                      0.0]]})
result = mainpipeline.infer(normal_input)
display(result)

Replace Pipeline Publish with new Pipeline Version

From our pipeline, we will publish the new version. The method wallaroo.pipeline.Pipeline.publish() creates a new publish from the most recent pipeline version and takes the following parameters.

We will replace our publish earlier, labeled pub, with the new publish generated from the pipeline labeled new_pub.

new_pub = mainpipeline.publish(replaces=[pub])
new_pub

Waiting for pipeline publish... It may take up to 600 sec.
Pipeline is publishing...... Published.

docker run -v $PERSISTENT_VOLUME_DIR:/persist \
    -e OCI_USERNAME=$OCI_USERNAME \
    -e OCI_PASSWORD=$OCI_PASSWORD \
    -e PIPELINE_URL=ghcr.io/wallaroolabs/doc-samples/pipelines/edge-inline-replacement-demon:2e78349c-d282-41c9-a435-0c943c3accf1\
    -e EDGE_BUNDLE=ZXhwb3J0IEJVTkRMRV9WRVJTSU9OPTEKZXhwb3J0IENPTkZJR19DUFVTPTQKZXhwb3J0IEVER0VfTkFNRT1ob3VzZXByaWNlLWVkZ2UtaW5saW5lLWRlbW9uc3RyYXRpb24KZXhwb3J0IE9QU0NFTlRFUl9IT1NUPWRvYy10ZXN0LmVkZ2Uud2FsbGFyb29jb21tdW5pdHkubmluamEKZXhwb3J0IFBJUEVMSU5FX1VSTD1naGNyLmlvL3dhbGxhcm9vbGFicy9kb2Mtc2FtcGxlcy9waXBlbGluZXMvZWRnZS1pbmxpbmUtcmVwbGFjZW1lbnQtZGVtb246MmU3ODM0OWMtZDI4Mi00MWM5LWE0MzUtMGM5NDNjM2FjY2YxCmV4cG9ydCBKT0lOX1RPS0VOPW9taXR0ZWRfZm9yX3JlcGxhY2UKZXhwb3J0IE9DSV9SRUdJU1RSWT1naGNyLmlv \
    ghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/fitzroy-mini:v2024.1.0-main-4806

helm upgrade $HELM_INSTALL_NAME \
    oci://ghcr.io/wallaroolabs/doc-samples/charts/edge-inline-replacement-demon \
    --namespace $HELM_INSTALL_NAMESPACE \
    --version 0.0.1-2e78349c-d282-41c9-a435-0c943c3accf1 \
    --set ociRegistry.username=$OCI_USERNAME \
    --set ociRegistry.password=$OCI_PASSWORD \
    --set edgeBundle=ZXhwb3J0IEJVTkRMRV9WRVJTSU9OPTEKZXhwb3J0IENPTkZJR19DUFVTPTQKZXhwb3J0IEVER0VfTkFNRT1ob3VzZXByaWNlLWVkZ2UtaW5saW5lLWRlbW9uc3RyYXRpb24KZXhwb3J0IE9QU0NFTlRFUl9IT1NUPWRvYy10ZXN0LmVkZ2Uud2FsbGFyb29jb21tdW5pdHkubmluamEKZXhwb3J0IFBJUEVMSU5FX1VSTD1naGNyLmlvL3dhbGxhcm9vbGFicy9kb2Mtc2FtcGxlcy9waXBlbGluZXMvZWRnZS1pbmxpbmUtcmVwbGFjZW1lbnQtZGVtb246MmU3ODM0OWMtZDI4Mi00MWM5LWE0MzUtMGM5NDNjM2FjY2YxCmV4cG9ydCBKT0lOX1RPS0VOPW9taXR0ZWRfZm9yX3JlcGxhY2UKZXhwb3J0IE9DSV9SRUdJU1RSWT1naGNyLmlv

Note that in the Replaces section, updates are given for each edge location. On the edge device, we deploy the new pipeline publish either by the provided Docker Run Command or Helm Install Command. Note that the Helm Install Command uses helm upgrade rather than helm install to create a new revision. The pipeline steps and models will be changed at the model edge location.

Before we do so, we will verify the edge locations associated for our pipelines. mainpipeline will still have its edges, since we only updated the pipeline version, while new_pipeline has no edges yet.

Inference Requests from Replaced Pipeline Version Deployment Examples

We deploy the in-line model update in our new publish above for the edge location, then perform the same checks of the models, pipeline, then a sample inference using the same inference input data as before.

We will see the new model name and model version used.

!curl HOSTNAME:8080/models

{"models":[{"model_version":{"name":"xgb-house-price-estimator","visibility":"private","workspace_id":7,"conversion":{"python_version":"3.8","requirements":[],"framework":"onnx"},"id":17,"image_path":null,"status":"ready","task_id":null,"file_info":{"version":"5363f0eb-852e-4883-a2c8-6195b36f8829","sha":"31e92d6ccb27b041a324a7ac22cf95d9d6cc3aa7e8263a229f7c4aec4938657c","file_name":"xgb_model.onnx"},"created_on_version":"2024.1.0"},"config":{"id":34,"model_version_id":17,"runtime":"onnx","filter_threshold":null,"tensor_fields":["tensor"],"input_schema":null,"output_schema":null,"batch_config":null,"sidekick_uri":null},"status":"Running"}]}

!curl HOSTNAME:8080/pipelines

{"pipelines":[{"id":"edge-inline-replacement-demon","status":"Running"}]}

We now perform the same inference request on the edge location deployment, and show the results match that as the pipeline with the new model.

import json

df = pd.DataFrame.from_records({"tensor": [[4.0, 
                                            2.5, 
                                            2900.0, 
                                            5505.0, 
                                            2.0, 
                                            0.0, 
                                            0.0, 
                                            3.0, 
                                            8.0, 
                                            2900.0, 
                                            0.0, 
                                            47.6063, 
                                            -122.02, 
                                            2970.0, 
                                            5251.0, 
                                            12.0, 
                                            0.0, 
                                            0.0]]})

data = df.to_dict(orient="records")

!curl -X POST HOSTNAME:8080/infer \
    -H "Content-Type: Content-Type: application/json; format=pandas-records" \
    --data '{json.dumps(data)}' > inferenceoutput.df.json

# get the dataframe from what we just did

df_result = pd.read_json('./inferenceoutput.df.json', orient="records")
df_result.loc[:, ['time', 'out', 'metadata']]

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   624  100   496  100   128   7639   1971 --:--:-- --:--:-- --:--:--  9750

How to Replace a Pipeline Publish from a New Pipeline

For this example, we will create a separate pipeline than the previous one with its own model. This model will still accept the same inputs and output fields as the models previously deployed. This is not a requirement - completely different models with different inputs and output schemas can be used; this example is used for simplicity.

For this set of examples, we will upload the third model, ./models/gbr_model.onnx and create a new pipeline named new-edge-inline-replacement. This model is added as a pipeline step and deployed.

housing_model_challenger02 = (wl.upload_model("gbr-house-price-estimator", 
                                        './models/gbr_model.onnx', 
                                        framework=Framework.ONNX)
                                        .configure(tensor_fields=["tensor"])
                        )

new_pipeline = wl.build_pipeline("new-edge-inline-replacement")

# undeploy if used before
new_pipeline.undeploy()
# clear the steps if used before
new_pipeline.clear()

new_pipeline.add_model_step(housing_model_challenger02)

#minimum deployment config
deploy_config = wallaroo.DeploymentConfigBuilder().replica_count(1).cpus(0.5).memory("1Gi").build()

new_pipeline.deploy(deployment_config = deploy_config)
new_pipeline.steps()

[{'ModelInference': {'models': [{'name': 'gbr-house-price-estimator', 'version': '30654768-3797-4076-9529-850979b983c9', 'sha': 'ed6065a79d841f7e96307bb20d5ef22840f15da0b587efb51425c7ad60589d6a'}]}}]

normal_input = pd.DataFrame.from_records({"tensor": [[4.0, 
                                                      2.5, 
                                                      2900.0, 
                                                      5505.0, 
                                                      2.0, 
                                                      0.0, 
                                                      0.0, 
                                                      3.0, 
                                                      8.0, 
                                                      2900.0, 
                                                      0.0, 
                                                      47.6063, 
                                                      -122.02, 
                                                      2970.0, 
                                                      5251.0, 
                                                      12.0, 
                                                      0.0, 
                                                      0.0]]})
result = new_pipeline.infer(normal_input)
display(result)

Replace Pipeline Publish with Publish from Different Pipepline

We will now publish the new pipeline, and replace the previous edge location publish from our original pipeline. The last publish is labeled new_pub is labeled new_pipeline_pub.

new_pipeline_pub = new_pipeline.publish(replaces=[new_pub])
new_pipeline_pub

Waiting for pipeline publish... It may take up to 600 sec.
Pipeline is publishing...... Published.

docker run -v $PERSISTENT_VOLUME_DIR:/persist \
    -e OCI_USERNAME=$OCI_USERNAME \
    -e OCI_PASSWORD=$OCI_PASSWORD \
    -e PIPELINE_URL=ghcr.io/wallaroolabs/doc-samples/pipelines/new-edge-inline-replacement:a7aceb15-68a8-4d8f-b7fa-0b70a46dfa62\
    -e EDGE_BUNDLE=ZXhwb3J0IEJVTkRMRV9WRVJTSU9OPTEKZXhwb3J0IENPTkZJR19DUFVTPTQKZXhwb3J0IEVER0VfTkFNRT1ob3VzZXByaWNlLWVkZ2UtaW5saW5lLWRlbW9uc3RyYXRpb24KZXhwb3J0IE9QU0NFTlRFUl9IT1NUPWRvYy10ZXN0LmVkZ2Uud2FsbGFyb29jb21tdW5pdHkubmluamEKZXhwb3J0IFBJUEVMSU5FX1VSTD1naGNyLmlvL3dhbGxhcm9vbGFicy9kb2Mtc2FtcGxlcy9waXBlbGluZXMvbmV3LWVkZ2UtaW5saW5lLXJlcGxhY2VtZW50OmE3YWNlYjE1LTY4YTgtNGQ4Zi1iN2ZhLTBiNzBhNDZkZmE2MgpleHBvcnQgSk9JTl9UT0tFTj1vbWl0dGVkX2Zvcl9yZXBsYWNlCmV4cG9ydCBPQ0lfUkVHSVNUUlk9Z2hjci5pbw== \
    ghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/fitzroy-mini:v2024.1.0-main-4806

helm upgrade $HELM_INSTALL_NAME \
    oci://ghcr.io/wallaroolabs/doc-samples/charts/new-edge-inline-replacement \
    --namespace $HELM_INSTALL_NAMESPACE \
    --version 0.0.1-a7aceb15-68a8-4d8f-b7fa-0b70a46dfa62 \
    --set ociRegistry.username=$OCI_USERNAME \
    --set ociRegistry.password=$OCI_PASSWORD \
    --set edgeBundle=ZXhwb3J0IEJVTkRMRV9WRVJTSU9OPTEKZXhwb3J0IENPTkZJR19DUFVTPTQKZXhwb3J0IEVER0VfTkFNRT1ob3VzZXByaWNlLWVkZ2UtaW5saW5lLWRlbW9uc3RyYXRpb24KZXhwb3J0IE9QU0NFTlRFUl9IT1NUPWRvYy10ZXN0LmVkZ2Uud2FsbGFyb29jb21tdW5pdHkubmluamEKZXhwb3J0IFBJUEVMSU5FX1VSTD1naGNyLmlvL3dhbGxhcm9vbGFicy9kb2Mtc2FtcGxlcy9waXBlbGluZXMvbmV3LWVkZ2UtaW5saW5lLXJlcGxhY2VtZW50OmE3YWNlYjE1LTY4YTgtNGQ4Zi1iN2ZhLTBiNzBhNDZkZmE2MgpleHBvcnQgSk9JTl9UT0tFTj1vbWl0dGVkX2Zvcl9yZXBsYWNlCmV4cG9ydCBPQ0lfUkVHSVNUUlk9Z2hjci5pbw==

Once complete, we update the edge deployment with the new replacement docker run or helm based deployment commands from the configuration.

With the edge location replacement complete with the new pipeline, we list the edge locations from the original pipeline and the new one to show the edge location is transferred to the new pipeline.

display(mainpipeline.list_edges())

(no pipelines)

display(new_pipeline.list_edges())

Sample Inference Requests with New Pipeline Publish on Edge Location

We deploy the new publish on the edge device either with the Docker Run Command or the Helm Install Command, with whatever modifications are required on the edge device. Once complete, we run through the /models and /pipelines endpoints, then perform an inference request. We will see the name of the new model - and the new pipeline used for the same edge location.

!curl HOSTNAME:8080/models

{"models":[{"model_version":{"name":"gbr-house-price-estimator","visibility":"private","workspace_id":7,"conversion":{"python_version":"3.8","requirements":[],"framework":"onnx"},"id":18,"image_path":null,"status":"ready","task_id":null,"file_info":{"version":"30654768-3797-4076-9529-850979b983c9","sha":"ed6065a79d841f7e96307bb20d5ef22840f15da0b587efb51425c7ad60589d6a","file_name":"gbr_model.onnx"},"created_on_version":"2024.1.0"},"config":{"id":36,"model_version_id":18,"runtime":"onnx","filter_threshold":null,"tensor_fields":["tensor"],"input_schema":null,"output_schema":null,"batch_config":null,"sidekick_uri":null},"status":"Running"}]}

!curl HOSTNAME:8080/pipelines

{"pipelines":[{"id":"new-edge-inline-replacement","status":"Running"}]}

# new inference

import json

df = pd.DataFrame.from_records({"tensor": [[4.0, 2.5, 2900.0, 5505.0, 2.0, 0.0, 0.0, 3.0, 8.0, 2900.0, 0.0, 47.6063, -122.02, 2970.0, 5251.0, 12.0, 0.0, 0.0]]})

data = df.to_dict(orient="records")

!curl -X POST HOSTNAME:8080/infer \
    -H "Content-Type: Content-Type: application/json; format=pandas-records" \
    --data '{json.dumps(data)}' > inferenceoutput.df.json

# get the dataframe from what we just did

df_result = pd.read_json('./inferenceoutput.df.json', orient="records")
df_result.loc[:, ['time', 'out', 'metadata']]

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   624  100   496  100   128   7407   1911 --:--:-- --:--:-- --:--:--  9454

With the tutorial complete, we undeploy the pipelines and return the resources back to the cluster.

mainpipeline.undeploy()
new_pipeline.undeploy()