In-Line Model Updates for Edge/Multicloud Model Deployments

How to update deployed models on on Edge And Multicloud environments.

Wallaroo pipeline publishes are containerized versions of the pipeline, models and model steps, and the inference engine published to an Open Container Initiative (OCI) Registry. Once published, they are used to deploy models to edge locations and used for inference requests.

Edge locations added to a pipeline published allow deployments on edge devices to connect with the Wallaroo Ops instance and transmit their inference results as part of the pipeline logs.

Pipeline publishes can be replaced by new pipeline publishes from the following sources:

  • A new version of the same pipeline that creates a new pipeline publish. In these instances, the edge locations associated with the pipeline publish are preserved with the original pipeline along with their pipeline inference logs. Any input or output schema changes from the new models or models steps are reflected in the pipeline logs.
  • A separate pipeline with its own models and model steps. In this scenario, the edge locations associated with the original pipeline publish are assigned to the new pipeline. The inference logs are stored with the original pipeline, and new inference logs for the edge locations are stored as part of the new pipeline.

For more information on pipeline inference logs, see Inference Logs.

Model Update Considerations

When updating an pipeline with new model or an edge location with a new pipeline publish from either a new pipeline version or a new pipeline, the following considerations should be taken.

  • Inference Log Updates: If the new model(s) have different inputs and outputs schemas, the pipeline inference logs will change to reflect the new data schemas.
  • Model Drift Detection: Assays are based on the pipeline and the model name. If either of these change as part of the either new pipeline versions, or a new pipeline, then the previous assays will cease to generate new analyses. It is recommended to “pause” assays where the pipeline or model name has chanced, the create new ones. For more information on creating assays, see Model Drift Detection with Model Insights.
  • Inference Request Updates: If the input or output schema for the pipeline changes including field names, field data types, etc, users should verify that the inference requests are modified to match.

How to Replace a Pipeline Publish

Pipeline publishes are replaced with the wallaroo.pipeline.Pipeline.publish(deployment_config, replaces) method. This creates a new publish from the most recent pipeline version and takes the following parameters.

ParameterTypeDescription
deployment_configwallaroo.deployment_config.DeploymentConfig (Optional)The deployment configuration used for the edge deployment. By default, this is the same deployment configuration used for the pipeline.
replaces[List[wallaroo.pipeline_publish]] (Optional)The pipeline publish(es) to replace.

When a pipeline published is replaced with a new one, the edge locations are transferred to the pipeline that the publish came from. In this example, this is the same pipeline. Inference results from the edge location deployments are stored with the pipeline that generated the publish. Note that if the model or pipeline steps have changed from one pipeline version to the next, the pipeline log schema will change with it. For more information, see Wallaroo SDK Essentials Guide: Pipeline Log Management.

For example:

# publish the pipeline and replace the following existing publishes
pipeline.publish(replace[publish1, publish2, publish3...])

Replace Pipeline Publish via the Wallaroo SDK Examples

Verify Pipeline Publishes and Edge Locations

We verify that the pipeline is published and is assigned edge locations via the wallaroo.pipeline.Pipeline.publishes() method and wallaroo.pipeline.Pipeline.list_edges() method.

display('Publishes:')
display(mainpipeline.publishes())
display('Edge Locations')
display(mainpipeline.list_edges())
'Publishes:'
idpipeline_version_nameengine_urlpipeline_urlcreated_bycreated_atupdated_at
12edb0c329-5b5f-4c4c-801a-07b620322a7bghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/fitzroy-mini:v2024.1.0-main-4806ghcr.io/wallaroolabs/doc-samples/pipelines/edge-inline-replacement-demon:edb0c329-5b5f-4c4c-801a-07b620322a7bjohn.hummel@wallaroo.ai2024-27-Mar 20:13:202024-27-Mar 20:13:20
'Edge Locations'
IDNameTagsSPIFFE ID
9f9d9afa-b6bc-4a75-b811-2a56da74f02bhouseprice-edge-inline-demonstration[]wallaroo.ai/ns/deployments/edge/9f9d9afa-b6bc-4a75-b811-2a56da74f02b

How to Replace a Pipeline Publish from a New Pipeline Version

The following demonstrates replacing a pipeline publish with a new publish from a pipeline version. In this procedure, the edge location has an existing model edge deployment. The publish that is used for that deployment is replaced, and an in-line update of the model deployment is performed on the edge device.

Create New Pipeline Version

For the next demonstration, we will use the same pipeline and update the pipeline steps with a new model. This time with the model ./models/xgb_model.onnx, which takes the same input as ./models/rf_model.onnx. We will also set a different name for this model to distinguish it from the one previously uploaded and used as the pipeline step.

Once set, we will deploy the pipeline with the new model, and perform a new inference request. From this, we will see that this new model outputs a slightly different prediction that the previous one - closer to $650k.

housing_model_challenger01 = (wl.upload_model("xgb-house-price-estimator", 
                                        './models/xgb_model.onnx', 
                                        framework=Framework.ONNX)
                                        .configure(tensor_fields=["tensor"])
                        )

mainpipeline.add_model_step(housing_model_challenger01)

Replace Pipeline Publish with new Pipeline Version

From our pipeline, we will publish the new version. The method wallaroo.pipeline.Pipeline.publish() creates a new publish from the most recent pipeline version and takes the following parameters.

We will replace our publish earlier, labeled pub, with the new publish generated from the pipeline labeled new_pub.

new_pub = mainpipeline.publish(replaces=[pub])
new_pub
Waiting for pipeline publish... It may take up to 600 sec.
Pipeline is publishing...... Published.
ID13
Pipeline Nameedge-inline-replacement-demon
Pipeline Version2e78349c-d282-41c9-a435-0c943c3accf1
StatusPublished
Engine URLghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/fitzroy-mini:v2024.1.0-main-4806
Pipeline URLghcr.io/wallaroolabs/doc-samples/pipelines/edge-inline-replacement-demon:2e78349c-d282-41c9-a435-0c943c3accf1
Helm Chart URLoci://ghcr.io/wallaroolabs/doc-samples/charts/edge-inline-replacement-demon
Helm Chart Referenceghcr.io/wallaroolabs/doc-samples/charts@sha256:0b4697ad1f3f6ec79f00d9b572e48d00a60db63a169afce1074f6670be915dd2
Helm Chart Version0.0.1-2e78349c-d282-41c9-a435-0c943c3accf1
Engine Config{'engine': {'resources': {'limits': {'cpu': 4.0, 'memory': '3Gi'}, 'requests': {'cpu': 4.0, 'memory': '3Gi'}, 'accel': 'none', 'arch': 'x86', 'gpu': False}}, 'engineAux': {'autoscale': {'type': 'none'}, 'images': None}, 'enginelb': {'resources': {'limits': {'cpu': 1.0, 'memory': '512Mi'}, 'requests': {'cpu': 0.2, 'memory': '512Mi'}, 'accel': 'none', 'arch': 'x86', 'gpu': False}}}
User Images[]
Created Byjohn.hummel@wallaroo.ai
Created At2024-03-27 20:18:55.313370+00:00
Updated At2024-03-27 20:18:55.313370+00:00
ReplacesPublish 12, Pipeline "edge-inline-replacement-demon", Version 43
Docker Run Command
EdgeCommand
houseprice-edge-inline-demonstration
docker run -v $PERSISTENT_VOLUME_DIR:/persist \
    -e OCI_USERNAME=$OCI_USERNAME \
    -e OCI_PASSWORD=$OCI_PASSWORD \
    -e PIPELINE_URL=ghcr.io/wallaroolabs/doc-samples/pipelines/edge-inline-replacement-demon:2e78349c-d282-41c9-a435-0c943c3accf1\
    -e EDGE_BUNDLE=ZXhwb3J0IEJVTkRMRV9WRVJTSU9OPTEKZXhwb3J0IENPTkZJR19DUFVTPTQKZXhwb3J0IEVER0VfTkFNRT1ob3VzZXByaWNlLWVkZ2UtaW5saW5lLWRlbW9uc3RyYXRpb24KZXhwb3J0IE9QU0NFTlRFUl9IT1NUPWRvYy10ZXN0LmVkZ2Uud2FsbGFyb29jb21tdW5pdHkubmluamEKZXhwb3J0IFBJUEVMSU5FX1VSTD1naGNyLmlvL3dhbGxhcm9vbGFicy9kb2Mtc2FtcGxlcy9waXBlbGluZXMvZWRnZS1pbmxpbmUtcmVwbGFjZW1lbnQtZGVtb246MmU3ODM0OWMtZDI4Mi00MWM5LWE0MzUtMGM5NDNjM2FjY2YxCmV4cG9ydCBKT0lOX1RPS0VOPW9taXR0ZWRfZm9yX3JlcGxhY2UKZXhwb3J0IE9DSV9SRUdJU1RSWT1naGNyLmlv \
    ghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/fitzroy-mini:v2024.1.0-main-4806

Note: Please set the PERSISTENT_VOLUME_DIR, OCI_USERNAME, and OCI_PASSWORD environment variables.
Helm Install Command
EdgeCommand
houseprice-edge-inline-demonstration
helm upgrade $HELM_INSTALL_NAME \
    oci://ghcr.io/wallaroolabs/doc-samples/charts/edge-inline-replacement-demon \
    --namespace $HELM_INSTALL_NAMESPACE \
    --version 0.0.1-2e78349c-d282-41c9-a435-0c943c3accf1 \
    --set ociRegistry.username=$OCI_USERNAME \
    --set ociRegistry.password=$OCI_PASSWORD \
    --set edgeBundle=ZXhwb3J0IEJVTkRMRV9WRVJTSU9OPTEKZXhwb3J0IENPTkZJR19DUFVTPTQKZXhwb3J0IEVER0VfTkFNRT1ob3VzZXByaWNlLWVkZ2UtaW5saW5lLWRlbW9uc3RyYXRpb24KZXhwb3J0IE9QU0NFTlRFUl9IT1NUPWRvYy10ZXN0LmVkZ2Uud2FsbGFyb29jb21tdW5pdHkubmluamEKZXhwb3J0IFBJUEVMSU5FX1VSTD1naGNyLmlvL3dhbGxhcm9vbGFicy9kb2Mtc2FtcGxlcy9waXBlbGluZXMvZWRnZS1pbmxpbmUtcmVwbGFjZW1lbnQtZGVtb246MmU3ODM0OWMtZDI4Mi00MWM5LWE0MzUtMGM5NDNjM2FjY2YxCmV4cG9ydCBKT0lOX1RPS0VOPW9taXR0ZWRfZm9yX3JlcGxhY2UKZXhwb3J0IE9DSV9SRUdJU1RSWT1naGNyLmlv

Note: Please set the PERSISTENT_VOLUME_DIR, HELM_INSTALL_NAME, HELM_INSTALL_NAMESPACE, OCI_USERNAME, and OCI_PASSWORD environment variables.

Note that in the Replaces section, updates are given for each edge location. On the edge device, we deploy the new pipeline publish either by the provided Docker Run Command or Helm Install Command. Note that the Helm Install Command uses helm upgrade rather than helm install to create a new revision. The pipeline steps and models will be changed at the model edge location.

How to Replace a Pipeline Publish from a New Pipeline

For this example, we will create a separate pipeline than the previous one with its own model. This model will still accept the same inputs and output fields as the models previously deployed. This is not a requirement - completely different models with different inputs and output schemas can be used; this example is used for simplicity.

For this set of examples, we will upload the third model, ./models/gbr_model.onnx and create a new pipeline named new-edge-inline-replacement. This model is added as a pipeline step and deployed.

housing_model_challenger02 = (wl.upload_model("gbr-house-price-estimator", 
                                        './models/gbr_model.onnx', 
                                        framework=Framework.ONNX)
                                        .configure(tensor_fields=["tensor"])
                        )

new_pipeline = wl.build_pipeline("new-edge-inline-replacement")

# clear the steps
new_pipeline.clear()

new_pipeline.add_model_step(housing_model_challenger02)

Replace Pipeline Publish with Publish from Different Pipeline

We will now publish the new pipeline, and replace the previous edge location publish from our original pipeline. The last publish is labeled new_pub is labeled new_pipeline_pub.

new_pipeline_pub = new_pipeline.publish(replaces=[new_pub])
new_pipeline_pub
Waiting for pipeline publish... It may take up to 600 sec.
Pipeline is publishing...... Published.
ID14
Pipeline Namenew-edge-inline-replacement
Pipeline Versiona7aceb15-68a8-4d8f-b7fa-0b70a46dfa62
StatusPublished
Engine URLghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/fitzroy-mini:v2024.1.0-main-4806
Pipeline URLghcr.io/wallaroolabs/doc-samples/pipelines/new-edge-inline-replacement:a7aceb15-68a8-4d8f-b7fa-0b70a46dfa62
Helm Chart URLoci://ghcr.io/wallaroolabs/doc-samples/charts/new-edge-inline-replacement
Helm Chart Referenceghcr.io/wallaroolabs/doc-samples/charts@sha256:7d5b6ec9fe65be4a6e10e6f266494f73593ef5ee26de286447302961f261871e
Helm Chart Version0.0.1-a7aceb15-68a8-4d8f-b7fa-0b70a46dfa62
Engine Config{'engine': {'resources': {'limits': {'cpu': 4.0, 'memory': '3Gi'}, 'requests': {'cpu': 4.0, 'memory': '3Gi'}, 'accel': 'none', 'arch': 'x86', 'gpu': False}}, 'engineAux': {'autoscale': {'type': 'none'}, 'images': None}, 'enginelb': {'resources': {'limits': {'cpu': 1.0, 'memory': '512Mi'}, 'requests': {'cpu': 0.2, 'memory': '512Mi'}, 'accel': 'none', 'arch': 'x86', 'gpu': False}}}
User Images[]
Created Byjohn.hummel@wallaroo.ai
Created At2024-03-27 20:23:59.059717+00:00
Updated At2024-03-27 20:23:59.059717+00:00
ReplacesPublish 13, Pipeline "edge-inline-replacement-demon", Version 45
Docker Run Command
EdgeCommand
houseprice-edge-inline-demonstration
docker run -v $PERSISTENT_VOLUME_DIR:/persist \
    -e OCI_USERNAME=$OCI_USERNAME \
    -e OCI_PASSWORD=$OCI_PASSWORD \
    -e PIPELINE_URL=ghcr.io/wallaroolabs/doc-samples/pipelines/new-edge-inline-replacement:a7aceb15-68a8-4d8f-b7fa-0b70a46dfa62\
    -e EDGE_BUNDLE=ZXhwb3J0IEJVTkRMRV9WRVJTSU9OPTEKZXhwb3J0IENPTkZJR19DUFVTPTQKZXhwb3J0IEVER0VfTkFNRT1ob3VzZXByaWNlLWVkZ2UtaW5saW5lLWRlbW9uc3RyYXRpb24KZXhwb3J0IE9QU0NFTlRFUl9IT1NUPWRvYy10ZXN0LmVkZ2Uud2FsbGFyb29jb21tdW5pdHkubmluamEKZXhwb3J0IFBJUEVMSU5FX1VSTD1naGNyLmlvL3dhbGxhcm9vbGFicy9kb2Mtc2FtcGxlcy9waXBlbGluZXMvbmV3LWVkZ2UtaW5saW5lLXJlcGxhY2VtZW50OmE3YWNlYjE1LTY4YTgtNGQ4Zi1iN2ZhLTBiNzBhNDZkZmE2MgpleHBvcnQgSk9JTl9UT0tFTj1vbWl0dGVkX2Zvcl9yZXBsYWNlCmV4cG9ydCBPQ0lfUkVHSVNUUlk9Z2hjci5pbw== \
    ghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/fitzroy-mini:v2024.1.0-main-4806

Note: Please set the PERSISTENT_VOLUME_DIR, OCI_USERNAME, and OCI_PASSWORD environment variables.
Helm Install Command
EdgeCommand
houseprice-edge-inline-demonstration
helm upgrade $HELM_INSTALL_NAME \
    oci://ghcr.io/wallaroolabs/doc-samples/charts/new-edge-inline-replacement \
    --namespace $HELM_INSTALL_NAMESPACE \
    --version 0.0.1-a7aceb15-68a8-4d8f-b7fa-0b70a46dfa62 \
    --set ociRegistry.username=$OCI_USERNAME \
    --set ociRegistry.password=$OCI_PASSWORD \
    --set edgeBundle=ZXhwb3J0IEJVTkRMRV9WRVJTSU9OPTEKZXhwb3J0IENPTkZJR19DUFVTPTQKZXhwb3J0IEVER0VfTkFNRT1ob3VzZXByaWNlLWVkZ2UtaW5saW5lLWRlbW9uc3RyYXRpb24KZXhwb3J0IE9QU0NFTlRFUl9IT1NUPWRvYy10ZXN0LmVkZ2Uud2FsbGFyb29jb21tdW5pdHkubmluamEKZXhwb3J0IFBJUEVMSU5FX1VSTD1naGNyLmlvL3dhbGxhcm9vbGFicy9kb2Mtc2FtcGxlcy9waXBlbGluZXMvbmV3LWVkZ2UtaW5saW5lLXJlcGxhY2VtZW50OmE3YWNlYjE1LTY4YTgtNGQ4Zi1iN2ZhLTBiNzBhNDZkZmE2MgpleHBvcnQgSk9JTl9UT0tFTj1vbWl0dGVkX2Zvcl9yZXBsYWNlCmV4cG9ydCBPQ0lfUkVHSVNUUlk9Z2hjci5pbw==

Note: Please set the PERSISTENT_VOLUME_DIR, HELM_INSTALL_NAME, HELM_INSTALL_NAMESPACE, OCI_USERNAME, and OCI_PASSWORD environment variables.

Once complete, we update the edge deployment with the new replacement docker run or helm based deployment commands from the configuration.

With the edge location replacement complete with the new pipeline, we list the edge locations from the original pipeline and the new one to show the edge location is transferred to the new pipeline.

display(mainpipeline.list_edges())

(no pipelines)

display(new_pipeline.list_edges())
IDNameTagsSPIFFE ID
9f9d9afa-b6bc-4a75-b811-2a56da74f02bhouseprice-edge-inline-demonstration[]wallaroo.ai/ns/deployments/edge/9f9d9afa-b6bc-4a75-b811-2a56da74f02b

Viewing In-Line Model Updates via the Wallaroo Dashboard

In-Line model updates are visible through the Wallaroo Dashboard via the Pipeline Details page through the following process.

  1. From the Wallaroo Dashboad, select the workspace from the workspace dropdown list.

  2. Select the pipeline the publishes are transferred to.

  3. From the Filter Edges list, the edges locations that were part of the previous pipeline version or were transferred from a different pipeline are displayed.

Examples

The following examples demonstrate in-line model replacement on multicloud or edge deployedments.