Wallaroo pipeline publishes are containerized versions of the pipeline, models and model steps, and the inference engine published to an Open Container Initiative (OCI) Registry. Once published, they are used to deploy models to edge locations and used for inference requests.

Edge locations added to a pipeline published allow deployments on edge devices to connect with the Wallaroo Ops instance and transmit their inference results as part of the pipeline logs.

Pipeline publishes can be replaced by new pipeline publishes from the following sources:

• A new version of the same pipeline that creates a new pipeline publish. In these instances, the edge locations associated with the pipeline publish are preserved with the original pipeline along with their pipeline inference logs. Any input or output schema changes from the new models or models steps are reflected in the pipeline logs.
• A separate pipeline with its own models and model steps. In this scenario, the edge locations associated with the original pipeline publish are assigned to the new pipeline. The inference logs are stored with the original pipeline, and new inference logs for the edge locations are stored as part of the new pipeline.

For more information on pipeline inference logs, see Inference Logs.

Model Update Considerations

When updating an pipeline with new model or an edge location with a new pipeline publish from either a new pipeline version or a new pipeline, the following considerations should be taken.

• Inference Log Updates: If the new model(s) have different inputs and outputs schemas, the pipeline inference logs will change to reflect the new data schemas.
• Model Drift Detection: Assays are based on the pipeline and the model name. If either of these change as part of the either new pipeline versions, or a new pipeline, then the previous assays will cease to generate new analyses. It is recommended to “pause” assays where the pipeline or model name has chanced, the create new ones. For more information on creating assays, see Model Drift Detection with Model Insights.
• Inference Request Updates: If the input or output schema for the pipeline changes including field names, field data types, etc, users should verify that the inference requests are modified to match.

How to Replace a Pipeline Publish

Pipeline publishes are replaced with the wallaroo.pipeline.Pipeline.publish(deployment_config, replaces) method. This creates a new publish from the most recent pipeline version and takes the following parameters.

When a pipeline published is replaced with a new one, the edge locations are transferred to the pipeline that the publish came from. In this example, this is the same pipeline. Inference results from the edge location deployments are stored with the pipeline that generated the publish. Note that if the model or pipeline steps have changed from one pipeline version to the next, the pipeline log schema will change with it. For more information, see Wallaroo SDK Essentials Guide: Pipeline Log Management.

For example:

# publish the pipeline and replace the following existing publishes
pipeline.publish(replace[publish1, publish2, publish3...])

Replace Pipeline Publish via the Wallaroo SDK Examples

Verify Pipeline Publishes and Edge Locations

We verify that the pipeline is published and is assigned edge locations via the wallaroo.pipeline.Pipeline.publishes() method and wallaroo.pipeline.Pipeline.list_edges() method.

display('Publishes:')
display(mainpipeline.publishes())
display('Edge Locations')
display(mainpipeline.list_edges())

'Publishes:'

'Edge Locations'

How to Replace a Pipeline Publish from a New Pipeline Version

The following demonstrates replacing a pipeline publish with a new publish from a pipeline version. In this procedure, the edge location has an existing model edge deployment. The publish that is used for that deployment is replaced, and an in-line update of the model deployment is performed on the edge device.

Create New Pipeline Version

For the next demonstration, we will use the same pipeline and update the pipeline steps with a new model. This time with the model ./models/xgb_model.onnx, which takes the same input as ./models/rf_model.onnx. We will also set a different name for this model to distinguish it from the one previously uploaded and used as the pipeline step.

Once set, we will deploy the pipeline with the new model, and perform a new inference request. From this, we will see that this new model outputs a slightly different prediction that the previous one - closer to $650k.

housing_model_challenger01 = (wl.upload_model("xgb-house-price-estimator", 
                                        './models/xgb_model.onnx', 
                                        framework=Framework.ONNX)
                                        .configure(tensor_fields=["tensor"])
                        )

mainpipeline.add_model_step(housing_model_challenger01)

Replace Pipeline Publish with new Pipeline Version

From our pipeline, we will publish the new version. The method wallaroo.pipeline.Pipeline.publish() creates a new publish from the most recent pipeline version and takes the following parameters.

We will replace our publish earlier, labeled pub, with the new publish generated from the pipeline labeled new_pub.

new_pub = mainpipeline.publish(replaces=[pub])
new_pub

Waiting for pipeline publish... It may take up to 600 sec.
Pipeline is publishing...... Published.

docker run -v $PERSISTENT_VOLUME_DIR:/persist \
    -e OCI_USERNAME=$OCI_USERNAME \
    -e OCI_PASSWORD=$OCI_PASSWORD \
    -e PIPELINE_URL=ghcr.io/wallaroolabs/doc-samples/pipelines/edge-inline-replacement-demon:2e78349c-d282-41c9-a435-0c943c3accf1\
    -e EDGE_BUNDLE=ZXhwb3J0IEJVTkRMRV9WRVJTSU9OPTEKZXhwb3J0IENPTkZJR19DUFVTPTQKZXhwb3J0IEVER0VfTkFNRT1ob3VzZXByaWNlLWVkZ2UtaW5saW5lLWRlbW9uc3RyYXRpb24KZXhwb3J0IE9QU0NFTlRFUl9IT1NUPWRvYy10ZXN0LmVkZ2Uud2FsbGFyb29jb21tdW5pdHkubmluamEKZXhwb3J0IFBJUEVMSU5FX1VSTD1naGNyLmlvL3dhbGxhcm9vbGFicy9kb2Mtc2FtcGxlcy9waXBlbGluZXMvZWRnZS1pbmxpbmUtcmVwbGFjZW1lbnQtZGVtb246MmU3ODM0OWMtZDI4Mi00MWM5LWE0MzUtMGM5NDNjM2FjY2YxCmV4cG9ydCBKT0lOX1RPS0VOPW9taXR0ZWRfZm9yX3JlcGxhY2UKZXhwb3J0IE9DSV9SRUdJU1RSWT1naGNyLmlv \
    ghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/fitzroy-mini:v2024.1.0-main-4806

helm upgrade $HELM_INSTALL_NAME \
    oci://ghcr.io/wallaroolabs/doc-samples/charts/edge-inline-replacement-demon \
    --namespace $HELM_INSTALL_NAMESPACE \
    --version 0.0.1-2e78349c-d282-41c9-a435-0c943c3accf1 \
    --set ociRegistry.username=$OCI_USERNAME \
    --set ociRegistry.password=$OCI_PASSWORD \
    --set edgeBundle=ZXhwb3J0IEJVTkRMRV9WRVJTSU9OPTEKZXhwb3J0IENPTkZJR19DUFVTPTQKZXhwb3J0IEVER0VfTkFNRT1ob3VzZXByaWNlLWVkZ2UtaW5saW5lLWRlbW9uc3RyYXRpb24KZXhwb3J0IE9QU0NFTlRFUl9IT1NUPWRvYy10ZXN0LmVkZ2Uud2FsbGFyb29jb21tdW5pdHkubmluamEKZXhwb3J0IFBJUEVMSU5FX1VSTD1naGNyLmlvL3dhbGxhcm9vbGFicy9kb2Mtc2FtcGxlcy9waXBlbGluZXMvZWRnZS1pbmxpbmUtcmVwbGFjZW1lbnQtZGVtb246MmU3ODM0OWMtZDI4Mi00MWM5LWE0MzUtMGM5NDNjM2FjY2YxCmV4cG9ydCBKT0lOX1RPS0VOPW9taXR0ZWRfZm9yX3JlcGxhY2UKZXhwb3J0IE9DSV9SRUdJU1RSWT1naGNyLmlv

Note that in the Replaces section, updates are given for each edge location. On the edge device, we deploy the new pipeline publish either by the provided Docker Run Command or Helm Install Command. Note that the Helm Install Command uses helm upgrade rather than helm install to create a new revision. The pipeline steps and models will be changed at the model edge location.

How to Replace a Pipeline Publish from a New Pipeline

For this example, we will create a separate pipeline than the previous one with its own model. This model will still accept the same inputs and output fields as the models previously deployed. This is not a requirement - completely different models with different inputs and output schemas can be used; this example is used for simplicity.

For this set of examples, we will upload the third model, ./models/gbr_model.onnx and create a new pipeline named new-edge-inline-replacement. This model is added as a pipeline step and deployed.

housing_model_challenger02 = (wl.upload_model("gbr-house-price-estimator", 
                                        './models/gbr_model.onnx', 
                                        framework=Framework.ONNX)
                                        .configure(tensor_fields=["tensor"])
                        )

new_pipeline = wl.build_pipeline("new-edge-inline-replacement")

# clear the steps
new_pipeline.clear()

new_pipeline.add_model_step(housing_model_challenger02)

Replace Pipeline Publish with Publish from Different Pipeline

We will now publish the new pipeline, and replace the previous edge location publish from our original pipeline. The last publish is labeled new_pub is labeled new_pipeline_pub.

new_pipeline_pub = new_pipeline.publish(replaces=[new_pub])
new_pipeline_pub

Waiting for pipeline publish... It may take up to 600 sec.
Pipeline is publishing...... Published.

docker run -v $PERSISTENT_VOLUME_DIR:/persist \
    -e OCI_USERNAME=$OCI_USERNAME \
    -e OCI_PASSWORD=$OCI_PASSWORD \
    -e PIPELINE_URL=ghcr.io/wallaroolabs/doc-samples/pipelines/new-edge-inline-replacement:a7aceb15-68a8-4d8f-b7fa-0b70a46dfa62\
    -e EDGE_BUNDLE=ZXhwb3J0IEJVTkRMRV9WRVJTSU9OPTEKZXhwb3J0IENPTkZJR19DUFVTPTQKZXhwb3J0IEVER0VfTkFNRT1ob3VzZXByaWNlLWVkZ2UtaW5saW5lLWRlbW9uc3RyYXRpb24KZXhwb3J0IE9QU0NFTlRFUl9IT1NUPWRvYy10ZXN0LmVkZ2Uud2FsbGFyb29jb21tdW5pdHkubmluamEKZXhwb3J0IFBJUEVMSU5FX1VSTD1naGNyLmlvL3dhbGxhcm9vbGFicy9kb2Mtc2FtcGxlcy9waXBlbGluZXMvbmV3LWVkZ2UtaW5saW5lLXJlcGxhY2VtZW50OmE3YWNlYjE1LTY4YTgtNGQ4Zi1iN2ZhLTBiNzBhNDZkZmE2MgpleHBvcnQgSk9JTl9UT0tFTj1vbWl0dGVkX2Zvcl9yZXBsYWNlCmV4cG9ydCBPQ0lfUkVHSVNUUlk9Z2hjci5pbw== \
    ghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/fitzroy-mini:v2024.1.0-main-4806

helm upgrade $HELM_INSTALL_NAME \
    oci://ghcr.io/wallaroolabs/doc-samples/charts/new-edge-inline-replacement \
    --namespace $HELM_INSTALL_NAMESPACE \
    --version 0.0.1-a7aceb15-68a8-4d8f-b7fa-0b70a46dfa62 \
    --set ociRegistry.username=$OCI_USERNAME \
    --set ociRegistry.password=$OCI_PASSWORD \
    --set edgeBundle=ZXhwb3J0IEJVTkRMRV9WRVJTSU9OPTEKZXhwb3J0IENPTkZJR19DUFVTPTQKZXhwb3J0IEVER0VfTkFNRT1ob3VzZXByaWNlLWVkZ2UtaW5saW5lLWRlbW9uc3RyYXRpb24KZXhwb3J0IE9QU0NFTlRFUl9IT1NUPWRvYy10ZXN0LmVkZ2Uud2FsbGFyb29jb21tdW5pdHkubmluamEKZXhwb3J0IFBJUEVMSU5FX1VSTD1naGNyLmlvL3dhbGxhcm9vbGFicy9kb2Mtc2FtcGxlcy9waXBlbGluZXMvbmV3LWVkZ2UtaW5saW5lLXJlcGxhY2VtZW50OmE3YWNlYjE1LTY4YTgtNGQ4Zi1iN2ZhLTBiNzBhNDZkZmE2MgpleHBvcnQgSk9JTl9UT0tFTj1vbWl0dGVkX2Zvcl9yZXBsYWNlCmV4cG9ydCBPQ0lfUkVHSVNUUlk9Z2hjci5pbw==

Once complete, we update the edge deployment with the new replacement docker run or helm based deployment commands from the configuration.

With the edge location replacement complete with the new pipeline, we list the edge locations from the original pipeline and the new one to show the edge location is transferred to the new pipeline.

display(mainpipeline.list_edges())

(no pipelines)

display(new_pipeline.list_edges())

Viewing In-Line Model Updates via the Wallaroo Dashboard

In-Line model updates are visible through the Wallaroo Dashboard via the Pipeline Details page through the following process.

1. From the Wallaroo Dashboad, select the workspace from the workspace dropdown list.

2. Select the pipeline the publishes are transferred to.

3. From the Filter Edges list, the edges locations that were part of the previous pipeline version or were transferred from a different pipeline are displayed.

   

Examples

The following examples demonstrate in-line model replacement on multicloud or edge deployedments.

• In-Line Model Updates at the Edge Tutorial