Edge and Multicloud Model Publish and Deploy

How to publish models for deployment on edge and multicloud environments.

Model Publish for Edge and Multicloud Deployment via the Wallaroo Dashboard

Wallaroo pipeline publications are managed through the Wallaroo Dashboard Pipeline pages. This requires that Edge Deployment Registry is enabled.

Wallaroo pipelines are published as containers to OCI registries, and are referred to as publishes.

Access Wallaroo Pipeline Publishes

To view the publishes for a specific pipeline through the Wallaroo Dashboard:

  1. Login to the Wallaroo Dashboard through your browser.
  2. From the Workspace select menu on the upper left, select the workspace the pipeline is associated in.
  3. Select the pipeline to view the Pipeline Versions, which contain the Pipeline Publishes for each Pipeline Versions.
  4. The list of pipeline versions are available in the Version History section.
    1. Unpublished versions are indicated with a black box (A) to the right of the pipeline version. Published pipelines are indicated with a gray box. (B). Publish details are visible by selecting Check Info (C).

      Wallaroo Published Pipeline Versions
    2. Select Check Info to view pipeline details.

      PIpeline Publish Details
      • Pipeline location (A): The URL for the containerized pipeline.
      • PIpeline Chart (B): The URL for the Helm chart of the published pipeline and engine.
      • Engine url (B): The URL for the Wallaroo Engine required to deploy the pipeline and perform inference requests.

Publish a Wallaroo Pipeline Version

To publish a version of the Wallaroo pipeline:

From the Pipeline Versions view:

  1. Select the black box to the right of a Pipeline Version identifier. Grey boxes indicate that the pipeline version is already published.
  2. Wait for the publish to complete. Depending on the number and size of the pipeline steps in the pipeline version, this may take anywhere from 1 to 10 minutes.

Publish a Pipeline to the Edge Registry Service

See also the reference documentation: wallaroo.pipeline.publish.

Publish a Pipeline

Pipelines are published as images to the edge registry set in the Enable Wallaroo Edge Registry with the wallaroo.pipeline.Pipeline.publish method.

When a pipeline is published, the containerized pipeline with its models, and the inference engine for the architecture and acceleration uploaded to the OCI registry. Once published, the publish is deployed on edge locations either with Docker or helm based deployments. See DevOps - Pipeline Edge Deployment for more details.

Publish a Pipeline Parameters

The wallaroo.pipeline.Pipeline.publish method takes the following parameters. The containerized pipeline will be pushed to the Edge registry service with the model, pipeline configurations, and other artifacts needed to deploy the pipeline.

ParameterTypeDescription
deployment_configwallaroo.deployment_config.DeploymentConfig (Optional)Sets the pipeline deployment configuration. For example: For more information on pipeline deployment configuration, see the Wallaroo SDK Essentials Guide: Pipeline Deployment Configuration.
replaces[List[wallaroo.pipeline_publish]] (Optional)The pipeline publish(es) to replace.

Publish a Pipeline Returns

The following parameters are returned when a pipeline is published. The following shows the PipelinePublish object details returned.

FieldTypeDescription
idIntegerNumerical Wallaroo id of the published pipeline.
pipeline_nameStringThe name of the pipeline the publish is generated from.
pipeline_version_idIntegerNumerical Wallaroo id of the pipeline version published.
statusStringThe status of the pipeline publication. Values include:
  • PendingPublish: The pipeline publication is about to be uploaded or is in the process of being uploaded.
  • Published: The pipeline is published and ready for use.
engine_urlStringThe URL of the published pipeline engine in the edge registry.
pipeline_urlStringThe URL of the published pipeline in the edge registry.
pipeline_version_nameStringThe pipeline version in UUID format.
helmDictThe details used for a helm based deployment of the with the following attributes:
  • reference: The OCI URL of the Helm reference.
  • values: Any additional values.
  • chart: The Helm chart for the edge deployment.
  • version: The Helm version to specify which published pipeline Helm version to use.
additional_propertiesDictAny additional properties for the publish.
docker_run_variablesThe Docker Run variables used for Docker based deployments. This includes:
  • PIPELINE_URL: The OCI registry URL of the containerized pipeline.
  • EDGE_BUNDLE: The Edge Bundle for edge locations. See Add Edge for more details.
engine_urlStringThe URL for the inference engine used for the edge deployment.
user_imagesListA List of custom images used for the edge deployment.
created_byStringThe Keycloak user ID of the user that created the publish in UUID format.
errorStringAny errors associated with the publish.
engine_configwallaroo.deployment_config.DeploymentConfigThe pipeline configuration included with the published pipeline.
created_atDateTimeWhen the published pipeline was created.
updated_atDateTimeWhen the published pipeline was updated.
created_on_versionStringThe version of Wallaroo the publish was generated from.
replacesList(Integer)List of other publishes that were replaced by this one.

When a publish is displayed using IPython.display, the following fields are generated. Some are generated from the fields above to make deployment easier for end users.

FieldTypeDescription
IDIntegerThe numberical ID of the publish.
Pipeline NameStringThe pipeline the publish was generated from.
Pipeline VersionStringThe pipeline version the publish was generated from, in UUID format.
StatusStringThe status of the publish. Values include:
  • PendingPublish: The pipeline publication is about to be uploaded or is in the process of being uploaded.
  • Published: The pipeline is published and ready for use.
Engine URLStringThe OCI Registry URL for the inference engine.
Pipeline URLStringThe OCI Registry URL of the containerized pipeline.
Helm Chart URLStringThe OCI Registry URL of the Helm chart.
Helm Chart ReferenceStringThe OCI Registry URL of the Helm Chart reference.
Helm Chart VersionStringThe Helm Chart Version.
Engine ConfigDictThe details of the wallaroo.engine_config used for the publish. Unless specified, it will use the same engine config for the pipeline, which inherits its arch and accel settings from the model upon upload. See Wallaroo SDK Essentials Guide: Model Uploads and Registrations for more details.
User ImagesListAny user images used with the deployment.
Created ByStringThe user name, typically the email address, of the user that created the publish.
Created AtDateTimeThe DateTime of the publish was created.
Updated AtDateTimeThe DateTime of the publish was updated.
ReplacesListA list of the publishes that were replaced by this one. These detail:
  • ID: The replaced publish id.
  • Pipeline: The name of the replaced pipeline.
  • Version: The pipeline version number replaced.
Docker Run CommandThe Docker Run commands for each edge location for the publish. Each Edge will show its Edge Location name, and the Docker Run command for that edge location. The following variables must be set before executing the command.
  • PERSISTENT_VOLUME_DIR: The location for the persistent volume used by the edge location to store session information, logs, etc.
  • OCI_USERNAME: The username for the OCI registry containing the publish.
  • OCI_PASSWORD: The password for the user used to authenticate to the OCI registry containing the publish.
Helm Install CommandThe Helm Install or Upgrade commands for each location or replaced locations for the pipeline. For replaced publishes, the helm upgrade command is shown for performing in-line model updates. The following variables must be set before executing the command.
  • OCI_USERNAME: The username for the OCI registry containing the publish.
  • OCI_PASSWORD: The password for the user used to authenticate to the OCI registry containing the publish.
  • HELM_INSTALL_NAME: The name of the Helm deployment installation.
  • HELM_INSTALL_NAMESPACE: The Kubernetes namespace the Helm based edge deployment is installed in.

Publish a Pipeline Example

The following example shows how to publish a pipeline to the edge registry service associated with the Wallaroo instance.

# set the configuration
deployment_config = wallaroo.DeploymentConfigBuilder().replica_count(1).cpus(0.5).memory("900Mi").build()

# build the pipeline
pipeline = wl.build_pipeline("publish-example")
# add a model as a model step
pipeline.add_model_step(m2)
publish = pipeline.publish(deployment_config)
display(publish)
  
ID4
Pipeline Version ID10
StatusPublished
Engine URLsample-registry.example.com/engine:main
Pipeline URLsample-registry.example.com/pipelines/p1:6c3d9899-1335-456b-aaa0-52d03a017cc4
Helm Chart URLsample-registry.example.com/charts/p1
Helm Chart Referencesample-registry.example.com/charts@sha256:5523891f66fde830a9fc08603b3536dc2c4c1b16b51931ad2fdf9839e6eba129
Helm Chart Version0.0.1-6c3d9899-1335-456b-aaa0-52d03a017cc4
Engine Config{’engine’: {‘resources’: {’limits’: {‘cpu’: 0.5, ‘memory’: ‘900Mi’}, ‘requests’: {‘cpu’: 0.5, ‘memory’: ‘900Mi’}}}, ’engineAux’: {‘images’: {}}, ’enginelb’: {}}
Created Bydb13ee60-4162-4a42-a571-61c32e225e3e
Created At2023-08-17 14:04:44.939862+00:00
Updated At2023-08-17 14:04:44.939862+00:00

List Publishes

All publishes created from a pipeline are displayed with the wallaroo.pipeline.publishes method.

List Publishes Parameters

N/A

List Publishes Returns

A List of the following fields:

FieldTypeDescription
idintegerNumerical Wallaroo id of the published pipeline.
pipeline_version_idintegerNumerical Wallaroo id of the pipeline version published.
engine_urlStringThe URL of the published pipeline engine in the edge registry.
pipeline_urlStringThe URL of the published pipeline in the edge registry.
created_byStringThe email address of the user that published the pipeline.
Created AtDateTimeWhen the published pipeline was created.
Updated AtDateTimeWhen the published pipeline was updated.

List Publishes Example

The following shows a list of publishes from a pipeline.

pipeline.publishes()
idpipeline_version_idengine_urlpipeline_urlcreated_bycreated_atupdated_at
14sample-registry.example.com/engine:mainsample-registry.example.com/pipelines/p1:01ab76f3-d007-48c4-870d-3d3c5a46902ddb13ee60-4162-4a42-a571-61c32e225e3e2023-16-Aug 20:22:312023-16-Aug 20:22:31
26sample-registry.example.com/engine:mainsample-registry.example.com/pipelines/p1:d3fb1b50-fe9a-4f54-bc32-fb9ff4ba49eedb13ee60-4162-4a42-a571-61c32e225e3e2023-17-Aug 12:43:462023-17-Aug 12:43:46
38sample-registry.example.com/engine:mainsample-registry.example.com/pipelines/p1:97548f13-a791-41f5-bd54-c75649b72856db13ee60-4162-4a42-a571-61c32e225e3e2023-17-Aug 13:52:382023-17-Aug 13:52:38
410sample-registry.example.com/engine:mainsample-registry.example.com/pipelines/p1:6c3d9899-1335-456b-aaa0-52d03a017cc4db13ee60-4162-4a42-a571-61c32e225e3e2023-17-Aug 14:04:442023-17-Aug 14:04:44

Replace Publish

Publishes are replaced either from a pipeline version, or an entirely new pipeline via the wallaroo.pipeline.Pipeline.publish(replace=List[wallaroo.pipeline_publish.PipelinePublish]) method. When replaced, the original publish’s edge locations are transferred to the pipeline associated with the new publish.

When a publish is replaced by a pipeline version, the pipeline logs will continue to include the edge locations with any new changes to the schema, based on the new pipeline steps.

When a publish is replaced by a entirely different pipeline, the previous pipeline logs are associated with the previous pipeline. New logs are associated with the new pipeline.

When a publish is replaced, the edge locations can be redeployed with the new publish information, which includes the new pipeline steps, models, and other settings.

Replace Publish Considerations

When a pipeline is publish, the following considerations must be taken.

  • Model Drift Observability updates: Assays created for the specific pipeline and model name will cease to generate new analyses if the pipeline steps change the model names or field outputs that were previously set. In these instances, the assay should be paused and a new assay generated with the updated model name and fields. For more details see Model Drift Detection with Model Insights.
  • Pipeline Log Schema Changes: The pipeline log schema will change based on the input and output parameters with the new pipeline steps. For more details see Wallaroo SDK Essentials Guide: Pipeline Log Management.

Replace Publish Parameters

The wallaroo.pipeline.Pipeline.publish method takes the following parameters. The containerized pipeline will be pushed to the Edge registry service with the model, pipeline configurations, and other artifacts needed to deploy the pipeline.

ParameterTypeDescription
deployment_configwallaroo.deployment_config.DeploymentConfig (Optional)Sets the pipeline deployment configuration. For example: For more information on pipeline deployment configuration, see the Wallaroo SDK Essentials Guide: Pipeline Deployment Configuration.
replaces[List[wallaroo.pipeline_publish]] (Optional)The pipeline publish(es) to replace.

Replace Publish Returns

The following parameters are returned when a pipeline is published. The following shows the PipelinePublish object details returned.

FieldTypeDescription
idIntegerNumerical Wallaroo id of the published pipeline.
pipeline_nameStringThe name of the pipeline the publish is generated from.
pipeline_version_idIntegerNumerical Wallaroo id of the pipeline version published.
statusStringThe status of the pipeline publication. Values include:
  • PendingPublish: The pipeline publication is about to be uploaded or is in the process of being uploaded.
  • Published: The pipeline is published and ready for use.
engine_urlStringThe URL of the published pipeline engine in the edge registry.
pipeline_urlStringThe URL of the published pipeline in the edge registry.
pipeline_version_nameStringThe pipeline version in UUID format.
helmDictThe details used for a helm based deployment of the with the following attributes:
  • reference: The OCI URL of the Helm reference.
  • values: Any additional values.
  • chart: The Helm chart for the edge deployment.
  • version: The Helm version to specify which published pipeline Helm version to use.
additional_propertiesDictAny additional properties for the publish.
docker_run_variablesThe Docker Run variables used for Docker based deployments. This includes:
  • PIPELINE_URL: The OCI registry URL of the containerized pipeline.
  • EDGE_BUNDLE: The Edge Bundle for edge locations. See Add Edge for more details.
engine_urlStringThe URL for the inference engine used for the edge deployment.
user_imagesListA List of custom images used for the edge deployment.
created_byStringThe Keycloak user ID of the user that created the publish in UUID format.
errorStringAny errors associated with the publish.
engine_configwallaroo.deployment_config.DeploymentConfigThe pipeline configuration included with the published pipeline.
created_atDateTimeWhen the published pipeline was created.
updated_atDateTimeWhen the published pipeline was updated.
created_on_versionStringThe version of Wallaroo the publish was generated from.
replacesList(Integer)List of other publishes that were replaced by this one.

When a publish is displayed using IPython.display, the following fields are generated. Some are generated from the fields above to make deployment easier for end users.

FieldTypeDescription
IDIntegerThe numberical ID of the publish.
Pipeline NameStringThe pipeline the publish was generated from.
Pipeline VersionStringThe pipeline version the publish was generated from, in UUID format.
StatusStringThe status of the publish. Values include:
  • PendingPublish: The pipeline publication is about to be uploaded or is in the process of being uploaded.
  • Published: The pipeline is published and ready for use.
Engine URLStringThe OCI Registry URL for the inference engine.
Pipeline URLStringThe OCI Registry URL of the containerized pipeline.
Helm Chart URLStringThe OCI Registry URL of the Helm chart.
Helm Chart ReferenceStringThe OCI Registry URL of the Helm Chart reference.
Helm Chart VersionStringThe Helm Chart Version.
Engine ConfigDictThe details of the wallaroo.engine_config used for the publish. Unless specified, it will use the same engine config for the pipeline, which inherits its arch and accel settings from the model upon upload. See Wallaroo SDK Essentials Guide: Model Uploads and Registrations for more details.
User ImagesListAny user images used with the deployment.
Created ByStringThe username, typically the email address, of the user that created the publish.
Created AtDateTimeThe DateTime of the publish was created.
Updated AtDateTimeThe DateTime of the publish was updated.
ReplacesListA list of the publishes that were replaced by this one. These detail:
  • ID: The replaced publish id.
  • Pipeline: The name of the replaced pipeline.
  • Version: The pipeline version number replaced.
Docker Run Command The Docker Run commands for each edge location for the publish. Each Edge will show its Edge Location name, and the Docker Run command for that edge location. The following variables must be set before executing the command.
  • PERSISTENT_VOLUME_DIR: The location for the persistent volume used by the edge location to store session information, logs, etc.
  • OCI_USERNAME: The username for the OCI registry containing the publish.
  • OCI_PASSWORD: The password for the user used to authenticate to the OCI registry containing the publish.
Helm Install Command The Helm Install or Upgrade commands for each location or replaced locations for the pipeline. For replaced publishes, the helm upgrade command is shown for performing in-line model updates. The following variables must be set before executing the command.
  • OCI_USERNAME: The username for the OCI registry containing the publish.
  • OCI_PASSWORD: The password for the user used to authenticate to the OCI registry containing the publish.
  • HELM_INSTALL_NAME: The name of the Helm deployment installation.
  • HELM_INSTALL_NAMESPACE: The Kubernetes namespace the Helm based edge deployment is installed in.

Replace Publish Example

The following shows a pipeline with publishes and edge locations. The full demonstration is available at In-Line Model Updates at the Edge Tutorial.

display(mainpipeline)
  
nameedge-inline-replacement-demo
created2024-03-26 18:53:24.630184+00:00
last_updated2024-03-26 22:01:14.641540+00:00
deployedFalse
archNone
accelNone
tags
versionsc0396244-66ea-4729-9830-3dd11ccdf85a, f3c59078-3245-4697-bdec-238c65cab749, e41738a7-9d80-4f20-a8d6-0b479c53fa5a, d9192be1-4d5d-49be-b106-bf77dba8173f, 446aeed9-2d52-47ae-9e5c-f2a05ef0d4d6, 85058098-d395-4af4-98ce-25754175547f, 90643d74-8883-451c-aa2b-4c41d9af04f8, 5a1bf0a2-9595-41de-9852-44d211c2c0fb, a5105a19-7215-4e6b-9a8a-5ddcc6aa875c
stepsrf-house-price-estimator
publishedTrue
display(mainpipeline.publishes())
idpipeline_version_nameengine_urlpipeline_urlcreated_bycreated_atupdated_at
11446aeed9-2d52-47ae-9e5c-f2a05ef0d4d6ghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/fitzroy-mini:v2024.1.0-main-4781ghcr.io/wallaroolabs/doc-samples/pipelines/edge-inline-replacement-demo:446aeed9-2d52-47ae-9e5c-f2a05ef0d4d6john.hummel@wallaroo.ai2024-26-Mar 19:07:432024-26-Mar 19:07:43
1090643d74-8883-451c-aa2b-4c41d9af04f8ghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/fitzroy-mini:v2024.1.0-main-4781ghcr.io/wallaroolabs/doc-samples/pipelines/edge-inline-replacement-demo:90643d74-8883-451c-aa2b-4c41d9af04f8john.hummel@wallaroo.ai2024-26-Mar 18:54:242024-26-Mar 18:54:24
14f3c59078-3245-4697-bdec-238c65cab749ghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/fitzroy-mini:v2024.1.0-main-4781ghcr.io/wallaroolabs/doc-samples/pipelines/edge-inline-replacement-demo:f3c59078-3245-4697-bdec-238c65cab749john.hummel@wallaroo.ai2024-26-Mar 21:31:162024-26-Mar 21:31:16
display(mainpipeline.list_edges())
IDNameTagsSPIFFE ID
50fa7f30-7725-4f27-8ddb-6000e8674cd6houseprice-edge-inline-replacement-demo2[]wallaroo.ai/ns/deployments/edge/50fa7f30-7725-4f27-8ddb-6000e8674cd6
Replace Publish from Pipeline Version

For the following, the pipeline model steps are replaced, and the new version is published. The previous publish set to the variable pub is replaced by the new publish, which is stored in the variable new_pub.

housing_model_challenger01 = (wl.upload_model("xgb-house-price-estimator", 
                                        './models/xgb_model.onnx', 
                                        framework=Framework.ONNX)
                                        .configure(tensor_fields=["tensor"])
                        )

mainpipeline.clear()

mainpipeline.add_model_step(housing_model_challenger01)

new_pub = mainpipeline.publish(replaces=[pub])
new_pub
Waiting for pipeline publish... It may take up to 600 sec.
Pipeline is publishing...... Published.
ID11
Pipeline Nameedge-inline-replacement-demo
Pipeline Version446aeed9-2d52-47ae-9e5c-f2a05ef0d4d6
StatusPublished
Engine URLghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/fitzroy-mini:v2024.1.0-main-4781
Pipeline URLghcr.io/wallaroolabs/doc-samples/pipelines/edge-inline-replacement-demo:446aeed9-2d52-47ae-9e5c-f2a05ef0d4d6
Helm Chart URLoci://ghcr.io/wallaroolabs/doc-samples/charts/edge-inline-replacement-demo
Helm Chart Referenceghcr.io/wallaroolabs/doc-samples/charts@sha256:268a8b7b22b3ab9a62127f56ac5152d264fec194212d2100550ecb6dd6b1cc37
Helm Chart Version0.0.1-446aeed9-2d52-47ae-9e5c-f2a05ef0d4d6
Engine Config{'engine': {'resources': {'limits': {'cpu': 4.0, 'memory': '3Gi'}, 'requests': {'cpu': 4.0, 'memory': '3Gi'}, 'accel': 'none', 'arch': 'x86', 'gpu': False}}, 'engineAux': {'autoscale': {'type': 'none'}, 'images': None}, 'enginelb': {'resources': {'limits': {'cpu': 1.0, 'memory': '512Mi'}, 'requests': {'cpu': 0.2, 'memory': '512Mi'}, 'accel': 'none', 'arch': 'x86', 'gpu': False}}}
User Images[]
Created Byjohn.hummel@wallaroo.ai
Created At2024-03-26 19:07:43.158229+00:00
Updated At2024-03-26 19:07:43.158229+00:00
ReplacesPublish 10, Pipeline "edge-inline-replacement-demo", Version 26
Docker Run Command
EdgeCommand
houseprice-edge-inline-replacement-demo
docker run -v $PERSISTENT_VOLUME_DIR:/persist \
    -e OCI_USERNAME=$OCI_USERNAME \
    -e OCI_PASSWORD=$OCI_PASSWORD \
    -e PIPELINE_URL=ghcr.io/wallaroolabs/doc-samples/pipelines/edge-inline-replacement-demo:446aeed9-2d52-47ae-9e5c-f2a05ef0d4d6\
    -e EDGE_BUNDLE=ZXhwb3J0IEJVTkRMRV9WRVJTSU9OPTEKZXhwb3J0IENPTkZJR19DUFVTPTQKZXhwb3J0IEVER0VfTkFNRT1ob3VzZXByaWNlLWVkZ2UtaW5saW5lLXJlcGxhY2VtZW50LWRlbW8KZXhwb3J0IE9QU0NFTlRFUl9IT1NUPWRvYy10ZXN0LmVkZ2Uud2FsbGFyb29jb21tdW5pdHkubmluamEKZXhwb3J0IFBJUEVMSU5FX1VSTD1naGNyLmlvL3dhbGxhcm9vbGFicy9kb2Mtc2FtcGxlcy9waXBlbGluZXMvZWRnZS1pbmxpbmUtcmVwbGFjZW1lbnQtZGVtbzo0NDZhZWVkOS0yZDUyLTQ3YWUtOWU1Yy1mMmEwNWVmMGQ0ZDYKZXhwb3J0IEpPSU5fVE9LRU49b21pdHRlZF9mb3JfcmVwbGFjZQpleHBvcnQgT0NJX1JFR0lTVFJZPWdoY3IuaW8= \
    ghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/fitzroy-mini:v2024.1.0-main-4781

Note: Please set the PERSISTENT_VOLUME_DIR, OCI_USERNAME, and OCI_PASSWORD environment variables.
Helm Install Command
EdgeCommand
houseprice-edge-inline-replacement-demo
helm upgrade $HELM_INSTALL_NAME \
    oci://ghcr.io/wallaroolabs/doc-samples/charts/edge-inline-replacement-demo \
    --namespace $HELM_INSTALL_NAMESPACE \
    --version 0.0.1-446aeed9-2d52-47ae-9e5c-f2a05ef0d4d6 \
    --set ociRegistry.username=$OCI_USERNAME \
    --set ociRegistry.password=$OCI_PASSWORD \
    --set edgeBundle=ZXhwb3J0IEJVTkRMRV9WRVJTSU9OPTEKZXhwb3J0IENPTkZJR19DUFVTPTQKZXhwb3J0IEVER0VfTkFNRT1ob3VzZXByaWNlLWVkZ2UtaW5saW5lLXJlcGxhY2VtZW50LWRlbW8KZXhwb3J0IE9QU0NFTlRFUl9IT1NUPWRvYy10ZXN0LmVkZ2Uud2FsbGFyb29jb21tdW5pdHkubmluamEKZXhwb3J0IFBJUEVMSU5FX1VSTD1naGNyLmlvL3dhbGxhcm9vbGFicy9kb2Mtc2FtcGxlcy9waXBlbGluZXMvZWRnZS1pbmxpbmUtcmVwbGFjZW1lbnQtZGVtbzo0NDZhZWVkOS0yZDUyLTQ3YWUtOWU1Yy1mMmEwNWVmMGQ0ZDYKZXhwb3J0IEpPSU5fVE9LRU49b21pdHRlZF9mb3JfcmVwbGFjZQpleHBvcnQgT0NJX1JFR0lTVFJZPWdoY3IuaW8=

Note: Please set the PERSISTENT_VOLUME_DIR, HELM_INSTALL_NAME, HELM_INSTALL_NAMESPACE, OCI_USERNAME, and OCI_PASSWORD environment variables.

Note that in the Replaces section, updates are given for each edge location.

Replace Publish from New Pipeline

For this example, a new pipeline is created, The publish new_pub is then replaced by the publish from this new pipeline.

housing_model_challenger02 = (wl.upload_model("gbr-house-price-estimator", 
                                        './models/gbr_model.onnx', 
                                        framework=Framework.ONNX)
                                        .configure(tensor_fields=["tensor"])
                        )

new_pipeline = wl.build_pipeline("new-edge-inline-replacement")
# clear the steps if used before
new_pipeline.clear()

new_pipeline.add_model_step(housing_model_challenger02)
new_pipeline_pub = new_pipeline.publish(replaces=[new_pub])
new_pipeline_pub
Waiting for pipeline publish... It may take up to 600 sec.
Pipeline is publishing...... Published.
ID13
Pipeline Namenew-edge-inline-replacement
Pipeline Version00921571-3db7-4bf9-94dc-377aab558475
StatusPublished
Engine URLghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/fitzroy-mini:v2024.1.0-main-4781
Pipeline URLghcr.io/wallaroolabs/doc-samples/pipelines/new-edge-inline-replacement:00921571-3db7-4bf9-94dc-377aab558475
Helm Chart URLoci://ghcr.io/wallaroolabs/doc-samples/charts/new-edge-inline-replacement
Helm Chart Referenceghcr.io/wallaroolabs/doc-samples/charts@sha256:fe9a2d04f09723d5f09d0bc83d4b02367c91bdfc442007a9881e64b2f6eccfad
Helm Chart Version0.0.1-00921571-3db7-4bf9-94dc-377aab558475
Engine Config{'engine': {'resources': {'limits': {'cpu': 4.0, 'memory': '3Gi'}, 'requests': {'cpu': 4.0, 'memory': '3Gi'}, 'accel': 'none', 'arch': 'x86', 'gpu': False}}, 'engineAux': {'autoscale': {'type': 'none'}, 'images': None}, 'enginelb': {'resources': {'limits': {'cpu': 1.0, 'memory': '512Mi'}, 'requests': {'cpu': 0.2, 'memory': '512Mi'}, 'accel': 'none', 'arch': 'x86', 'gpu': False}}}
User Images[]
Created Byjohn.hummel@wallaroo.ai
Created At2024-03-26 19:16:41.564433+00:00
Updated At2024-03-26 19:16:41.564433+00:00
ReplacesPublish 12, Pipeline "new-edge-replacement", Version 31
Docker Run Command
EdgeCommand
houseprice-edge-inline-replacement-demo
docker run -v $PERSISTENT_VOLUME_DIR:/persist \
    -e OCI_USERNAME=$OCI_USERNAME \
    -e OCI_PASSWORD=$OCI_PASSWORD \
    -e PIPELINE_URL=ghcr.io/wallaroolabs/doc-samples/pipelines/new-edge-inline-replacement:00921571-3db7-4bf9-94dc-377aab558475\
    -e EDGE_BUNDLE=ZXhwb3J0IEJVTkRMRV9WRVJTSU9OPTEKZXhwb3J0IENPTkZJR19DUFVTPTQKZXhwb3J0IEVER0VfTkFNRT1ob3VzZXByaWNlLWVkZ2UtaW5saW5lLXJlcGxhY2VtZW50LWRlbW8KZXhwb3J0IE9QU0NFTlRFUl9IT1NUPWRvYy10ZXN0LmVkZ2Uud2FsbGFyb29jb21tdW5pdHkubmluamEKZXhwb3J0IFBJUEVMSU5FX1VSTD1naGNyLmlvL3dhbGxhcm9vbGFicy9kb2Mtc2FtcGxlcy9waXBlbGluZXMvbmV3LWVkZ2UtaW5saW5lLXJlcGxhY2VtZW50OjAwOTIxNTcxLTNkYjctNGJmOS05NGRjLTM3N2FhYjU1ODQ3NQpleHBvcnQgSk9JTl9UT0tFTj1vbWl0dGVkX2Zvcl9yZXBsYWNlCmV4cG9ydCBPQ0lfUkVHSVNUUlk9Z2hjci5pbw== \
    ghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/fitzroy-mini:v2024.1.0-main-4781

Note: Please set the PERSISTENT_VOLUME_DIR, OCI_USERNAME, and OCI_PASSWORD environment variables.
Helm Install Command
EdgeCommand
houseprice-edge-inline-replacement-demo
helm upgrade $HELM_INSTALL_NAME \
    oci://ghcr.io/wallaroolabs/doc-samples/charts/new-edge-inline-replacement \
    --namespace $HELM_INSTALL_NAMESPACE \
    --version 0.0.1-00921571-3db7-4bf9-94dc-377aab558475 \
    --set ociRegistry.username=$OCI_USERNAME \
    --set ociRegistry.password=$OCI_PASSWORD \
    --set edgeBundle=ZXhwb3J0IEJVTkRMRV9WRVJTSU9OPTEKZXhwb3J0IENPTkZJR19DUFVTPTQKZXhwb3J0IEVER0VfTkFNRT1ob3VzZXByaWNlLWVkZ2UtaW5saW5lLXJlcGxhY2VtZW50LWRlbW8KZXhwb3J0IE9QU0NFTlRFUl9IT1NUPWRvYy10ZXN0LmVkZ2Uud2FsbGFyb29jb21tdW5pdHkubmluamEKZXhwb3J0IFBJUEVMSU5FX1VSTD1naGNyLmlvL3dhbGxhcm9vbGFicy9kb2Mtc2FtcGxlcy9waXBlbGluZXMvbmV3LWVkZ2UtaW5saW5lLXJlcGxhY2VtZW50OjAwOTIxNTcxLTNkYjctNGJmOS05NGRjLTM3N2FhYjU1ODQ3NQpleHBvcnQgSk9JTl9UT0tFTj1vbWl0dGVkX2Zvcl9yZXBsYWNlCmV4cG9ydCBPQ0lfUkVHSVNUUlk9Z2hjci5pbw==

Note: Please set the PERSISTENT_VOLUME_DIR, HELM_INSTALL_NAME, HELM_INSTALL_NAMESPACE, OCI_USERNAME, and OCI_PASSWORD environment variables.

With the edge location replacement complete with the new pipeline, we list the edges to the original pipeline and the new one to show the edge location is transferred to the new pipeline.

mainpipeline.list_edges()

(no pipelines)

new_pipeline.list_edges()
IDNameTagsSPIFFE ID
4a9f77c4-c9bc-4e4b-b59e-b232b80ebce7houseprice-edge-inline-replacement-demo[]wallaroo.ai/ns/deployments/edge/4a9f77c4-c9bc-4e4b-b59e-b232b80ebce7

Edge Observability

Edge Observability allows edge deployments of Wallaroo Server to transmit inference results back to the Wallaroo Ops center and become part of the pipeline’s logs. This is valuable for data scientists and MLOps engineers to retrieve edge deployment logs for use in model observability, drift, and other use cases.

Before starting, the Edge Observability Service must be enabled in the Wallaroo Ops center. See the Edge Deployment Registry Guide for details on enabling the Wallaroo Edge Deployment service.

Wallaroo Server edge observability is enabled when a new edge location is added to the pipeline publish. Each location has its own EDGE_BUNDLE settings, a Base64 encoded set of instructions informing the edge deployed Wallaroo Server on how to communicate with Wallaroo Ops center.

Add Edge

Wallaroo Servers edge deployments are added to a Wallaroo pipeline’s publish with the wallaroo.pipeline_publish.add_edge(name: string, tags: List[string]) method. The name is the unique primary key for each edge added to the pipeline publish and must be unique.

Add Edge Parameters

wallaroo.pipeline_publish.add_edge(name: string, tags: List[string]) has the following parameters.

FieldTypeDescription
nameString (Required)The name of the edge location. This must be a unique value across all edges in the Wallaroo instance.
tagsList[String] (Optional)A list of optional tags.

Add Edge Returns

This returns a Publish Edge with the following fields:

FieldTypeDescription
idIntegerThe integer ID of the pipeline publish.
created_atDateTimeThe DateTime of the pipeline publish.
docker_run_variablesStringThe Docker variables in JSON entry with the key EDGE_BUNDLE as a base64 encoded value that includes the following: The BUNDLE_VERSION, EDGE_NAME, JOIN_TOKEN_, OPSCENTER_HOST, PIPELINE_URL, and WORKSPACE_ID. For example: {'EDGE_BUNDLE': 'abcde'}
engine_configStringThe Wallaroo wallaroo.deployment_config.DeploymentConfig for the pipeline.
pipeline_version_idIntegerThe integer identifier of the pipeline version published.
statusStringThe status of the publish. Published is a successful publish.
updated_atDateTimeThe DateTime when the pipeline publish was updated.
user_imagesList[String]User images used in the pipeline publish.
created_byStringThe UUID of the Wallaroo user that created the pipeline publish.
engine_urlStringThe URL for the published pipeline’s Wallaroo engine in the OCI registry.
errorStringAny errors logged.
helmStringThe helm chart, helm reference and helm version.
pipeline_urlStringThe URL for the published pipeline’s container in the OCI registry.
pipeline_version_nameStringThe UUID identifier of the pipeline version published.
additional_propertiesStringAny other properties.

Add Edge Example

The following example demonstrates creating a publish from a pipeline, then adding a new edge to the publish.

# create publish
xgb_pub=xgboost_pipeline_version.publish(deploy_config)
display(xgb_pub)
Waiting for pipeline publish... It may take up to 600 sec.
Pipeline is Publishing...Published.
ID1
Pipeline Versionf388c109-8d57-4ed2-9806-aa13f854576b
StatusPublished
Engine URLghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/standalone-mini:v2023.4.0-main-4079
Pipeline URLghcr.io/wallaroolabs/doc-samples/pipelines/edge-pipeline:f388c109-8d57-4ed2-9806-aa13f854576b
Helm Chart URLoci://ghcr.io/wallaroolabs/doc-samples/charts/edge-pipeline
Helm Chart Referenceghcr.io/wallaroolabs/doc-samples/charts@sha256:429aae187be641c22de5a333c737219a5ffaf908ac3673781cdf83f4ebbf7abc
Helm Chart Version0.0.1-f388c109-8d57-4ed2-9806-aa13f854576b
Engine Config{'engine': {'resources': {'limits': {'cpu': 1.0, 'memory': '512Mi'}, 'requests': {'cpu': 1.0, 'memory': '512Mi'}}}, 'engineAux': {'images': {}}, 'enginelb': {'resources': {'limits': {'cpu': 1.0, 'memory': '512Mi'}, 'requests': {'cpu': 1.0, 'memory': '512Mi'}}}}
User Images[]
Created Byjohn.hummel@wallaroo.ai
Created At2023-10-29 23:35:03.508703+00:00
Updated At2023-10-29 23:35:03.508703+00:00
Docker Run Variables{}
xgb_edge = xgb_pub.add_edge("xgb-ccfraud-edge-publish")
print(xgb_edge)
ID2
Pipeline Version60fb5c6e-db3e-497d-afc8-ccc149beba4a
StatusPublished
Engine URLghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/standalone-mini:v2023.4.0-main-4079
Pipeline URLghcr.io/wallaroolabs/doc-samples/pipelines/edge-pipeline:60fb5c6e-db3e-497d-afc8-ccc149beba4a
Helm Chart URLoci://ghcr.io/wallaroolabs/doc-samples/charts/edge-pipeline
Helm Chart Referenceghcr.io/wallaroolabs/doc-samples/charts@sha256:2de830d875ac8e60984c391091e5fdc981ad74e56925545c99b5e5b222c612bc
Helm Chart Version0.0.1-60fb5c6e-db3e-497d-afc8-ccc149beba4a
Engine Config{'engine': {'resources': {'limits': {'cpu': 1.0, 'memory': '512Mi'}, 'requests': {'cpu': 1.0, 'memory': '512Mi'}}}, 'engineAux': {'images': {}}, 'enginelb': {'resources': {'limits': {'cpu': 1.0, 'memory': '512Mi'}, 'requests': {'cpu': 1.0, 'memory': '512Mi'}}}}
User Images[]
Created Byjohn.hummel@wallaroo.ai
Created At2023-10-29 23:35:21.956532+00:00
Updated At2023-10-29 23:35:21.956532+00:00
Docker Run Variables{'EDGE_BUNDLE': 'abcde'}

Remove Edge

Edges are removed with the wallaroo.pipeline_publish.remove_edge(name: string)

Remove Edge Parameters

wallaroo.pipeline_publish.remove_edge(name: string) has the following parameters.

FieldTypeDescription
nameString (Required)The name of the edge location being removed.

Remove Edge Returns

Null

Remove Edge Example

This example will add two edges to a pipeline publish, list the edges for the pipeline, then remove one of the edges.

edge_01_name = f'edge-ccfraud-observability{random_suffix}'
edge01 = pub.add_edge(edge_01_name)

edge_02_name = f'edge-ccfraud-observability-02{random_suffix}'
edge02 = pub.add_edge(edge_02_name)

pipeline.list_edges()
IDNameTagsPipeline VersionSPIFFE ID
898bb58c-77c2-4164-b6cc-f004dc39e125edge-ccfraud-observabilityymgy[]6wallaroo.ai/ns/deployments/edge/898bb58c-77c2-4164-b6cc-f004dc39e125
1f35731a-f4f6-4cd0-a23a-c4a326b73277edge-ccfraud-observability-02ymgy[]6wallaroo.ai/ns/deployments/edge/1f35731a-f4f6-4cd0-a23a-c4a326b73277
sample = pub.remove_edge(edge_02_name)
display(sample)
IDNameTagsPipeline VersionSPIFFE ID
898bb58c-77c2-4164-b6cc-f004dc39e125edge-ccfraud-observabilityymgy[]6wallaroo.ai/ns/deployments/edge/898bb58c-77c2-4164-b6cc-f004dc39e125

List Edges

The method wallaroo.pipeline.list_edges() displays any edges added to a pipeline’s publishes.

List Edges Parameters

None

List Edges Returns

The following fields are returned from a List of edges.

ParameterTypeDescription
cpusFloatThe number of cpus assigned as part of the pipeline configuration.
idStringThe identifier of the edge in UUID format.
memoryStringThe memory assigned as part of the pipeline configuration in Kubernetes memory format.
nameStringThe assigned name for the edge. Edge names are used as the primary key.
tagsList[String]A list of tags assigned to the edge.
pipeline_version_idIntegerThe pipeline version numerical idenfier.
spiffe_idStringThe deployment edge identifier used to for edge communications.
additional_propertiesDictAny additional properties.

List Edges Example

edge_01_name = f'edge-ccfraud-observability{random_suffix}'
edge01 = pub.add_edge(edge_01_name)

edge_02_name = f'edge-ccfraud-observability-02{random_suffix}'
edge02 = pub.add_edge(edge_02_name)

pipeline.list_edges()
IDNameTagsPipeline VersionSPIFFE ID
898bb58c-77c2-4164-b6cc-f004dc39e125edge-ccfraud-observabilityymgy[]6wallaroo.ai/ns/deployments/edge/898bb58c-77c2-4164-b6cc-f004dc39e125
1f35731a-f4f6-4cd0-a23a-c4a326b73277edge-ccfraud-observability-02ymgy[]6wallaroo.ai/ns/deployments/edge/1f35731a-f4f6-4cd0-a23a-c4a326b73277

Edge Bundle Token TTL

When an edge is added to a pipeline publish, the field docker_run_variables contains a JSON value for edge devices to connect to the Wallaroo Ops instance. The settings are stored in the key EDGE_BUNDLE as a base64 encoded value that include the following:

  • BUNDLE_VERSION: The current version of the bundled Wallaroo pipeline.
  • EDGE_NAME: The edge name as defined when created and added to the pipeline publish.
  • JOIN_TOKEN_: The one time authentication token for authenticating to the Wallaroo Ops instance.
  • OPSCENTER_HOST: The hostname of the Wallaroo Ops edge service. See Edge Deployment Registry Guide for full details on enabling pipeline publishing and edge observability to Wallaroo.
  • PIPELINE_URL
  • WORKSPACE_ID.

For example:

{'edgeBundle': 'ZXhwb3J0IEJVTkRMRV9WRVJTSU9OPTEKZXhwb3J0IEVER0VfTkFNRT14Z2ItY2NmcmF1ZC1lZGdlLXRlc3QKZXhwb3J0IEpPSU5fVE9LRU49MzE0OGFkYTUtMjg1YS00ZmNhLWIzYjgtYjUwYTQ4ZDc1MTFiCmV4cG9ydCBPUFNDRU5URVJfSE9TVD1kb2MtdGVzdC5lZGdlLndhbGxhcm9vY29tbXVuaXR5Lm5pbmphCmV4cG9ydCBQSVBFTElORV9VUkw9Z2hjci5pby93YWxsYXJvb2xhYnMvZG9jLXNhbXBsZXMvcGlwZWxpbmVzL2VkZ2UtcGlwZWxpbmU6ZjM4OGMxMDktOGQ1Ny00ZWQyLTk4MDYtYWExM2Y4NTQ1NzZiCmV4cG9ydCBXT1JLU1BBQ0VfSUQ9NQ=='}
base64 -D
ZXhwb3J0IEJVTkRMRV9WRVJTSU9OPTEKZXhwb3J0IEVER0VfTkFNRT14Z2ItY2NmcmF1ZC1lZGdlLXRlc3QKZXhwb3J0IEpPSU5fVE9LRU49MzE0OGFkYTUtMjg1YS00ZmNhLWIzYjgtYjUwYTQ4ZDc1MTFiCmV4cG9ydCBPUFNDRU5URVJfSE9TVD1kb2MtdGVzdC5lZGdlLndhbGxhcm9vY29tbXVuaXR5Lm5pbmphCmV4cG9ydCBQSVBFTElORV9VUkw9Z2hjci5pby93YWxsYXJvb2xhYnMvZG9jLXNhbXBsZXMvcGlwZWxpbmVzL2VkZ2UtcGlwZWxpbmU6ZjM4OGMxMDktOGQ1Ny00ZWQyLTk4MDYtYWExM2Y4NTQ1NzZiCmV4cG9ydCBXT1JLU1BBQ0VfSUQ9NQ==^D
export BUNDLE_VERSION=1
export EDGE_NAME=xgb-ccfraud-edge-test
export JOIN_TOKEN=3148ada5-285a-4fca-b3b8-b50a48d7511b
export OPSCENTER_HOST=doc-test.edge.wallaroocommunity.ninja
export PIPELINE_URL=ghcr.io/wallaroolabs/doc-samples/pipelines/edge-pipeline:f388c109-8d57-4ed2-9806-aa13f854576b
export WORKSPACE_ID=5

The JOIN_TOKEN is a one time access token. Once used, a JOIN_TOKEN expires. The authentication session data is stored in persistent volumes. Persistent volumes must be specified for docker and docker compose based deployments of Wallaroo pipelines; helm based deployments automatically provide persistent volumes to store authentication credentials.

The JOIN_TOKEN has the following time to live (TTL) parameters.

  • Once created, the JOIN_TOKEN is valid for 24 hours. After it expires the edge will not be allowed to contact the OpsCenter the first time and a new edge bundle will have to be created.
  • After an Edge joins to Wallaroo Ops for the first time with persistent storage, the edge must contact the Wallaroo Ops instance at least once every 7 days.
    • If this period is exceeded, the authentication credentials will expire and a new edge bundle must be created with a new and valid JOIN_TOKEN.

Wallaroo edges require unique names. To create a new edge bundle with the same name:

  • Use the Remove Edge to remove the edge by name.
  • Use Add Edge to add the edge with the same name. A new EDGE_BUNDLE is generated with a new JOIN_TOKEN.

DevOps - Pipeline Edge Deployment

Once a pipeline is deployed to the Edge Registry service, it can be deployed in environments such as Docker, Kubernetes, or similar container running services by a DevOps engineer.

Docker Deployment

First, the DevOps engineer must authenticate to the same OCI Registry service used for the Wallaroo Edge Deployment registry.

For more details, check with the documentation on your artifact service. The following are provided for the three major cloud services:

For the deployment, the engine URL is specified with the following environmental variables:

  • DEBUG (true|false): Whether to include debug output.
  • OCI_REGISTRY: The URL of the registry service.
  • CONFIG_CPUS: The number of CPUs to use.
  • OCI_USERNAME: The edge registry username.
  • OCI_PASSWORD: The edge registry password or token.
  • PIPELINE_URL: The published pipeline URL.
  • EDGE_BUNDLE (Optional): The base64 encoded edge token and other values to connect to the Wallaroo Ops instance. This is used for edge management and transmitting inference results for observability. IMPORTANT NOTE: The token for EDGE_BUNDLE is valid for one deployment. For subsequent deployments, generate a new edge location with its own EDGE_BUNDLE.
  • LOCAL_INFERENCE_STORAGE (Optional): Sets amount of storage to allocate for the edge deployments inference log storage capacity. This is in the format {size as number}{unit value}. The values are similar to the Kubernetes memory resource units format. If used, must be used with PLATEAU_PAGE_SIZE. The accepted unit values are:
    • Ki (for KiloBytes)
    • Mi (for MegaBytes)
    • Gi (for GigaBytes)
    • Ti (for TeraBytes)
  • PLATEAU_PAGE_SIZE (Optional): How many inference log rows to upload from the edge deployment at a time. Must be used with LOCAL_INFERENCE_STORAGE.

Docker Deployment Example

Using our sample environment, here’s sample deployment using Docker with a computer vision ML model, the same used in the Wallaroo Use Case Tutorials Computer Vision: Retail tutorials.

  1. Login through docker to confirm access to the registry service. First, docker login. For example, logging into the artifact registry with the token stored in the variable tok:

    cat $tok | docker login -u _json_key_base64 --password-stdin https://sample-registry.com
    
  2. Then deploy the Wallaroo published pipeline with an edge added to the pipeline publish through docker run.

    IMPORTANT NOTE: Edge deployments with Edge Observability enabled with the EDGE_BUNDLE option include an authentication token that only authenticates once. To store the token long term, include the persistent volume flag -v {path to storage} setting.

    Deployment with EDGE_BUNDLE for observability.

    docker run -p 8080:8080 \
    -v ./data:/persist \
    -e DEBUG=true \
    -e OCI_REGISTRY=$REGISTRYURL \
    -e EDGE_BUNDLE=ZXhwb3J0IEJVTkRMRV9WRVJTSU9OPTEKZXhwb3J0IEVER0VfTkFNRT1lZGdlLWNjZnJhdWQtb2JzZXJ2YWJpbGl0eXlhaWcKZXhwb3J0IEpPSU5fVE9LRU49MjZmYzFjYjgtMjUxMi00YmU3LTk0ZGUtNjQ2NGI1MGQ2MzhiCmV4cG9ydCBPUFNDRU5URVJfSE9TVD1kb2MtdGVzdC5lZGdlLndhbGxhcm9vY29tbXVuaXR5Lm5pbmphCmV4cG9ydCBQSVBFTElORV9VUkw9Z2hjci5pby93YWxsYXJvb2xhYnMvZG9jLXNhbXBsZXMvcGlwZWxpbmVzL2VkZ2Utb2JzZXJ2YWJpbGl0eS1waXBlbGluZTozYjQ5ZmJhOC05NGQ4LTRmY2EtYWVjYy1jNzUyNTdmZDE2YzYKZXhwb3J0IFdPUktTUEFDRV9JRD03 \
    -e CONFIG_CPUS=1 \
    -e OCI_USERNAME=$REGISTRYUSERNAME \
    -e OCI_PASSWORD=$REGISTRYPASSWORD \
    -e PIPELINE_URL=ghcr.io/wallaroolabs/doc-samples/pipelines/edge-observability-pipeline:3b49fba8-94d8-4fca-aecc-c75257fd16c6 \
    ghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/standalone-mini:v2023.4.0-main-4079
    

    Connection to the Wallaroo Ops instance from edge deployment with EDGE_BUNDLE is verified with the long entry Node attestation was successful.

    Deployment without observability.

    docker run -p 8080:8080 \
    -e DEBUG=true \
    -e OCI_REGISTRY=$REGISTRYURL \
    -e CONFIG_CPUS=1 \
    -e OCI_USERNAME=$REGISTRYUSERNAME \
    -e OCI_PASSWORD=$REGISTRYPASSWORD \
    -e PIPELINE_URL=ghcr.io/wallaroolabs/doc-samples/pipelines/edge-observability-pipeline:3b49fba8-94d8-4fca-aecc-c75257fd16c6 \
    ghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/standalo
    

Docker Compose Deployment

For users who prefer to use docker compose, the following sample compose.yaml file is used to launch the Wallaroo Edge pipeline. This is the same used in the Wallaroo Use Case Tutorials Computer Vision: Retail tutorials. The volumes tag is used to preserve the login session from the one-time token generated as part of the EDGE_BUNDLE.

EDGE_BUNDLE is only required when adding an edge to a Wallaroo publish for observability. The following is deployed without observability.

services:
  engine:
    image: {Your Engine URL}
    ports:
      - 8080:8080
    environment:
      PIPELINE_URL: {Your Pipeline URL}
      OCI_REGISTRY: {Your Edge Registry URL}
      OCI_USERNAME:  {Your Registry Username}
      OCI_PASSWORD: {Your Token or Password}
      CONFIG_CPUS: 4

The procedure is:

  1. Login through docker to confirm access to the registry service. First, docker login. For example, logging into the artifact registry with the token stored in the variable tok to the registry us-west1-docker.pkg.dev:

    cat $tok | docker login -u _json_key_base64 --password-stdin https://sample-registry.com
    
  2. Set up the compose.yaml file.

    IMPORTANT NOTE: Edge deployments with Edge Observability enabled with the EDGE_BUNDLE option include an authentication token that only authenticates once. To store the token long term, include the persistent volume with the volumes: tag.

    services:
    engine:
        image: sample-registry.com/engine:v2023.3.0-main-3707
        ports:
            - 8080:8080
        volumes:
            - ./data:/persist
    environment:
        PIPELINE_URL: sample-registry.com/pipelines/edge-cv-retail:bf70eaf7-8c11-4b46-b751-916a43b1a555
        EDGE_BUNDLE: ZXhwb3J0IEJVTkRMRV9WRVJTSU9OPTEKZXhwb3J0IEVER0VfTkFNRT1lZGdlLWNjZnJhdWQtb2JzZXJ2YWJpbGl0eXlhaWcKZXhwb3J0IEpPSU5fVE9LRU49MjZmYzFjYjgtMjUxMi00YmU3LTk0ZGUtNjQ2NGI1MGQ2MzhiCmV4cG9ydCBPUFNDRU5URVJfSE9TVD1kb2MtdGVzdC5lZGdlLndhbGxhcm9vY29tbXVuaXR5Lm5pbmphCmV4cG9ydCBQSVBFTElORV9VUkw9Z2hjci5pby93YWxsYXJvb2xhYnMvZG9jLXNhbXBsZXMvcGlwZWxpbmVzL2VkZ2Utb2JzZXJ2YWJpbGl0eS1waXBlbGluZTozYjQ5ZmJhOC05NGQ4LTRmY2EtYWVjYy1jNzUyNTdmZDE2YzYKZXhwb3J0IFdPUktTUEFDRV9JRD03
        OCI_REGISTRY: sample-registry.com
        OCI_USERNAME:  _json_key_base64
        OCI_PASSWORD: abc123
        CONFIG_CPUS: 4
    
  3. Then deploy with docker compose up.

Docker Compose Deployment Example

The deployment and undeployment is then just a simple docker compose up and docker compose down. The following shows an example of deploying the Wallaroo edge pipeline using docker compose.

docker compose up
[+] Running 1/1
 ✔ Container cv_data-engine-1  Recreated                                                                                                                                                                 0.5s
Attaching to cv_data-engine-1
cv_data-engine-1  | Wallaroo Engine - Standalone mode
cv_data-engine-1  | Login Succeeded
cv_data-engine-1  | Fetching manifest and config for pipeline: sample-registry.com/pipelines/edge-cv-retail:bf70eaf7-8c11-4b46-b751-916a43b1a555
cv_data-engine-1  | Fetching model layers
cv_data-engine-1  | digest: sha256:c6c8869645962e7711132a7e17aced2ac0f60dcdc2c7faa79b2de73847a87984
cv_data-engine-1  |   filename: c6c8869645962e7711132a7e17aced2ac0f60dcdc2c7faa79b2de73847a87984
cv_data-engine-1  |   name: resnet-50
cv_data-engine-1  |   type: model
cv_data-engine-1  |   runtime: onnx
cv_data-engine-1  |   version: 693e19b5-0dc7-4afb-9922-e3f7feefe66d
cv_data-engine-1  |
cv_data-engine-1  | Fetched
cv_data-engine-1  | Starting engine
cv_data-engine-1  | Looking for preexisting `yaml` files in //modelconfigs
cv_data-engine-1  | Looking for preexisting `yaml` files in //pipelines

Helm Deployment

Published pipelines can be deployed through the use of helm charts.

Helm deployments take up to two steps - the first step is in retrieving the required values.yaml and making updates to override.

IMPORTANT NOTE: Edge deployments with Edge Observability enabled with the EDGE_BUNDLE option include an authentication token that only authenticates once. Helm chart installations automatically add a persistent volume during deployment to store the authentication session data for future deployments.

  1. Login to the registry service with helm registry login. For example, if the token is stored in the variable tok:

    helm registry login sample-registry.com --username _json_key_base64 --password $tok
    
  2. Pull the helm charts from the published pipeline. The two fields are the Helm Chart URL and the Helm Chart version to specify the OCI . This typically takes the format of:

    helm pull oci://{published.helm_chart_url} --version {published.helm_chart_version}
    
  3. Extract the tgz file and copy the values.yaml and copy the values used to edit engine allocations, etc. The following are required for the deployment to run:

    ociRegistry:
        registry: {your registry service}
        username:  {registry username here}
        password: {registry token here}
    

    For Wallaroo Server deployments with edge location set, the values include edgeBundle as generated when the edge was added to the pipeline publish.

    ociRegistry:
        registry: {your registry service}
        username:  {registry username here}
        password: {registry token here}
    edgeBundle: abcdefg
    

Store this into another file, suc as local-values.yaml.

  1. Create the namespace to deploy the pipeline to. For example, the namespace wallaroo-edge-pipeline would be:

    kubectl create -n wallaroo-edge-pipeline
    
  2. Deploy the helm installation with helm install through one of the following options:

    1. Specify the tgz file that was downloaded and the local values file. For example:

      helm install --namespace {namespace} --values {local values file} {helm install name} {tgz path}
      
    2. Specify the expended directory from the downloaded tgz file.

      helm install --namespace {namespace} --values {local values file} {helm install name} {helm directory path}
      
    3. Specify the Helm Pipeline Helm Chart and the Pipeline Helm Version.

      helm install --namespace {namespace} --values {local values file} {helm install name} oci://{published.helm_chart_url} --version {published.helm_chart_version}
      
  3. Once deployed, the DevOps engineer will have to forward the appropriate ports to the svc/engine-svc service in the specific pipeline. For example, using kubectl port-forward to the namespace ccfraud that would be:

    kubectl port-forward svc/engine-svc -n ccfraud01 8080 --address 0.0.0.0`