Model Deploy

Table of Contents

Organizations manage model deployments through pipelines.

Pipelines represent how data is submitted to your uploaded Machine Learning (ML) models. Pipelines allow you to:

• Submit information through an uploaded file or through the Pipeline’s Deployment URL.
• Have the Pipeline submit the information to one or more models in sequence.
• Once complete, output the result from the model(s).

How to Manage Pipelines via the Wallaroo Dashboard

The Wallaroo Dashboard provides users methods to create, deploy, and other pipeline operations through the Wallaroo Dashboard user interface.

How to Create a Pipeline Using the Wallaroo Dashboard

To create a pipeline:

1. From the Wallaroo Dashboard, set the current workspace from the top left dropdown list.

2. Select View Pipelines from the pipeline’s row.

3. From the upper right hand corner, select Create Pipeline.

   

4. Enter the following:

   1. Pipeline Name: The name of the new pipeline. Pipeline names should be unique across the Wallaroo instance.
   2. Add Pipeline Step: Select the models to be used as the pipeline steps.

   

   When finished, select Next.

5. Review the name of the pipeline and the steps. If any adjustments need to be made, select either Back to rename the pipeline or Add Step(s) to change the pipeline’s steps.

   

6. When finished, select Build to create the pipeline in this workspace. The pipeline will be built and be ready for deployment within a minute.

   

How to Deploy and Undeploy a Pipeline using the Wallaroo Dashboard

Deployed pipelines create new namespaces in the Kubernetes environment where the Wallaroo instance is deployed, and allocate resources from the Kubernetes environment to run the pipeline and its steps.

To deploy a pipeline:

1. From the Wallaroo Dashboard, set the current workspace from the top left dropdown list.

2. Select View Pipelines from the pipeline’s row.

3. Select the pipeline to deploy.

4. From the right navigation panel, select Deploy.

   

5. A popup module will request verification to deploy the pipeline. Select Deploy again to deploy the pipeline.

Undeploying a pipeline returns resources back to the Kubernetes environment and removes the namespaces created when the pipeline was deployed.

To undeploy a pipeline:

1. From the Wallaroo Dashboard, set the current workspace from the top left dropdown list.

2. Select View Pipelines from the pipeline’s row.

3. Select the pipeline to deploy.

4. From the right navigation panel, select Undeploy.

   

5. A popup module will request verification to undeploy the pipeline. Select Undeploy again to undeploy the pipeline.

How to Update Deployment Configuration

Pipeline deployment configurations are updated through the Wallaroo Dashboard for previously deployed pipelines and currently undeployed pipelines. For more details, see Deployment Configuration via the Wallaroo Dashboard.

How to View a Pipeline Details and Metrics using the Wallaroo Dashboard

To view a pipeline’s details:

1. From the Wallaroo Dashboard, set the current workspace from the top left dropdown list.
2. Select View Pipelines from the pipeline’s row.
3. To view details on the pipeline, select the name of the pipeline.
4. A list of the pipeline’s details will be displayed.

To view a pipeline’s metrics:

1. From the Wallaroo Dashboard, set the current workspace from the top left dropdown list.
2. Select View Pipelines from the pipeline’s row.
3. To view details on the pipeline, select the name of the pipeline.
4. A list of the pipeline’s details will be displayed.
5. Select Metrics to view the following information. From here you can select the time period to display metrics from through the drop down to display the following:
   1. Requests per second
   2. Cluster inference rate
   3. Inference latency
6. The Audit Log and Anomaly Log are available to view further details of the pipeline’s activities.

Viewing Pipeline Details through the Wallaroo Dashboard

The following is available from the Pipeline Details page:

• The name of the pipeline.
• The pipeline ID: This is in UUID format.
• Pipeline steps: The steps and the models in each pipeline step.
• Version History: how the pipeline has been updated over time.

How to Manage Pipelines via the Wallaroo SDK

Pipelines are the method of taking submitting data and processing that data through the models. Each pipeline can have one or more steps that submit the data from the previous step to the next one. Information can be submitted to a pipeline as a file, or through the pipeline’s URL.

The following is subset of pipeline management. For full details, see Wallaroo Pipeline Management.

A pipeline’s metrics can be viewed through the Wallaroo Dashboard Pipeline Details and Metrics page.

Pipeline Naming Requirements

Pipeline names map onto Kubernetes objects, and must be DNS compliant. Pipeline names must be ASCII alpha-numeric characters or dash (-) only. . and _ are not allowed.

Create a Pipeline

New pipelines are created in the current workspace.

To create a new pipeline, use the Wallaroo Client build_pipeline("{Pipeline Name}") command.

The following example creates a new pipeline imdb-pipeline through a Wallaroo Client connection wl:

imdb_pipeline = wl.build_pipeline("imdb-pipeline")

imdb_pipeline.status()
{'status': 'Pipeline imdb-pipeline is not deployed'}

Get Pipeline

Pipelines are retrieved via the method wallaroo.client.get_pipeline. This retrieves the most recent version of the specified pipeline in the current workspace that that matches the requested parameters.

For more details on workspaces, see How to Set the Current Workspace.

Get Pipeline Parameters

wallaroo.client.get_pipeline takes the following parameters.

Get Pipeline Returns

wallaroo.client.get_pipeline returns wallaroo.pipeline_version.PipelineVersion for the specified pipeline and version in the current workspace. This includes the following fields:

If the pipeline or pipeline version do not match the requested parameters, an error is returned.

Get Pipeline Examples

Get Pipeline by Name Example

The following demonstrates retrieving the most recent pipeline version of the requested pipeline in the current workspace.

wl.get_pipeline(name=pipeline_name)

Get Pipeline By Name Error Example

The following demonstrates requesting a pipeline when no pipeline matching the name exists in the current workspace.

wl.get_pipeline(name="no-such-pipeline")

---------------------------------------------------------------------------

Exception                                 Traceback (most recent call last)

/tmp/ipykernel_208/1835144440.py in <module>
----> 1 wl.get_pipeline(name="no-such-pipeline")


~/.local/lib/python3.9/site-packages/wallaroo/client.py in get_pipeline(self, name, version)
   1357         )
   1358         if pipeline is None:
-> 1359             raise Exception(f"Pipeline {name} not found in this workspace.")
   1360         if version is not None:
   1361             pipeline_version = next(

Exception: Pipeline no-such-pipeline not found in this workspace.

Get Specific Pipeline Version Example

wl.get_pipeline(name=pipeline_name, version="62e0236f-e040-40c1-8666-7b703fd5d2f0")

Get Specific Pipeline Version Error Example

The following demonstrates requesting a pipeline version when no pipeline version matching the requested version parameter exists in the current workspace and pipeline.

pipeline_name="helper-demo-pipeline-1"

wl.get_pipeline(name=pipeline_name, version="no-such-version")

---------------------------------------------------------------------------

Exception                                 Traceback (most recent call last)

/tmp/ipykernel_208/2171664880.py in <module>
----> 1 wl.get_pipeline(name=pipeline_name, version="no-such-version")


~/.local/lib/python3.9/site-packages/wallaroo/client.py in get_pipeline(self, name, version)
   1366                 return pipeline
   1367             else:
-> 1368                 raise Exception(
   1369                     f"Pipeline version {version} not found in this workspace."
   1370                 )

Exception: Pipeline version no-such-version not found in this workspace.

List All Pipelines

The Wallaroo Client method list_pipelines() lists all pipelines in Wallaroo in workspaces the SDK user is a member of. The list is filterable by the workspace id or the workspace name, listed in reverse chronological order.

Admin users have unrestricted access to all workspaces. For more details, see Wallaroo Enterprise User Management.

List All Pipelines Parameters

List All Pipelines Returns

The following fields are returned from the list_pipeline method.

List All Pipelines Example

The following retrieves all pipelines in workspaces the SDK user is a member of.

wl.list_pipelines()

Filter by workspace id:

wl.list_pipelines(workspace_id=25)

Filter by workspace name:

wl.list_pipelines(workspace_name=workspace_name)

Pipeline Steps

Once a pipeline has been created, or during its creation process, a pipeline step can be added. The pipeline step refers to the model that will perform an inference off of the data submitted to it. Each time a step is added, it is added to the pipeline’s models array.

Pipeline steps are not saved until the pipeline is deployed. Until then, pipeline steps are stored in local memory as a potential pipeline configuration until the pipeline is deployed.

Add a Step to a Pipeline

A pipeline step is added through the pipeline add_model_step({Model}) command.

In the following example, two models uploaded to the workspace are added as pipeline step:

imdb_pipeline.add_model_step(embedder)
imdb_pipeline.add_model_step(smodel)

imdb_pipeline.status()

{'name': 'imdb-pipeline', 'create_time': datetime.datetime(2022, 3, 30, 21, 21, 31, 127756, tzinfo=tzutc()), 'definition': "[{'ModelInference': {'models': [{'name': 'embedder-o', 'version': '1c16d21d-fe4c-4081-98bc-65fefa465f7d', 'sha': 'd083fd87fa84451904f71ab8b9adfa88580beb92ca77c046800f79780a20b7e4'}]}}, {'ModelInference': {'models': [{'name': 'smodel-o', 'version': '8d311ba3-c336-48d3-99cd-85d95baa6f19', 'sha': '3473ea8700fbf1a1a8bfb112554a0dde8aab36758030dcde94a9357a83fd5650'}]}}]"}

Clear All Pipeline Steps

The Pipeline clear() method removes all pipeline steps from a pipeline. Note that pipeline steps are not saved until the pipeline is deployed.

Deploy a Pipeline

When a pipeline step is added or removed, the pipeline must be deployed through the pipeline wallaroo.pipeline.Pipeline.deploy(deployment_config, wait_for_status). This allocates resources to the pipeline from the Kubernetes environment and make it available to submit information to perform inferences. For full details on pipeline deployment configurations, see Wallaroo SDK Essentials Guide: Pipeline Deployment Configuration.

Deploy a Pipeline Parameters

The method wallaroo.pipeline.Pipeline.deploy takes the following parameters.

Deploy Pipelines Asynchronously

By default, wallaroo.pipeline.Pipeline.deploy sets wait_for_status=True. In this situation, the Python code pauses and display the deployment process until either:

• The pipeline successfully deploys, at which point the Python code continues.
• The pipeline deployment reaches its timeout.
• The pipeline errors out for one or more reasons. In these situations:
  • The Python code is stopped if the exception is not caught.
  • An error is displayed.

If wait_for_status=False, then Python code will not wait for the success or failure of the pipeline to deploy, but proceed to the next instruction without pause.

In these situations, before performing an inference request, use pipeline status to verify that the pipeline status is Running.

For example, the following example:

1. Deploys three pipelines with wait_for_status=False.
2. Before performing an inference request, verify the pipeline sample_pipeline_1 status is status='Running'.

for pipe, deploy_config in [(sample_pipeline_1, deployment_config_1), (sample_pipeline_2, deployment_config_2), (sample_pipeline_2, deployment_config_3)]:
    pipe.deploy(deployment_config=deploy_config, wait_for_status=False)

# perform other code

# check the pipeline status before performing an inference

if sample_pipeline_1.status()['status'] == 'Running':
   sample_pipeline_1.infer(sample_data)

Deploy a Pipeline Deployment Defaults

Deployment configurations default to the following^*.

*: For Kubernetes limits and requests.
**: Resources are always allocated for the Wallaroo Native Runtime engine even if there are no Wallaroo Native Runtimes included in the deployment, so it is recommended to decrease these resources when pipelines use Containerized Runtimes.
***: Resources for Wallaroo Containerized Runtimes only apply with a Wallaroo Containerized Runtime is part of the deployment.

Deploy a Pipeline with a New Deployment Configuration

Pipelines do not need to be undeployed to deploy new pipeline versions or pipeline deployment configurations. For example, the following pipeline is deployed, new pipeline steps are set, and the pipeline deploy command is issues again. This creates a new version of the pipeline and updates the deployed pipeline with the new configuration.

# clear all steps
pipeline.clear()
# set modelA as the step
pipeline.add_model_step(modelA)
# deploy the pipeline - the version is saved and the resources allocated to the pipeline
pipeline.deploy()
# clear the steps - this configuration is only stored in the local SDK session until the deploy or create_version command is given
pipeline.clear()
# set modelB as the step
pipeline.add_model_step(modelB)
# deploy the pipeline - the pipeline configuration is saved and the pipeline deployment updated without significant downtime
pipeline.deploy()

Model Deployment Architecture Inheritance

Deployment configurations inherit the model’s architecture setting. This is set during model upload by specifying the arch parameter. By default, models uploaded to Wallaroo default to the x86 architecture.

The following model operations inherit the model’s architecture setting.

• Model Deployment: Model deployment and Model Deployment Deployment Configuration inherit the the model’s architecture. No specification of the architecture is required for model deployment.
• Pipeline Publishing: The Wallaroo engine set when a pipeline is containerized and published to an Open Container Initiative (OCI) Registry inherits the model’s architecture setting.

The following example shows uploading a model set with the architecture set to ARM, and how the deployment inherits that architecture without additional deployment configuration changes. For this example, an ONNX model is uploaded.

import wallaroo

housing_model_control_arm = (wl.upload_model(model_name_arm, 
                                        model_file_name, 
                                        framework=Framework.ONNX,
                                        arch=wallaroo.engine_config.Architecture.ARM)
                                        .configure(tensor_fields=["tensor"])
                        )

display(housing_model_control_arm)

Note that the deployment configuration settings, no architecture is specified. When pipeline_arm is displayed, we see the arch setting inherited the model’s arch setting.

pipeline_arm = wl.build_pipeline(arm_pipeline_name)

# set the model step with the ARM targeted model
pipeline_arm.add_model_step(housing_model_control_arm)

#minimum deployment config for this model
deploy_config = wallaroo.DeploymentConfigBuilder().replica_count(1).cpus(1).memory("1Gi").build()

pipeline_arm.deploy(deployment_config = deploy_config)

    Waiting for deployment - this will take up to 45s .......... ok

display(pipeline_arm)

Deploy Current Pipeline Version

By default, deploying a Wallaroo pipeline will deploy the most current version. For example:

sample_pipeline = wl.build_pipeline("test-pipeline")
sample_pipeline.add_model_step(model)
sample_pipeline.deploy()
sample_pipeline.status()

{'status': 'Running',
 'details': None,
 'engines': [{'ip': '10.12.1.65',
   'name': 'engine-778b65459-f9mt5',
   'status': 'Running',
   'reason': None,
   'pipeline_statuses': {'pipelines': [{'id': 'imdb-pipeline',
      'status': 'Running'}]},
   'model_statuses': {'models': [{'name': 'embedder-o',
      'version': '1c16d21d-fe4c-4081-98bc-65fefa465f7d',
      'sha': 'd083fd87fa84451904f71ab8b9adfa88580beb92ca77c046800f79780a20b7e4',
      'status': 'Running'},
     {'name': 'smodel-o',
      'version': '8d311ba3-c336-48d3-99cd-85d95baa6f19',
      'sha': '3473ea8700fbf1a1a8bfb112554a0dde8aab36758030dcde94a9357a83fd5650',
      'status': 'Running'}]}}],
 'engine_lbs': [{'ip': '10.12.1.66',
   'name': 'engine-lb-85846c64f8-ggg2t',
   'status': 'Running',
   'reason': None}]}

Deploy Previous Pipeline Version

Pipeline versions are deployed with the method wallaroo.pipeline_variant.deploy(deployment_name, model_configs, config: Optional[wallaroo.deployment_config.DeploymentConfig]). Note that the deployment_name and model_configs are required. The model_configs are retrieved with the wallaroo.pipeline_variant.model_configs() method.

The following demonstrates retrieving a previous version of a pipeline, deploying it, and retrieving the deployment status.

pipeline_version = pipeline.versions()[7]

display(pipeline_version.name())
display(pipeline_version)
pipeline_version.deploy("houseprice-estimator", pipeline_version.model_configs())
display(pipeline.status())

{'status': 'Running',
 'details': [],
 'engines': [{'ip': '10.244.3.211',
   'name': 'engine-578dc7cdcf-qkx5n',
   'status': 'Running',
   'reason': None,
   'details': [],
   'pipeline_statuses': {'pipelines': [{'id': 'houseprice-estimator',
      'status': 'Running'}]},
   'model_statuses': {'models': [{'name': 'house-price-rf-model',
      'version': '616c2306-bf93-417b-9656-37bee6f14379',
      'sha': 'e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6',
      'status': 'Running'}]}}],
 'engine_lbs': [{'ip': '10.244.4.243',
   'name': 'engine-lb-584f54c899-2rtvg',
   'status': 'Running',
   'reason': None,
   'details': []}],
 'sidekicks': []}

Pipeline Status

Once complete, the pipeline status() command will show 'status':'Running'.

Pipeline deployments can be modified to enable auto-scaling to allow pipelines to allocate more or fewer resources based on need by setting the pipeline’s This will then be applied to the deployment of the pipelineccfraudPipelineby specifying it'sdeployment_config` optional parameter. If this optional parameter is not passed, then the deployment will defer to default values. For more information, see Manage Pipeline Deployment Configuration.

In the following example, the pipeline imdb-pipeline that contains two steps will be deployed with default deployment configuration:

imdb_pipeline.status

<bound method Pipeline.status of {'name': 'imdb-pipeline', 'create_time': datetime.datetime(2022, 3, 30, 21, 21, 31, 127756, tzinfo=tzutc()), 'definition': "[{'ModelInference': {'models': [{'name': 'embedder-o', 'version': '1c16d21d-fe4c-4081-98bc-65fefa465f7d', 'sha': 'd083fd87fa84451904f71ab8b9adfa88580beb92ca77c046800f79780a20b7e4'}]}}, {'ModelInference': {'models': [{'name': 'smodel-o', 'version': '8d311ba3-c336-48d3-99cd-85d95baa6f19', 'sha': '3473ea8700fbf1a1a8bfb112554a0dde8aab36758030dcde94a9357a83fd5650'}]}}]"}>

imdb_pipeline.deploy()
Waiting for deployment - this will take up to 45s ...... ok

imdb_pipeline.status()

{'status': 'Running',
 'details': None,
 'engines': [{'ip': '10.12.1.65',
   'name': 'engine-778b65459-f9mt5',
   'status': 'Running',
   'reason': None,
   'pipeline_statuses': {'pipelines': [{'id': 'imdb-pipeline',
      'status': 'Running'}]},
   'model_statuses': {'models': [{'name': 'embedder-o',
      'version': '1c16d21d-fe4c-4081-98bc-65fefa465f7d',
      'sha': 'd083fd87fa84451904f71ab8b9adfa88580beb92ca77c046800f79780a20b7e4',
      'status': 'Running'},
     {'name': 'smodel-o',
      'version': '8d311ba3-c336-48d3-99cd-85d95baa6f19',
      'sha': '3473ea8700fbf1a1a8bfb112554a0dde8aab36758030dcde94a9357a83fd5650',
      'status': 'Running'}]}}],
 'engine_lbs': [{'ip': '10.12.1.66',
   'name': 'engine-lb-85846c64f8-ggg2t',
   'status': 'Running',
   'reason': None}]}

Manage Pipeline Deployment Configuration

For full details on pipeline deployment configurations, see Wallaroo SDK Essentials Guide: Pipeline Deployment Configuration.

Troubleshooting Pipeline Deployment

If you deploy more pipelines than your environment can handle, or if you deploy more pipelines than your license allows, you may see an error like the following:

LimitError: You have reached a license limit in your Wallaroo instance. In order to add additional resources, you can remove some of your existing resources. If you have any questions contact us at community@wallaroo.ai: MAX_PIPELINES_LIMIT_EXCEEDED

Undeploy any unnecessary pipelines either through the SDK or through the Wallaroo Pipeline Dashboard, then attempt to redeploy the pipeline in question again.

Undeploy a Pipeline

When a pipeline is not currently needed, it can be undeployed and its resources turned back to the Kubernetes environment. To undeploy a pipeline, use the pipeline undeploy() command.

In this example, the aloha_pipeline will be undeployed:

aloha_pipeline.undeploy()

{'name': 'aloha-test-demo', 'create_time': datetime.datetime(2022, 3, 29, 20, 34, 3, 960957, tzinfo=tzutc()), 'definition': "[{'ModelInference': {'models': [{'name': 'aloha-2', 'version': 'a8e8abdc-c22f-416c-a13c-5fe162357430', 'sha': 'fd998cd5e4964bbbb4f8d29d245a8ac67df81b62be767afbceb96a03d1a01520'}]}}]"}

Get Pipeline Status

The pipeline status() command shows the current status, models, and other information on a pipeline.

The following example shows the pipeline imdb_pipeline status before and after it is deployed:

imdb_pipeline.status

<bound method Pipeline.status of {'name': 'imdb-pipeline', 'create_time': datetime.datetime(2022, 3, 30, 21, 21, 31, 127756, tzinfo=tzutc()), 'definition': "[{'ModelInference': {'models': [{'name': 'embedder-o', 'version': '1c16d21d-fe4c-4081-98bc-65fefa465f7d', 'sha': 'd083fd87fa84451904f71ab8b9adfa88580beb92ca77c046800f79780a20b7e4'}]}}, {'ModelInference': {'models': [{'name': 'smodel-o', 'version': '8d311ba3-c336-48d3-99cd-85d95baa6f19', 'sha': '3473ea8700fbf1a1a8bfb112554a0dde8aab36758030dcde94a9357a83fd5650'}]}}]"}>

imdb_pipeline.deploy()
Waiting for deployment - this will take up to 45s ...... ok

imdb_pipeline.status()

{'status': 'Running',
 'details': None,
 'engines': [{'ip': '10.12.1.65',
   'name': 'engine-778b65459-f9mt5',
   'status': 'Running',
   'reason': None,
   'pipeline_statuses': {'pipelines': [{'id': 'imdb-pipeline',
      'status': 'Running'}]},
   'model_statuses': {'models': [{'name': 'embedder-o',
      'version': '1c16d21d-fe4c-4081-98bc-65fefa465f7d',
      'sha': 'd083fd87fa84451904f71ab8b9adfa88580beb92ca77c046800f79780a20b7e4',
      'status': 'Running'},
     {'name': 'smodel-o',
      'version': '8d311ba3-c336-48d3-99cd-85d95baa6f19',
      'sha': '3473ea8700fbf1a1a8bfb112554a0dde8aab36758030dcde94a9357a83fd5650',
      'status': 'Running'}]}}],
 'engine_lbs': [{'ip': '10.12.1.66',
   'name': 'engine-lb-85846c64f8-ggg2t',
   'status': 'Running',
   'reason': None}]}

Get Pipeline URL Endpoint

The Pipeline URL Endpoint or the Pipeline Deploy URL is used to submit data to a pipeline to use for an inference. This is done through the pipeline _deployment._url() method.

In this example, the pipeline URL endpoint for the pipeline ccfraud_pipeline will be displayed:

ccfraud_pipeline._deployment._url()

'http://engine-lb.ccfraud-pipeline-1:29502/pipelines/ccfraud-pipeline'