This can be downloaded as part of the Wallaroo Tutorials repository.

Wallaroo Connection and ML Workload Orchestration API Tutorial

This tutorial provides a quick set of methods and examples regarding Wallaroo Connections and Wallaroo ML Workload Orchestration. For full details, see the Wallaroo Documentation site.

Wallaroo provides Data Connections and ML Workload Orchestrations to provide organizations with a method of creating and managing automated tasks that can either be run on demand or a regular schedule.

Definitions

• Orchestration: A set of instructions written as a python script with a requirements library. Orchestrations are uploaded to the Wallaroo instance as a .zip file.
• Task: An implementation of an orchestration. Tasks are run either once when requested, on a repeating schedule, or as a service.
• Connection: Definitions set by MLOps engineers that are used by other Wallaroo users for connection information to a data source. Usually paired with orchestrations.

Tutorial Goals

The tutorial will demonstrate the following:

1. Create a workspace and pipeline with a sample model.
2. Upload Wallaroo ML Workload Orchestration through the Wallaroo MLOps API.
3. List available orchestrations through the Wallaroo MLOps API.
4. Run the orchestration once as a Run Once Task through the Wallaroo MLOps API and verify that the information was saved the pipeline logs.

Prerequisites

• An installed Wallaroo instance.
• The following Python libraries installed. These are included by default in a Wallaroo instance’s JupyterHub service.
  • os
  • wallaroo: The Wallaroo SDK. Included with the Wallaroo JupyterHub service by default.
  • pandas: Pandas, mainly used for Pandas DataFrame
  • pyarrow: PyArrow for Apache Arrow support

Initial Steps

For this tutorial, we’ll create a workspace, upload our sample model and deploy a pipeline. We’ll perform some quick sample inferences to verify that everything it working.

Load Libraries

Here we’ll import the various libraries we’ll use for the tutorial.

import wallaroo
from wallaroo.object import EntityNotFoundError, RequiredAttributeMissing

# to display dataframe tables
from IPython.display import display
# used to display dataframe information without truncating
import pandas as pd
pd.set_option('display.max_colwidth', None)
import pyarrow as pa

import time

import requests

# Used to create unique workspace and pipeline names
import string
import random

# make a random 4 character suffix
suffix= ''.join(random.choice(string.ascii_lowercase) for i in range(4))
display(suffix)

'gsze'

Connect to the Wallaroo Instance

The first step is to connect to Wallaroo through the Wallaroo client. The Python library is included in the Wallaroo install and available through the Jupyter Hub interface provided with your Wallaroo environment.

This is accomplished using the wallaroo.Client() command, which provides a URL to grant the SDK permission to your specific Wallaroo environment. When displayed, enter the URL into a browser and confirm permissions. Store the connection into a variable that can be referenced later.

If logging into the Wallaroo instance through the internal JupyterHub service, use wl = wallaroo.Client(). If logging in externally, update the wallarooPrefix and wallarooSuffix variables with the proper DNS information. For more information on Wallaroo Client settings, see the Client Connection guide.

# Login through local Wallaroo instance

wl = wallaroo.Client()

API URL

The variable APIURL is used to specify the connection to the Wallaroo instance’s MLOps API URL, and is composed of the Wallaroo DNS prefix and suffix. For full details, see the Wallaroo API Connection Guide
.

The variables wallarooPrefix and wallarooSuffix variables will be used to derive the API url. For example, if the Wallaroo Prefix is doc-test and the url is example.com, then the MLOps API URL would be doc-test.api.example.com/v1/api/{request}.

Note the . is part of the prefix. If there is no prefix, then wallarooPrefix = ""

Set the Wallaroo Prefix and Suffix in the code segment below based on your Wallaroo instance.

# Setting variables for later steps

wallarooPrefix = "YOUR PREFIX."

wallarooSuffix = "YOUR SUFFIX"

APIURL = f"https://{wallarooPrefix}api.{wallarooSuffix}"

workspace_name = f'apiorchestrationworkspace{suffix}'
pipeline_name = f'apipipeline{suffix}'
model_name = f'apiorchestrationmodel{suffix}'
model_file_name = './models/rf_model.onnx'

Helper Methods

The following helper methods are used to either create or get workspaces, pipelines, and connections.

# helper methods to retrieve workspaces and pipelines

def get_workspace(name):
    workspace = None
    for ws in wl.list_workspaces():
        if ws.name() == name:
            workspace= ws
    if(workspace == None):
        workspace = wl.create_workspace(name)
    return workspace

def get_pipeline(name):
    try:
        pipeline = wl.pipelines_by_name(name)[0]
    except EntityNotFoundError:
        pipeline = wl.build_pipeline(name)
    return pipeline

def get_connection(name, connection_type, connection_arguments):
    try:
        connection = wl.get_connection(name)
    except RequiredAttributeMissing:
        connection =wl.create_connection(name, 
                  connection_type, 
                  connection_arguments)
    return connection

Create the Workspace and Pipeline

We’ll now create our workspace and pipeline for the tutorial. If this tutorial has been run previously, then this will retrieve the existing ones with the assumption they’re for us with this tutorial.

We’ll set the retrieved workspace as the current workspace in the SDK, so all commands will default to that workspace.

workspace = get_workspace(workspace_name)
wl.set_current_workspace(workspace)

workspace_id = workspace.id()

pipeline = get_pipeline(pipeline_name)

Upload the Model and Deploy Pipeline

We’ll upload our model into our sample workspace, then add it as a pipeline step before deploying the pipeline to it’s ready to accept inference requests.

# Upload the model

housing_model_control = wl.upload_model(model_name, model_file_name, framework=wallaroo.framework.Framework.ONNX).configure()

# Add the model as a pipeline step

pipeline.add_model_step(housing_model_control)

#deploy the pipeline
pipeline.deploy()

Waiting for deployment - this will take up to 45s ......... ok

Wallaroo ML Workload Orchestration Example

With the pipeline deployed and our connections set, we will now generate our ML Workload Orchestration. See the Wallaroo ML Workload Orchestrations guide for full details.

Orchestrations are uploaded to the Wallaroo instance as a ZIP file with the following requirements:

For our example, our orchestration will:

1. Use the inference_results_connection to open a HTTP Get connection to the inference data file and use it in an inference request in the deployed pipeline.
2. Submit the inference results to the location specified in the external_inference_connection.

This sample script is stored in remote_inference/main.py with an empty requirements.txt file, and packaged into the orchestration as ./remote_inference/remote_inference.zip. We’ll display the steps in uploading the orchestration to the Wallaroo instance.

Note that the orchestration assumes the pipeline is already deployed.

API Upload the Orchestration

Orchestrations are uploaded via POST as a application/octet-stream with MLOps API route:

• REQUEST
  • POST /v1/api/orchestration/upload
  • Content-Type multipart/form-data
• PARAMETERS
  • file: The file uploaded as Content-Type as application/octet-stream.
  • metadata: Included as Content-Type as application/json with:
    • workspace_id: The numerical id of the workspace to upload the orchestration to.
    • name: The name of the orchestration.
    • The metadata specifying the workspace id and Content-Type as application/json.

Once uploaded, the deployment will be prepared and any requirements will be downloaded and installed.

# retrieve the authorization token
headers = wl.auth.auth_header()

url = f"{APIURL}/v1/api/orchestration/upload"

fp = open("./api_inference_orchestration.zip", "rb")

metadata = f'{{"workspace_id": {workspace_id},"name": "apiorchestrationsample"}}'

response = requests.post(
    url,
    headers=headers,
    files=[
        ("file", 
            ("api_inference_orchestration.zip", fp, "application/octet-stream")
        ),
        ("metadata", 
            ("metadata", metadata, "application/json")
        )
    ],
).json()

display(response)
orchestration_id = response['id']

{'id': 'b951f7b8-0690-4004-86bf-cc9802359313'}

API List Orchestrations

A list of orchestrations retrieved via POST MLOps API route:

• REQUEST
  • POST /v1/api/orchestration/list
• PARAMETERS
  • workspace_id: The numerical identifier of the workspace associated with the orchestration.

# retrieve the authorization token
headers = wl.auth.auth_header()

url = f"{APIURL}/v1/api/orchestration/list"

data = {
    'workspace_id': workspace_id
}

response=requests.post(url, headers=headers, json=data)
display(response.json())

[{'id': 'b951f7b8-0690-4004-86bf-cc9802359313',
  'workspace_id': 8,
  'sha': 'd3b93c9f280734106376e684792aa8b4285d527092fe87d89c74ec804f8e169e',
  'name': 'apiorchestrationsample',
  'file_name': 'api_inference_orchestration.zip',
  'task_id': 'c9622cf8-cbe5-4e3c-b64c-dabd6c5b7fef',
  'owner_id': 'a6857628-b0aa-451e-a0a0-bbc1d6eea6e0',
  'status': 'pending_packaging',
  'created_at': '2023-05-22T20:48:41.233482+00:00',
  'updated_at': '2023-05-22T20:48:41.233482+00:00'}]

API Get Orchestration

A list of orchestrations retrieved via POST MLOps API route:

• REQUEST
  • POST /v1/api/orchestration/get_by_id
• PARAMETERS
  • id: The UUID of the orchestration being retrieved.
• RETURNS
  • id: The ID of the orchestration in UUID format.
  • workspace_id: Numerical value of the workspace the orchestration was uploaded to.
  • sha: The SHA hash value of the orchestration.
  • file_name: The file name the orchestration was uploaded as.
  • task_id: The task id in UUID format for unpacking and preparing the orchestration.
  • owner_id: The Keycloak ID of the user that uploaded the orchestration.
  • status: The status of the orchestration. Status values are:
    • packing: Preparing the orchestration to be used as a task.
    • ready: The orchestration is ready to be deployed as a task.

# retrieve the authorization token
headers = wl.auth.auth_header()

url = f"{APIURL}/v1/api/orchestration/get_by_id"

data = {
    'id': orchestration_id
}

# loop until status is ready
status = None

while status != 'ready':
    response=requests.post(url, headers=headers, json=data).json()
    display(response)
    status = response['status']
    time.sleep(10)

orchestration_sha = response['sha']

{'id': 'b951f7b8-0690-4004-86bf-cc9802359313',
 'workspace_id': 8,
 'sha': 'd3b93c9f280734106376e684792aa8b4285d527092fe87d89c74ec804f8e169e',
 'file_name': 'api_inference_orchestration.zip',
 'task_id': 'c9622cf8-cbe5-4e3c-b64c-dabd6c5b7fef',
 'owner_id': 'a6857628-b0aa-451e-a0a0-bbc1d6eea6e0',
 'status': 'packaging'}
{‘id’: ‘b951f7b8-0690-4004-86bf-cc9802359313’,

‘workspace_id’: 8,

‘sha’: ‘d3b93c9f280734106376e684792aa8b4285d527092fe87d89c74ec804f8e169e’,

‘file_name’: ‘api_inference_orchestration.zip’,

’task_id’: ‘c9622cf8-cbe5-4e3c-b64c-dabd6c5b7fef’,

‘owner_id’: ‘a6857628-b0aa-451e-a0a0-bbc1d6eea6e0’,

‘status’: ‘packaging’}
{‘id’: ‘b951f7b8-0690-4004-86bf-cc9802359313’,

‘workspace_id’: 8,

‘sha’: ‘d3b93c9f280734106376e684792aa8b4285d527092fe87d89c74ec804f8e169e’,

‘file_name’: ‘api_inference_orchestration.zip’,

’task_id’: ‘c9622cf8-cbe5-4e3c-b64c-dabd6c5b7fef’,

‘owner_id’: ‘a6857628-b0aa-451e-a0a0-bbc1d6eea6e0’,

‘status’: ‘packaging’}
{‘id’: ‘b951f7b8-0690-4004-86bf-cc9802359313’,

‘workspace_id’: 8,

‘sha’: ‘d3b93c9f280734106376e684792aa8b4285d527092fe87d89c74ec804f8e169e’,

‘file_name’: ‘api_inference_orchestration.zip’,

’task_id’: ‘c9622cf8-cbe5-4e3c-b64c-dabd6c5b7fef’,

‘owner_id’: ‘a6857628-b0aa-451e-a0a0-bbc1d6eea6e0’,

‘status’: ‘packaging’}
{‘id’: ‘b951f7b8-0690-4004-86bf-cc9802359313’,

‘workspace_id’: 8,

‘sha’: ‘d3b93c9f280734106376e684792aa8b4285d527092fe87d89c74ec804f8e169e’,

‘file_name’: ‘api_inference_orchestration.zip’,

’task_id’: ‘c9622cf8-cbe5-4e3c-b64c-dabd6c5b7fef’,

‘owner_id’: ‘a6857628-b0aa-451e-a0a0-bbc1d6eea6e0’,

‘status’: ‘packaging’}
{‘id’: ‘b951f7b8-0690-4004-86bf-cc9802359313’,

‘workspace_id’: 8,

‘sha’: ‘d3b93c9f280734106376e684792aa8b4285d527092fe87d89c74ec804f8e169e’,

‘file_name’: ‘api_inference_orchestration.zip’,

’task_id’: ‘c9622cf8-cbe5-4e3c-b64c-dabd6c5b7fef’,

‘owner_id’: ‘a6857628-b0aa-451e-a0a0-bbc1d6eea6e0’,

‘status’: ‘packaging’}
{‘id’: ‘b951f7b8-0690-4004-86bf-cc9802359313’,

‘workspace_id’: 8,

‘sha’: ‘d3b93c9f280734106376e684792aa8b4285d527092fe87d89c74ec804f8e169e’,

‘file_name’: ‘api_inference_orchestration.zip’,

’task_id’: ‘c9622cf8-cbe5-4e3c-b64c-dabd6c5b7fef’,

‘owner_id’: ‘a6857628-b0aa-451e-a0a0-bbc1d6eea6e0’,

‘status’: ‘packaging’}
{‘id’: ‘b951f7b8-0690-4004-86bf-cc9802359313’,

‘workspace_id’: 8,

‘sha’: ‘d3b93c9f280734106376e684792aa8b4285d527092fe87d89c74ec804f8e169e’,

‘file_name’: ‘api_inference_orchestration.zip’,

’task_id’: ‘c9622cf8-cbe5-4e3c-b64c-dabd6c5b7fef’,

‘owner_id’: ‘a6857628-b0aa-451e-a0a0-bbc1d6eea6e0’,

‘status’: ‘packaging’}
{‘id’: ‘b951f7b8-0690-4004-86bf-cc9802359313’,

‘workspace_id’: 8,

‘sha’: ‘d3b93c9f280734106376e684792aa8b4285d527092fe87d89c74ec804f8e169e’,

‘file_name’: ‘api_inference_orchestration.zip’,

’task_id’: ‘c9622cf8-cbe5-4e3c-b64c-dabd6c5b7fef’,

‘owner_id’: ‘a6857628-b0aa-451e-a0a0-bbc1d6eea6e0’,

‘status’: ‘ready’}

Task Management Tutorial

Once an Orchestration has the status ready, it can be run as a task. Tasks have three run options.

Run Task Once via API

We’ll do both a Run Once task and generate our Run Once Task from our orchestration. Orchestrations are started as a run once task with the following request:

• REQUEST
  • POST /v1/api/orchestration/task/run_once
• PARAMETERS
  • name (String Required): The name to assign to the task.
  • workspace_id (Integer Required): The numerical identifier of the workspace associated with the orchestration.
  • orch_id(String Required): The orchestration ID represented by a UUID.
  • json(Dict Required): The parameters to pass to the task.

Tasks are generated and run once with the Orchestration run_once method. Any arguments for the orchestration are passed in as a Dict. If there are no arguments, then an empty set {} is passed.

# retrieve the authorization token
headers = wl.auth.auth_header()

data = {
    "name": "api run once task",
    "workspace_id": workspace_id,
    "orch_id": orchestration_id,
    "json": {
        "workspace_name": workspace_name,
        "pipeline_name": pipeline_name
    }
}

import datetime
task_start = datetime.datetime.now()

url=f"{APIURL}/v1/api/task/run_once"

response=requests.post(url, headers=headers, json=data).json()
display(response)
task_id = response['id']

{'id': 'c868aa44-f7fe-4e3d-b11d-e1e6af3ec150'}

Task Status via API

The list of tasks in the Wallaroo instance is retrieves through the Wallaroo MLOPs API request:

• REQUEST
  • POST /v1/api/task/get_by_id
• PARAMETERS
  • task: The numerical identifier of the workspace associated with the orchestration.
  • orch_id: The orchestration ID represented by a UUID.
  • json: The parameters to pass to the task.

For this example, the status of the previously created task will be generated, then looped until it has reached status started.

# retrieve the authorization token
headers = wl.auth.auth_header()

url=f"{APIURL}/v1/api/task/get_by_id"

data = {
    "id": task_id
}

status = None

while status != 'started':
    response=requests.post(url, headers=headers, json=data).json()
    display(response)
    status = response['status']
    time.sleep(10)

{'name': 'api run once task',
 'id': 'c868aa44-f7fe-4e3d-b11d-e1e6af3ec150',
 'image': 'proxy.replicated.com/proxy/wallaroo/ghcr.io/wallaroolabs/arbex-orch-deploy',
 'image_tag': 'v2023.2.0-main-3271',
 'bind_secrets': ['minio'],
 'extra_env_vars': {'MINIO_URL': 'http://minio.wallaroo.svc.cluster.local:9000',
  'ORCH_OWNER_ID': 'a6857628-b0aa-451e-a0a0-bbc1d6eea6e0',
  'ORCH_SHA': 'd3b93c9f280734106376e684792aa8b4285d527092fe87d89c74ec804f8e169e',
  'TASK_DEBUG': 'false',
  'TASK_ID': 'c868aa44-f7fe-4e3d-b11d-e1e6af3ec150'},
 'auth_init': True,
 'workspace_id': 8,
 'flavor': 'exec_orch_oneshot',
 'reap_threshold_secs': 900,
 'exec_type': 'job',
 'status': 'pending',
 'input_data': {'pipeline_name': 'apipipelinegsze',
  'workspace_name': 'apiorchestrationworkspacegsze'},
 'killed': False,
 'created_at': '2023-05-22T21:08:31.099447+00:00',
 'updated_at': '2023-05-22T21:08:31.105312+00:00',
 'last_runs': []}
{’name’: ‘api run once task’,

‘id’: ‘c868aa44-f7fe-4e3d-b11d-e1e6af3ec150’,

‘image’: ‘proxy.replicated.com/proxy/wallaroo/ghcr.io/wallaroolabs/arbex-orch-deploy’,

‘image_tag’: ‘v2023.2.0-main-3271’,

‘bind_secrets’: [‘minio’],

’extra_env_vars’: {‘MINIO_URL’: ‘http://minio.wallaroo.svc.cluster.local:9000’,

‘ORCH_OWNER_ID’: ‘a6857628-b0aa-451e-a0a0-bbc1d6eea6e0’,

‘ORCH_SHA’: ‘d3b93c9f280734106376e684792aa8b4285d527092fe87d89c74ec804f8e169e’,

‘TASK_DEBUG’: ‘false’,

‘TASK_ID’: ‘c868aa44-f7fe-4e3d-b11d-e1e6af3ec150’},

‘auth_init’: True,

‘workspace_id’: 8,

‘flavor’: ’exec_orch_oneshot’,

‘reap_threshold_secs’: 900,

’exec_type’: ‘job’,

‘status’: ‘started’,

‘input_data’: {‘pipeline_name’: ‘apipipelinegsze’,

‘workspace_name’: ‘apiorchestrationworkspacegsze’},

‘killed’: False,

‘created_at’: ‘2023-05-22T21:08:31.099447+00:00’,

‘updated_at’: ‘2023-05-22T21:08:36.585775+00:00’,

’last_runs’: [{‘run_id’: ‘96a7f85f-e30c-40b5-9185-0dee5bd1a15e’,

‘status’: ‘running’,

‘created_at’: ‘2023-05-22T21:08:33.112805+00:00’,

‘updated_at’: ‘2023-05-22T21:08:33.112805+00:00’}]}

Task Results

We can view the inferences from our logs and verify that new entries were added from our task. In our case, we’ll assume the task once started takes about 1 minute to run (deploy the pipeline, run the inference, undeploy the pipeline). We’ll add in a wait of 1 minute, then display the logs during the time period the task was running.

time.sleep(30)

task_end = datetime.datetime.now()
display(task_end)

pipeline.logs(start_datetime = task_start, end_datetime = task_end)

datetime.datetime(2023, 5, 22, 21, 9, 32, 447321)

Get Tasks by Orchestration SHA

Tasks tied to the same orchestration are retrieved through the following request.

• REQUEST
  • POST /v1/api/task/get_tasks_by_orch_sha
• PARAMETERS
  • sha: The orchestrations SHA hash.
• RETURNS
  • ids: List[string] List of tasks by UUID.

# retrieve the authorization token
headers = wl.auth.auth_header()

url=f"{APIURL}/v1/api/task/get_tasks_by_orch_sha"

data = {
    "sha": orchestration_sha
}

response=requests.post(url, headers=headers, json=data).json()
display(response)

{'ids': ['2424a9a7-2331-42f6-bd84-90643386b130',
  'c868aa44-f7fe-4e3d-b11d-e1e6af3ec150',
  'a41fe4ae-b8a4-4e1f-a45a-114df64ae2bc']}

Task Last Runs History

The history of a task, which each deployment of the task is known as a task run is retrieved with the Task last_runs method that takes the following arguments. It returns the reverse chronological order of tasks runs listed by updated_at.

• REQUEST
  • POST /v1/api/task/list_task_runs
• PARAMETERS
  • task_id: The numerical identifier of the task.
  • status: Filters the task history by the status. If all, returns all statuses. Status values are:
    • running: The task has started.
    • failure: The task failed.
    • success: The task completed.
  • limit: The number of tasks runs to display.
• RETURNS
  • ids: List of task runs ids in UUID.

# retrieve the authorization token
headers = wl.auth.auth_header()

url=f"{APIURL}/v1/api/task/list_task_runs"

data = {
    "task_id": task_id
}

response=requests