Wallaroo ML Workload Orchestration Management

How to use Wallaroo ML Workload Orchestrations

Wallaroo provides ML Workload Orchestrations and Tasks to automate processes in a Wallaroo instance. For example:

  • Deploy a pipeline, retrieve data through a data connector, submit the data for inferences, undeploy the pipeline
  • Replace a model with a new version
  • Retrieve shadow deployed inference results and submit them to a database

Orchestration Flow

ML Workload Orchestration flow works within 3 tiers:

ML Workload OrchestrationUser created custom instructions that provide automated processes that follow the same steps every time without error. Orchestrations contain the instructions to be performed, uploaded as a .ZIP file with the instructions, requirements, and artifacts.
TaskInstructions on when to run an Orchestration as a scheduled Task. Tasks can be Run Once, where is creates a single Task Run, or Run Scheduled, where a Task Run is created on a regular schedule based on the Kubernetes cronjob specifications. If a Task is Run Scheduled, it will create a new Task Run every time the schedule parameters are met until the Task is killed.
Task RunThe execution of an task. These validate business operations are successful identify any unsuccessful task runs. If the Task is Run Once, then only one Task Run is generated. If the Task is a Run Scheduled task, then a new Task Run will be created each time the schedule parameters are met, with each Task Run having its own results and logs.
Wallaroo Components

One example may be of making donuts.

  • The ML Workload Orchestration is the recipe.
  • The Task is the order to make the donuts. It might be Run Once, so only one set of donuts are made, or Run Scheduled, so donuts are made every 2nd Sunday at 6 AM. If Run Scheduled, the donuts are made every time the schedule hits until the order is cancelled (aka killed).
  • The Task Run are the donuts with their own receipt of creation (logs, etc).

For details on uploading ML Workload Orchestrations, see the Wallaroo SDK Essentials Guide: ML Workload Orchestration and the Wallaroo MLOps API Essentials Guide: ML Workload Orchestration Management.

1 - ML Workload Orchestration Configuration Guide

How to enable or disable Wallaroo ML Workload Orchestration

Wallaroo ML Workload Orchestration allows organizations to automate processes and run them either on demand or on a reoccurring schedule.

The following guide shows how to enable Wallaroo ML Workload Orchestration in a Wallaroo instance.

Enable Wallaroo ML Workload Orchestration for Kots Installations

Organizations that install Wallaroo using kots can enable or disable Wallaroo ML Workload Orchestration from the Wallaroo Administrative Dashboard through the following process.

  1. From a terminal shell with access to the Kubernetes environment hosting the Wallaroo instance, run the following. Replace wallaroo with the namespace used for the Wallaroo installation.

    kubectl kots admin-console --namespace wallaroo
  2. Access the Wallaroo Administrative Dashboard through a browser. By default, http://localhost:8080. Enter the administrative password when requested.

  3. From the top navigation menu, select Config.

    Configure Installation
  4. From Pipeline Orchestration, either enable or disable Enable Pipeline Orchestration service.

    Enable Pipeline Orchestration
  5. Scroll to the bottom and select Save config.

  6. From the The config for Wallaroo has been updated. module, select Go to updated version.

  7. Select the most recent version with the updated configuration and select Deploy.

The update process will take 10-15 minutes depending on your Wallaroo instance and other changes made.

Enable Wallaroo ML Workload Orchestration for Helm Installations

To enable Wallaroo ML Workload Orchestration for Helm based installations of Wallaroo:

  1. Set the local values YAML file - by default local-values.yaml with the following:

        enabled: true
  2. If installing Wallaroo, then format the helm install command as follows, setting the $RELEASE, $REGISTRYURL, $VERSION, and $LOCALVALUES.yaml as required for the installation settings.

    helm install $RELEASE $REGISTRYURL --version $VERSION --values $LOCALVALUES.yaml
  3. If performing an update to the Wallaroo instance configuration, then use the helm upgrade, , setting the $RELEASE, $REGISTRYURL, $VERSION, and $LOCALVALUES.yaml as required for the installation settings.

For example, to upgrade the registration wallaroo from the EE channel the command would be:

helm upgrade wallaroo oci://registry.replicated.com/wallaroo/uat/wallaroo --version 2023.1.0-2662 --values local-values.yaml

The update process will take 10-15 minutes depending on your Wallaroo instance and other changes made.


2 - Wallaroo ML Workload Orchestrations User Interface

How to view uploaded ML Workload Orchestrations and their associates Tasks

Orchestrations and their Tasks and Task Runs are visible through the Wallaroo Dashboard through the following process.

  1. From the Wallaroo Dashboard, select the workspace where the workloads were uploaded to.

  2. From the upper right corner, select Workloads.

    Access workloads
  3. The list of uploaded ML Workload Orchestrations are displayed with the following:

    • Search Bar (A): Filter the list by workload name.
    • Status Filter (B): Filter the list by:
      • Only Active: Only show Active workloads.
      • Only Inactive: Only show Inactive workloads.
      • Only Error: Only show workloads flagged with an Error.
    • Workload (C): The assigned name of the workload.
    • Status (D): Whether the workload orchestration status is Active, Inactive, or has an Error.
    • Created At (E): The date the Orchestration was uploaded.
    • ID (F): The unique workload orchestration ID in UUID format.
    Workload List
  4. Select a workload orchestration to view Tasks generated from this workload. The Orchestration Page has the following details:

    Tasks List
    • Orchestration Details:
      • Orchestration Name (A): The name assigned when the workload was created from the workload orchestration.
      • Orchestration ID (B): The ID of the workload in UUID format.
      • Created (C): The date the workload was uploaded.
      • File (D): The file the workload was uploaded from.
      • File Hash (E): The hash of the workload file.
    • Workload Log: Each Task generated from the Orchestration displays the following:
      • Type (F): Tasks are shown as either Run Once (a lightning bolt icon) or Run Scheduled (circular arrow icon).
      • Task Name (G): The name of the task.
      • Task ID (H): The ID of the task in UUID format.
      • Last Run Status (I): The status of the task’s last Task Run as either Success or Failure.
      • Run At (J): For Run Once tasks, shows the date and time the Task Run was started. For Run Scheduled tasks, shows the date and time the last Task Run was started, and the next scheduled run for the next Task Run. The actual Task Run start time may vary due to multiple factors.

3 - Wallaroo ML Workload Orchestration Requirements

Requirements for uploading a Wallaroo ML Workload Orchestration

Orchestration Requirements

Orchestrations are uploaded to the Wallaroo instance as a ZIP file with the following requirements:

User Code(Required) Python script as .py filesIf main.py exists, then that will be used as the task entrypoint. Otherwise, the first main.py found in any subdirectory will be used as the entrypoint. If no main.py is found, the orchestration will not be accepted.
Python Library Requirements(Optional) requirements.txt file in the requirements file format.A standard Python requirements.txt for any dependencies to be provided in the task environment. The Wallaroo SDK will already be present and should not be included in the requirements.txt. Multiple requirements.txt files are not allowed.
Other artifacts Other artifacts such as files, data, or code to support the orchestration.

Zip Instructions

In a terminal with the zip command, assemble artifacts as above and then create the archive. The zip command is included by default with the Wallaroo JupyterHub service.

zip commands take the following format, with {zipfilename}.zip as the zip file to save the artifacts to, and each file thereafter as the files to add to the archive.

zip {zipfilename}.zip file1, file2, file3....

For example, the following command will add the files main.py and requirements.txt into the file hello.zip.

$ zip hello.zip main.py requirements.txt 
  adding: main.py (deflated 47%)
  adding: requirements.txt (deflated 52%)

Example requirements.txt file


Orchestration Recommendations

The following recommendations will make using Wallaroo orchestrations.

  • The version of Python used should match the same version as in the Wallaroo JupyterHub service.
  • The same version of the Wallaroo SDK should match the server. For a 2023.2.1 Wallaroo instance, use the Wallaroo SDK version 2023.2.1.
  • Specify the version of pip dependencies.
  • The wallaroo.Client constructor auth_type argument is ignored. Using wallaroo.Client() is sufficient.
  • The following methods will assist with orchestrations:
    • wallaroo.in_task() : Returns True if the code is running within an orchestration task.
    • wallaroo.task_args(): Returns a Dict of invocation-specific arguments passed to the run_ calls.
  • Orchestrations will be run in the same way as running within the Wallaroo JupyterHub service, from the version of Python libraries (unless specifically overridden by the requirements.txt setting, which is not recommended), and running in the virtualized directory /home/jovyan/.

Orchestration Code Samples

The following demonstres using the wallaroo.in_task() and wallaroo.task_args() methods within an Orchestration. This sample code uses wallaroo.in_task() to verify whether or not the script is running as a Wallaroo Task. If true, it will gather the wallaroo.task_args() and use them to set the workspace and pipeline. If False, then it sets the pipeline and workspace manually.

# get the arguments
wl = wallaroo.Client()

# if true, get the arguments passed to the task
if wl.in_task():
  arguments = wl.task_args()
  # arguments is a key/value pair, set the workspace and pipeline name
  workspace_name = arguments['workspace_name']
  pipeline_name = arguments['pipeline_name']
# False:  We're not in a Task, so set the pipeline manually