Wallaroo ML Workload Orchestration Management

How to use Wallaroo ML Workload Orchestrations

Wallaroo provides ML Workload Orchestrations and Tasks to automate processes in a Wallaroo instance. For example:

  • Deploy a pipeline, retrieve data through a data connector, submit the data for inferences, undeploy the pipeline
  • Replace a model with a new version
  • Retrieve shadow deployed inference results and submit them to a database

Orchestration Flow

ML Workload Orchestration flow works within 3 tiers:

ML Workload OrchestrationUser created custom instructions that provide automated processes that follow the same steps every time without error. Orchestrations contain the instructions to be performed, uploaded as a .ZIP file with the instructions, requirements, and artifacts.
TaskInstructions on when to run an Orchestration as a scheduled Task. Tasks can be Run Once, where is creates a single Task Run, or Run Scheduled, where a Task Run is created on a regular schedule based on the Kubernetes cronjob specifications. If a Task is Run Scheduled, it will create a new Task Run every time the schedule parameters are met until the Task is killed.
Task RunThe execution of an task. These validate business operations are successful identify any unsuccessful task runs. If the Task is Run Once, then only one Task Run is generated. If the Task is a Run Scheduled task, then a new Task Run will be created each time the schedule parameters are met, with each Task Run having its own results and logs.
Wallaroo Components

One example may be of making donuts.

  • The ML Workload Orchestration is the recipe.
  • The Task is the order to make the donuts. It might be Run Once, so only one set of donuts are made, or Run Scheduled, so donuts are made every 2nd Sunday at 6 AM. If Run Scheduled, the donuts are made every time the schedule hits until the order is cancelled (aka killed).
  • The Task Run are the donuts with their own receipt of creation (logs, etc).

For details on uploading ML Workload Orchestrations, see the Wallaroo SDK Essentials Guide: ML Workload Orchestration and the Wallaroo MLOps API Essentials Guide: ML Workload Orchestration Management.

Orchestrations and their Tasks and Task Runs are visible through the Wallaroo Dashboard through the following process.

Task User Interface Overview

  1. From the Wallaroo Dashboard, select the workspace where the workloads were uploaded to.

  2. From the upper right corner, select Workloads.

    Access workloads
  3. The list of uploaded ML Workload Orchestrations are displayed with the following:

    Workload List
    • Search Bar (A): Filter the list by workload name.
    • Status Filter (B): Filter the list by:
      • Only Active: Only show Active workloads.
      • Only Inactive: Only show Inactive workloads.
      • Only Error: Only show workloads flagged with an Error.
    • Workload (C): The assigned name of the workload.
    • Status (D): Whether the workload orchestration status is Active, Inactive, or has an Error.
    • Created At (E): The date the Orchestration was uploaded.
    • ID (F): The unique workload orchestration ID in UUID format.
  4. Select a workload orchestration to view Tasks generated from this workload. The Orchestration Page has the following details:

    • Orchestration Details:

      Orchestration Details
      • Orchestration Name (A): The name assigned when the workload was created from the workload orchestration.
      • Orchestration ID (B): The ID of the workload in UUID format.
      • Created (C): The date the workload was uploaded.
      • File (D): The file the workload was uploaded from.
      • File Hash (E): The hash of the workload file.
      • Task and Task Runs buttons as detailed below.
    • Task: Each Task generated from the Orchestration as either a Run Once or Run Scheduled. Run Once tasks generate a single Task Run, then terminate. Run Scheduled tasks generate a Task Run each time the schedule pattern occurs. For example, a Run Scheduled task set to run every 1st of the month at 4 PM will have 12 Task Runs generated from it a year later.

      Task Details
      • Type (F): Tasks are shown as either Run Once (a lightning bolt icon) or Run Scheduled (circular arrow icon).
      • Task Name (G): The name of the task.
      • Task ID (H): The ID of the task in UUID format.
      • Run At (I): The last scheduled run, and for Run Scheduled tasks, the next scheduled run. The actual time of execution may depend on system load, time differences and other factors.
      • Actions (J): Allows the task to be Stopped. If a Run Scheduled Task, this will stop any further Task Runs.
    • Task Run: Each Task generates one or more Task Runs with the following details.

      Task Run Details
      • Task Run (A): The name of the Task that generated the Task Run, including the symbol for Run Once (a lightning bolt icon) or Run Scheduled (circular arrow icon).

      • Task ID (B): The ID of the Task that generated the Task Run in UUID format.

      • Status (C): The status of the Task Run as either Success, Failure, or Timed_out.

      • Run At (D): The date and time the task previous ran, and for Run Scheduled tasks, the next scheduled run.

      • Task Run Logs (E): The logs of the Task Run that displays an abbreviated Task Run logs.

        Task Run Logs

Create a Task

To create a task using the Workload Orchestration UI from the Orchestration Details page:

  1. Select Create Task.

  2. Add the following:

    Create Task from Workload Orchestration
    1. Task Name: The name of the Task created from the Workload Orchestration.
    2. Task Type:
      1. Select Run immediately to generate a Run Once task.
      2. Select Run recurringly to generate a Run Scheduled task. For Run Scheduled tasks, select the Months, Days, and Times that the task it to run. For example, selecting only January 15 8:00 PM will only run on that date, while selecting January, March, May at 1, 7, 15, and 30 at 8:00 PM will generate a Task Run for each of those days that meet those conditions.
    3. Advanced Settings:
      1. Json arguments: Tasks accept Json arguments that are used to modify how each tasks is run. See the Wallaroo SDK Essentials Guide: ML Workload Orchestration for details on creating a Workload Orchestration that accepts additional Task arguments.
      2. Timeout (sec): How long the Task should run before stopping. This is used to terminate tasks that may be in an infinite loop.

ML Workload Orchestration Configuration Guide

How to enable or disable Wallaroo ML Workload Orchestration

Wallaroo ML Workload Orchestration Requirements

Requirements for uploading a Wallaroo ML Workload Orchestration