Wallaroo ML Workload Orchestration Management

Wallaroo provides ML Workload Orchestrations and Tasks to automate processes in a Wallaroo instance. For example:

• Deploy a pipeline, retrieve data through a data connector, submit the data for inferences, undeploy the pipeline
• Replace a model with a new version
• Retrieve shadow deployed inference results and submit them to a database

Orchestration Flow

ML Workload Orchestration flow works within 3 tiers:

One example may be of making donuts.

• The ML Workload Orchestration is the recipe.
• The Task is the order to make the donuts. It might be Run Once, so only one set of donuts are made, or Run Scheduled, so donuts are made every 2nd Sunday at 6 AM. If Run Scheduled, the donuts are made every time the schedule hits until the order is cancelled (aka killed).
• The Task Run are the donuts with their own receipt of creation (logs, etc).

For details on uploading ML Workload Orchestrations, see the Wallaroo SDK Essentials Guide: ML Workload Orchestration and the Wallaroo MLOps API Essentials Guide: ML Workload Orchestration Management.

Orchestrations and their Tasks and Task Runs are visible through the Wallaroo Dashboard through the following process.

Task User Interface Overview

1. From the Wallaroo Dashboard, select the workspace where the workloads were uploaded to.

2. From the upper right corner, select Workloads.

   

3. The list of uploaded ML Workload Orchestrations are displayed with the following:

   

   • Search Bar (A): Filter the list by workload name.
   • Status Filter (B): Filter the list by:
     • Only Active: Only show Active workloads.
     • Only Inactive: Only show Inactive workloads.
     • Only Error: Only show workloads flagged with an Error.
   • Workload (C): The assigned name of the workload.
   • Status (D): Whether the workload orchestration status is Active, Inactive, or has an Error.
   • Created At (E): The date the Orchestration was uploaded.
   • ID (F): The unique workload orchestration ID in UUID format.

4. Select a workload orchestration to view Tasks generated from this workload. The Orchestration Page has the following details:

   • Orchestration Details:

     

     • Orchestration Name (A): The name assigned when the workload was created from the workload orchestration.
     • Orchestration ID (B): The ID of the workload in UUID format.
     • Created (C): The date the workload was uploaded.
     • File (D): The file the workload was uploaded from.
     • File Hash (E): The hash of the workload file.
     • Task and Task Runs buttons as detailed below.

   • Task: Each Task generated from the Orchestration as either a Run Once or Run Scheduled. Run Once tasks generate a single Task Run, then terminate. Run Scheduled tasks generate a Task Run each time the schedule pattern occurs. For example, a Run Scheduled task set to run every 1st of the month at 4 PM will have 12 Task Runs generated from it a year later.

     

     • Type (F): Tasks are shown as either Run Once (a lightning bolt icon) or Run Scheduled (circular arrow icon).
     • Task Name (G): The name of the task.
     • Task ID (H): The ID of the task in UUID format.
     • Run At (I): The last scheduled run, and for Run Scheduled tasks, the next scheduled run. The actual time of execution may depend on system load, time differences and other factors.
     • Actions (J): Allows the task to be Stopped. If a Run Scheduled Task, this will stop any further Task Runs.

   • Task Run: Each Task generates one or more Task Runs with the following details.

     

     • Task Run (A): The name of the Task that generated the Task Run, including the symbol for Run Once (a lightning bolt icon) or Run Scheduled (circular arrow icon).

     • Task ID (B): The ID of the Task that generated the Task Run in UUID format.

     • Status (C): The status of the Task Run as either Success, Failure, or Timed_out.

     • Run At (D): The date and time the task previous ran, and for Run Scheduled tasks, the next scheduled run.

     • Task Run Logs (E): The logs of the Task Run that displays an abbreviated Task Run logs.

       

Create a Task

To create a task using the Workload Orchestration UI from the Orchestration Details page:

1. Select Create Task.

2. Add the following:

   

   1. Task Name: The name of the Task created from the Workload Orchestration.
   2. Task Type:
      1. Select Run immediately to generate a Run Once task.
      2. Select Run recurringly to generate a Run Scheduled task. For Run Scheduled tasks, select the Months, Days, and Times that the task it to run. For example, selecting only January 15 8:00 PM will only run on that date, while selecting January, March, May at 1, 7, 15, and 30 at 8:00 PM will generate a Task Run for each of those days that meet those conditions.
   3. Advanced Settings:
      1. Json arguments: Tasks accept Json arguments that are used to modify how each tasks is run. See the Wallaroo SDK Essentials Guide: ML Workload Orchestration for details on creating a Workload Orchestration that accepts additional Task arguments.
      2. Timeout (sec): How long the Task should run before stopping. This is used to terminate tasks that may be in an infinite loop.