This can be downloaded as part of the Wallaroo Tutorials repository.

Wallaroo ML Automation Run Continuously Task Tutorial

This tutorial provides a tutorial for the Run Continuously Task for ML Orchestrations.

Definitions

• Orchestration: A set of instructions written as a python script with a requirements library. Orchestrations are uploaded to the Wallaroo instance as a .zip file.
• Task: An implementation of an orchestration. Tasks are run either once when requested, on a repeating schedule, or continuously.
• Connection: Definitions set by MLOps engineers that are used by other Wallaroo users for connection information to a data source. Usually paired with orchestrations.

Tasks are created from an orchestration through the following methods.

Tutorial Goals

The tutorial will demonstrate the following:

1. Create a Wallaroo connection to retrieving information from an external source.
2. Upload Wallaroo ML Workload Orchestration with a task meant for Run Continuously Tasks. The instructions are:
3. Deploy a previously defined pipeline.
4. Every 1 minute, download data through the defined connection. This simulates using a database connection to retrieve the latest data from a data store.
5. Perform an inference with the downloaded data.
6. Create a Run Continuously Task and verify that the loop has pulled data and performed the inference request.
7. Kill the task and undeploy the pipeline used to end the tutorial.

Prerequisites

• An installed Wallaroo instance.
• The following Python libraries installed. These are included by default in a Wallaroo instance’s JupyterHub service.
  • os
  • wallaroo: The Wallaroo SDK. Included with the Wallaroo JupyterHub service by default.
  • pandas: Pandas, mainly used for Pandas DataFrame
  • pyarrow: PyArrow for Apache Arrow support

Orchestration Setup Steps

For this tutorial, we’ll create a workspace, upload our sample model and deploy a pipeline. We’ll perform some quick sample inferences to verify that everything it working.

Load Libraries

Here we’ll import the various libraries we’ll use for the tutorial.

import wallaroo
from wallaroo.object import EntityNotFoundError, RequiredAttributeMissing

# to display dataframe tables
from IPython.display import display
# used to display dataframe information without truncating
import pandas as pd
pd.set_option('display.max_colwidth', None)
pd.set_option('display.max_columns', None)
import pyarrow as pa

import time

Connect to the Wallaroo Instance

The first step is to connect to Wallaroo through the Wallaroo client. The Python library is included in the Wallaroo install and available through the Jupyter Hub interface provided with your Wallaroo environment.

This is accomplished using the wallaroo.Client() command, which provides a URL to grant the SDK permission to your specific Wallaroo environment. When displayed, enter the URL into a browser and confirm permissions. Store the connection into a variable that can be referenced later.

If logging into the Wallaroo instance through the internal JupyterHub service, use wl = wallaroo.Client(). For more information on Wallaroo Client settings, see the Client Connection guide.

# Login through local Wallaroo instance

wl = wallaroo.Client()

# Setting variables for later steps

workspace_name = f'continuoustutorial'
pipeline_name = f'continuouspipeline'
model_name = f'houseprice'
model_file_name = './models/rf_model.onnx'

connection_name = f'continuous_data_sample'
connection_type = "HTTP"
connection_argument = {'host':'https://github.com/WallarooLabs/Wallaroo_Tutorials/raw/refs/heads/wallaroo2024.3_tutorials/wallaroo-automate/orchestration_sdk_simple_tutorial/data/xtest-1k.arrow'}

Create the Workspace and Pipeline

We’ll now create our workspace and pipeline for the tutorial. If this tutorial has been run previously, then this will retrieve the existing ones with the assumption they’re for us with this tutorial.

We’ll set the retrieved workspace as the current workspace in the SDK, so all commands will default to that workspace.

workspace = wl.get_workspace(name=workspace_name, create_if_not_exist=True)
wl.set_current_workspace(workspace)

pipeline = wl.build_pipeline(pipeline_name)

Upload the Model and Deploy Pipeline

We’ll upload our model into our sample workspace, then add it as a pipeline step before deploying the pipeline to it’s ready to accept inference requests.

# Upload the model

housing_model_control = (wl.upload_model(model_name, 
                                         model_file_name, 
                                         framework=wallaroo.framework.Framework.ONNX)
                                         .configure(tensor_fields=["tensor"])
                        )

# Add the model as a pipeline step

pipeline.add_model_step(housing_model_control)

#deploy the pipeline
deploy_config = wallaroo.DeploymentConfigBuilder().replica_count(1).cpus(0.5).memory("1Gi").build()

pipeline.deploy(deployment_config=deploy_config, wait_for_status=False)

Deployment initiated for continuouspipeline. Please check pipeline status.

import time

while pipeline.status()['status'] != 'Running':
    time.sleep(15)
    print("Waiting for deployment.")

pipeline.status()

Waiting for deployment.

{'status': 'Running',
 'details': [],
 'engines': [{'ip': '10.28.2.54',
   'name': 'engine-df98f4788-q9jjw',
   'status': 'Running',
   'reason': None,
   'details': [],
   'pipeline_statuses': {'pipelines': [{'id': 'continuouspipeline',
      'status': 'Running',
      'version': '759bd273-6b20-4cc2-87c1-7c48b81686f4'}]},
   'model_statuses': {'models': [{'model_version_id': 7,
      'name': 'houseprice',
      'sha': 'e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6',
      'status': 'Running',
      'version': '5b7438ae-04f0-4398-8dae-3d7a0370e746'}]}}],
 'engine_lbs': [{'ip': '10.28.2.53',
   'name': 'engine-lb-6b59985857-j7475',
   'status': 'Running',
   'reason': None,
   'details': []}],
 'sidekicks': []}

Create Connections

We will create the data source connection via the Wallaroo client command create_connection.

Connections are created with the Wallaroo client command create_connection with the following parameters.

• IMPORTANT NOTE: Data connections names must be unique. Attempting to create a data connection with the same name as an existing data connection will result in an error.

We’ll also create a data connection named inference_results_connection with our helper function get_connection that will either create or retrieve a connection if it already exists. From there we’ll create out connections:

• houseprice_arrow_table: An Apache Arrow file stored on GitHub that will be used for our inference input.

wl.create_connection(connection_name, connection_type, connection_argument)

Test Orchestration Script

To verify the orchestration will work, we’ll test the relevant code here. The complete code is in ./remote_inference/main.py.

The run below will run through the inference loop twice to verify the loop code works.

print("Deploy the pipeline.")
deploy_config = wallaroo.DeploymentConfigBuilder().replica_count(1).cpus(0.5).memory("1Gi").build()

pipeline.deploy(deployment_config=deploy_config, wait_for_status=False)

# verify the pipeline is running

while pipeline.status()['status'] != 'Running':
    time.sleep(15)
    print("Waiting for deployment.")

print(pipeline.status())

# Get the connection - assuming it will be the only one

inference_source_connection = wl.get_connection(name=connection_name)

# our continuous loop - check every minute for the file.
# in a real example, this would be a database with a search filter to inference on the latest
# updates, then store the inference results in another data store

# for this test, we'll just run through twice to verify it works
for x in range (2):
    time.sleep(60)

    print(f"Getting arrow table file")
    # Retrieve the file
    # set accept as apache arrow table
    headers = {
        'Accept': 'application/vnd.apache.arrow.file'
    }

    response = requests.get(
                        inference_source_connection.details()['host'], 
                        headers=headers
                    )

    # Arrow table is retrieved 
    with pa.ipc.open_file(response.content) as reader:
        arrow_table = reader.read_all()

    print("Inference time.  Displaying results after.")
    # Perform the inference
    result = pipeline.infer(arrow_table)

    result_dataframe = result.to_pandas()

    print(result_dataframe.head(5))

Deploy the pipeline.
Deployment initiated for continuouspipeline. Please check pipeline status.
{'status': 'Running', 'details': [], 'engines': [{'ip': '10.28.2.56', 'name': 'engine-8595d79f94-8zvrs', 'status': 'Running', 'reason': None, 'details': [], 'pipeline_statuses': {'pipelines': [{'id': 'continuouspipeline', 'status': 'Running', 'version': 'fd88bf40-1305-491f-9c93-7c27fc49e754'}]}, 'model_statuses': {'models': [{'model_version_id': 7, 'name': 'houseprice', 'sha': 'e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6', 'status': 'Running', 'version': '5b7438ae-04f0-4398-8dae-3d7a0370e746'}]}}], 'engine_lbs': [{'ip': '10.28.2.55', 'name': 'engine-lb-6b59985857-jwwtb', 'status': 'Running', 'reason': None, 'details': []}], 'sidekicks': []}
Getting arrow table file
Inference time.  Displaying results after.
                     time  \
0 2024-09-24 17:58:40.884   
1 2024-09-24 17:58:40.884   
2 2024-09-24 17:58:40.884   
3 2024-09-24 17:58:40.884   
4 2024-09-24 17:58:40.884   

                                                                                                               in.tensor  \
0     [4.0, 2.5, 2900.0, 5505.0, 2.0, 0.0, 0.0, 3.0, 8.0, 2900.0, 0.0, 47.6063, -122.02, 2970.0, 5251.0, 12.0, 0.0, 0.0]   
1     [2.0, 2.5, 2170.0, 6361.0, 1.0, 0.0, 2.0, 3.0, 8.0, 2170.0, 0.0, 47.7109, -122.017, 2310.0, 7419.0, 6.0, 0.0, 0.0]   
2      [3.0, 2.5, 1300.0, 812.0, 2.0, 0.0, 0.0, 3.0, 8.0, 880.0, 420.0, 47.5893, -122.317, 1300.0, 824.0, 6.0, 0.0, 0.0]   
3    [4.0, 2.5, 2500.0, 8540.0, 2.0, 0.0, 0.0, 3.0, 9.0, 2500.0, 0.0, 47.5759, -121.994, 2560.0, 8475.0, 24.0, 0.0, 0.0]   
4  [3.0, 1.75, 2200.0, 11520.0, 1.0, 0.0, 0.0, 4.0, 7.0, 2200.0, 0.0, 47.7659, -122.341, 1690.0, 8038.0, 62.0, 0.0, 0.0]   

  out.variable  anomaly.count  
0   [718013.7]              0  
1   [615094.6]              0  
2   [448627.8]              0  
3   [758714.3]              0  
4  [513264.66]              0  
Getting arrow table file
Inference time.  Displaying results after.
                     time  \
0 2024-09-24 17:59:41.327   
1 2024-09-24 17:59:41.327   
2 2024-09-24 17:59:41.327   
3 2024-09-24 17:59:41.327   
4 2024-09-24 17:59:41.327   

                                                                                                               in.tensor  \
0     [4.0, 2.5, 2900.0, 5505.0, 2.0, 0.0, 0.0, 3.0, 8.0, 2900.0, 0.0, 47.6063, -122.02, 2970.0, 5251.0, 12.0, 0.0, 0.0]   
1     [2.0, 2.5, 2170.0, 6361.0, 1.0, 0.0, 2.0, 3.0, 8.0, 2170.0, 0.0, 47.7109, -122.017, 2310.0, 7419.0, 6.0, 0.0, 0.0]   
2      [3.0, 2.5, 1300.0, 812.0, 2.0, 0.0, 0.0, 3.0, 8.0, 880.0, 420.0, 47.5893, -122.317, 1300.0, 824.0, 6.0, 0.0, 0.0]   
3    [4.0, 2.5, 2500.0, 8540.0, 2.0, 0.0, 0.0, 3.0, 9.0, 2500.0, 0.0, 47.5759, -121.994, 2560.0, 8475.0, 24.0, 0.0, 0.0]   
4  [3.0, 1.75, 2200.0, 11520.0, 1.0, 0.0, 0.0, 4.0, 7.0, 2200.0, 0.0, 47.7659, -122.341, 1690.0, 8038.0, 62.0, 0.0, 0.0]   

  out.variable  anomaly.count  
0   [718013.7]              0  
1   [615094.6]              0  
2   [448627.8]              0  
3   [758714.3]              0  
4  [513264.66]              0  

Before we continue to the Run Continuously Task example, we’ll undeploy the pipeline since our Orchestration deploys the pipeline as part of the task cycle.

pipeline.undeploy()

Waiting for undeployment - this will take up to 45s ................................... ok

Wallaroo ML Workload Run Continuously Task Example

With the pipeline deployed and our connections set, we will now generate our ML Workload Orchestration. See the Wallaroo ML Workload Orchestrations guide for full details.

Orchestrations are uploaded to the Wallaroo instance as a ZIP file with the following requirements:

For our example, our orchestration will:

1. Upload Wallaroo ML Workload Orchestration with a task meant for Run Continuously Tasks. The instructions are:
2. Deploy a previously defined pipeline.
3. Every 1 minute, download data through the defined connection. This simulates using a database connection to retrieve the latest data from a data store.
4. Perform an inference with the downloaded data.

This sample script is stored in remote_inference/main.py with an empty requirements.txt file, and packaged into the orchestration as ./remote_inference/remote_inference.zip. We’ll display the steps in uploading the orchestration to the Wallaroo instance.

Upload the Orchestration

Orchestrations are uploaded with the Wallaroo client upload_orchestration(path) method with the following parameters.

Once uploaded, the deployment will be prepared and any requirements will be downloaded and installed.

For this example, the orchestration ./remote_inference/remote_inference.zip will be uploaded and saved to the variable orchestration.

Orchestration Status

We will loop until the uploaded orchestration’s status displays ready.

orchestration = wl.upload_orchestration(path="./remote_inference/remote_inference.zip")

while orchestration.status() != 'ready' and orchestration.status() != 'error':
    print(orchestration.status())
    time.sleep(5)

pending_packaging
pending_packaging
packaging
packaging
packaging
packaging
packaging
packaging
packaging
packaging

Task Management Tutorial

Once an Orchestration has the status ready, it can be run as a Task.

Tasks are created from an Orchestration through the following methods.

Run Continuously Task

We’ll create a Run Continuously Task from our Orchestration. This Task has an infinite while loop, so it will continue to loop until the Task is issued the kill command.

Run Continuously Tasks are generated Orchestration run_continuously(name, json_args) method. Any arguments for the orchestration are passed in through the json_args parameter as a Dict. If there are no arguments, then an empty set {} is passed.

# Example: run continuously

import datetime
task_start = datetime.datetime.now()
task_continuously = orchestration.run_continuously(name="run continuous sample", 
                              json_args={"workspace_name": workspace_name, 
                                         "pipeline_name": pipeline_name,
                                         "connection_name": connection_name
                                            })

Task Status

The list of tasks in the Wallaroo instance is retrieves through the Wallaroo Client list_tasks() method. This returns an array list of the following.

For this example, the status of the previously created task will be generated, then looped until it has reached status started.

while task_continuously.status() != "started":
    display(task_continuously.status())
    time.sleep(5)

Task Results

We can view the inferences from our logs and verify that new entries were added from our task. We can do that with the task last_runs() method to see the list of task runs executed, then show the log from the last completed task run.

# give time for the first task run to generate
time.sleep(30)
task_continuously.last_runs()

For this example, we’ll update the Task Run logs ever 20 seconds to show the loop is executing.

for x in range (10):  
    time.sleep(20)
    display(task_continuously.last_runs()[0].logs())

(no logs yet)

(no logs yet)

2024-24-Sep 20:17:10 Getting the workspace continuoustutorial
2024-24-Sep 20:17:10 Getting the pipeline continuouspipeline
2024-24-Sep 20:17:11 Deploy the pipeline.
2024-24-Sep 20:17:11 Deployment initiated for continuouspipeline. Please check pipeline status.
2024-24-Sep 20:17:11 {'status': 'Running', 'details': [], 'engines': [{'ip': '10.28.2.58', 'name': 'engine-77cb97f568-lpgk4', 'status': 'Running', 'reason': None, 'details': [], 'pipeline_statuses': {'pipelines': [{'id': 'continuouspipeline', 'status': 'Running', 'version': '0bb33df8-532b-46f6-990f-5bc3e69dd04e'}]}, 'model_statuses': {'models': [{'model_version_id': 7, 'name': 'houseprice', 'sha': 'e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6', 'status': 'Running', 'version': '5b7438ae-04f0-4398-8dae-3d7a0370e746'}]}}], 'engine_lbs': [{'ip': '10.28.2.57', 'name': 'engine-lb-6b59985857-tkx7c', 'status': 'Running', 'reason': None, 'details': []}], 'sidekicks': []}
2024-24-Sep 20:17:11 Inference time.  Displaying results after.
2024-24-Sep 20:17:11 Getting arrow table file
2024-24-Sep 20:17:11 0 2024-09-24 20:17:11.906  ...             0
2024-24-Sep 20:17:11 1 2024-09-24 20:17:11.906  ...             0
2024-24-Sep 20:17:11                      time  ... anomaly.count
2024-24-Sep 20:17:11 2 2024-09-24 20:17:11.906  ...             0
2024-24-Sep 20:17:11 3 2024-09-24 20:17:11.906  ...             0
2024-24-Sep 20:17:11 
2024-24-Sep 20:17:11 4 2024-09-24 20:17:11.906  ...             0
2024-24-Sep 20:17:11 [5 rows x 4 columns]
2024-24-Sep 20:18:11 Getting arrow table file
2024-24-Sep 20:18:12 Inference time.  Displaying results after.
2024-24-Sep 20:18:12                      time  ... anomaly.count
2024-24-Sep 20:18:12 0 2024-09-24 20:18:12.353  ...             0
2024-24-Sep 20:18:12 1 2024-09-24 20:18:12.353  ...             0
2024-24-Sep 20:18:12 2 2024-09-24 20:18:12.353  ...             0
2024-24-Sep 20:18:12 3 2024-09-24 20:18:12.353  ...             0
2024-24-Sep 20:18:12 4 2024-09-24 20:18:12.353  ...             0
2024-24-Sep 20:18:12 [5 rows x 4 columns]
2024-24-Sep 20:18:12 

2024-24-Sep 20:17:10 Getting the workspace continuoustutorial
2024-24-Sep 20:17:10 Getting the pipeline continuouspipeline
2024-24-Sep 20:17:11 Deploy the pipeline.
2024-24-Sep 20:17:11 Deployment initiated for continuouspipeline. Please check pipeline status.
2024-24-Sep 20:17:11 {'status': 'Running', 'details': [], 'engines': [{'ip': '10.28.2.58', 'name': 'engine-77cb97f568-lpgk4', 'status': 'Running', 'reason': None, 'details': [], 'pipeline_statuses': {'pipelines': [{'id': 'continuouspipeline', 'status': 'Running', 'version': '0bb33df8-532b-46f6-990f-5bc3e69dd04e'}]}, 'model_statuses': {'models': [{'model_version_id': 7, 'name': 'houseprice', 'sha': 'e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6', 'status': 'Running', 'version': '5b7438ae-04f0-4398-8dae-3d7a0370e746'}]}}], 'engine_lbs': [{'ip': '10.28.2.57', 'name': 'engine-lb-6b59985857-tkx7c', 'status': 'Running', 'reason': None, 'details': []}], 'sidekicks': []}
2024-24-Sep 20:17:11 Inference time.  Displaying results after.
2024-24-Sep 20:17:11 Getting arrow table file
2024-24-Sep 20:17:11 0 2024-09-24 20:17:11.906  ...             0
2024-24-Sep 20:17:11 1 2024-09-24 20:17:11.906  ...             0
2024-24-Sep 20:17:11                      time  ... anomaly.count
2024-24-Sep 20:17:11 2 2024-09-24 20:17:11.906  ...             0
2024-24-Sep 20:17:11 3 2024-09-24 20:17:11.906  ...             0
2024-24-Sep 20:17:11 
2024-24-Sep 20:17:11 4 2024-09-24 20:17:11.906  ...             0
2024-24-Sep 20:17:11 [5 rows x 4 columns]
2024-24-Sep 20:18:11 Getting arrow table file
2024-24-Sep 20:18:12 Inference time.  Displaying results after.
2024-24-Sep 20:18:12                      time  ... anomaly.count
2024-24-Sep 20:18:12 0 2024-09-24 20:18:12.353  ...             0
2024-24-Sep 20:18:12 1 2024-09-24 20:18:12.353  ...             0
2024-24-Sep 20:18:12 2 2024-09-24 20:18:12.353  ...             0
2024-24-Sep 20:18:12 3 2024-09-24 20:18:12.353  ...             0
2024-24-Sep 20:18:12 4 2024-09-24 20:18:12.353  ...             0
2024-24-Sep 20:18:12 [5 rows x 4 columns]
2024-24-Sep 20:18:12 

2024-24-Sep 20:17:10 Getting the workspace continuoustutorial
2024-24-Sep 20:17:10 Getting the pipeline continuouspipeline
2024-24-Sep 20:17:11 Deploy the pipeline.
2024-24-Sep 20:17:11 Deployment initiated for continuouspipeline. Please check pipeline status.
2024-24-Sep 20:17:11 {'status': 'Running', 'details': [], 'engines': [{'ip': '10.28.2.58', 'name': 'engine-77cb97f568-lpgk4', 'status': 'Running', 'reason': None, 'details': [], 'pipeline_statuses': {'pipelines': [{'id': 'continuouspipeline', 'status': 'Running', 'version': '0bb33df8-532b-46f6-990f-5bc3e69dd04e'}]}, 'model_statuses': {'models': [{'model_version_id': 7, 'name': 'houseprice', 'sha': 'e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6', 'status': 'Running', 'version': '5b7438ae-04f0-4398-8dae-3d7a0370e746'}]}}], 'engine_lbs': [{'ip': '10.28.2.57', 'name': 'engine-lb-6b59985857-tkx7c', 'status': 'Running', 'reason': None, 'details': []}], 'sidekicks': []}
2024-24-Sep 20:17:11 Inference time.  Displaying results after.
2024-24-Sep 20:17:11 Getting arrow table file
2024-24-Sep 20:17:11 0 2024-09-24 20:17:11.906  ...             0
2024-24-Sep 20:17:11 1 2024-09-24 20:17:11.906  ...             0
2024-24-Sep 20:17:11                      time  ... anomaly.count
2024-24-Sep 20:17:11 2 2024-09-24 20:17:11.906  ...             0
2024-24-Sep 20:17:11 3 2024-09-24 20:17:11.906  ...             0
2024-24-Sep 20:17:11 
2024-24-Sep 20:17:11 4 2024-09-24 20:17:11.906  ...             0
2024-24-Sep 20:17:11 [5 rows x 4 columns]
2024-24-Sep 20:18:11 Getting arrow table file
2024-24-Sep 20:18:12 Inference time.  Displaying results after.
2024-24-Sep 20:18:12                      time  ... anomaly.count
2024-24-Sep 20:18:12 0 2024-09-24 20:18:12.353  ...             0
2024-24-Sep 20:18:12 1 2024-09-24 20:18:12.353  ...             0
2024-24-Sep 20:18:12 2 2024-09-24 20:18:12.353  ...             0
2024-24-Sep 20:18:12 3 2024-09-24 20:18:12.353  ...             0
2024-24-Sep 20:18:12 4 2024-09-24 20:18:12.353  ...             0
2024-24-Sep 20:18:12 [5 rows x 4 columns]
2024-24-Sep 20:18:12 

2024-24-Sep 20:17:10 Getting the workspace continuoustutorial
2024-24-Sep 20:17:10 Getting the pipeline continuouspipeline
2024-24-Sep 20:17:11 Deploy the pipeline.
2024-24-Sep 20:17:11 Deployment initiated for continuouspipeline. Please check pipeline status.
2024-24-Sep 20:17:11 {'status': 'Running', 'details': [], 'engines': [{'ip': '10.28.2.58', 'name': 'engine-77cb97f568-lpgk4', 'status': 'Running', 'reason': None, 'details': [], 'pipeline_statuses': {'pipelines': [{'id': 'continuouspipeline', 'status': 'Running', 'version': '0bb33df8-532b-46f6-990f-5bc3e69dd04e'}]}, 'model_statuses': {'models': [{'model_version_id': 7, 'name': 'houseprice', 'sha': 'e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6', 'status': 'Running', 'version': '5b7438ae-04f0-4398-8dae-3d7a0370e746'}]}}], 'engine_lbs': [{'ip': '10.28.2.57', 'name': 'engine-lb-6b59985857-tkx7c', 'status': 'Running', 'reason': None, 'details': []}], 'sidekicks': []}
2024-24-Sep 20:17:11 Inference time.  Displaying results after.
2024-24-Sep 20:17:11 Getting arrow table file
2024-24-Sep 20:17:11                      time  ... anomaly.count
2024-24-Sep 20:17:11 1 2024-09-24 20:17:11.906  ...             0
2024-24-Sep 20:17:11 0 2024-09-24 20:17:11.906  ...             0
2024-24-Sep 20:17:11 3 2024-09-24 20:17:11.906  ...             0
2024-24-Sep 20:17:11 2 2024-09-24 20:17:11.906  ...             0
2024-24-Sep 20:17:11 4 2024-09-24 20:17:11.906  ...             0
2024-24-Sep 20:17:11 
2024-24-Sep 20:17:11 [5 rows x 4 columns]
2024-24-Sep 20:18:11 Getting arrow table file
2024-24-Sep 20:18:12 Inference time.  Displaying results after.
2024-24-Sep 20:18:12                      time  ... anomaly.count
2024-24-Sep 20:18:12 0 2024-09-24 20:18:12.353  ...             0
2024-24-Sep 20:18:12 1 2024-09-24 20:18:12.353  ...             0
2024-24-Sep 20:18:12 2 2024-09-24 20:18:12.353  ...             0
2024-24-Sep 20:18:12 3 2024-09-24 20:18:12.353  ...             0
2024-24-Sep 20:18:12 4 2024-09-24 20:18:12.353  ...             0
2024-24-Sep 20:18:12 [5 rows x 4 columns]
2024-24-Sep 20:18:12 
2024-24-Sep 20:19:12 Getting arrow table file
2024-24-Sep 20:19:12 Inference time.  Displaying results after.
2024-24-Sep 20:19:12                      time  ... anomaly.count
2024-24-Sep 20:19:12 0 2024-09-24 20:19:12.748  ...             0
2024-24-Sep 20:19:12 1 2024-09-24 20:19:12.748  ...             0
2024-24-Sep 20:19:12 2 2024-09-24 20:19:12.748  ...             0
2024-24-Sep 20:19:12 3 2024-09-24 20:19:12.748  ...             0
2024-24-Sep 20:19:12 4 2024-09-24 20:19:12.748  ...             0
2024-24-Sep 20:19:12 
2024-24-Sep 20:19:12 [5 rows x 4 columns]

2024-24-Sep 20:17:10 Getting the workspace continuoustutorial
2024-24-Sep 20:17:10 Getting the pipeline continuouspipeline
2024-24-Sep 20:17:11 Deploy the pipeline.
2024-24-Sep 20:17:11 Deployment initiated for continuouspipeline. Please check pipeline status.
2024-24-Sep 20:17:11 {'status': 'Running', 'details': [], 'engines': [{'ip': '10.28.2.58', 'name': 'engine-77cb97f568-lpgk4', 'status': 'Running', 'reason': None, 'details': [], 'pipeline_statuses': {'pipelines': [{'id': 'continuouspipeline', 'status': 'Running', 'version': '0bb33df8-532b-46f6-990f-5bc3e69dd04e'}]}, 'model_statuses': {'models': [{'model_version_id': 7, 'name': 'houseprice', 'sha': 'e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6', 'status': 'Running', 'version': '5b7438ae-04f0-4398-8dae-3d7a0370e746'}]}}], 'engine_lbs': [{'ip': '10.28.2.57', 'name': 'engine-lb-6b59985857-tkx7c', 'status': 'Running', 'reason': None, 'details': []}], 'sidekicks': []}
2024-24-Sep 20:17:11 Inference time.  Displaying results after.
2024-24-Sep 20:17:11 Getting arrow table file
2024-24-Sep 20:17:11                      time  ... anomaly.count
2024-24-Sep 20:17:11 1 2024-09-24 20:17:11.906  ...             0
2024-24-Sep 20:17:11 0 2024-09-24 20:17:11.906  ...             0
2024-24-Sep 20:17:11 3 2024-09-24 20:17:11.906  ...             0
2024-24-Sep 20:17:11 2 2024-09-24 20:17:11.906  ...             0
2024-24-Sep 20:17:11 4 2024-09-24 20:17:11.906  ...             0
2024-24-Sep 20:17:11 
2024-24-Sep 20:17:11 [5 rows x 4 columns]
2024-24-Sep 20:18:11 Getting arrow table file
2024-24-Sep 20:18:12 Inference time.  Displaying results after.
2024-24-Sep 20:18:12                      time  ... anomaly.count
2024-24-Sep 20:18:12 0 2024-09-24 20:18:12.353  ...             0
2024-24-Sep 20:18:12 1 2024-09-24 20:18:12.353  ...             0
2024-24-Sep 20:18:12 2 2024-09-24 20:18:12.353  ...             0
2024-24-Sep 20:18:12 3 2024-09-24 20:18:12.353  ...             0
2024-24-Sep 20:18:12 4 2024-09-24 20:18:12.353  ...             0
2024-24-Sep 20:18:12 [5 rows x 4 columns]
2024-24-Sep 20:18:12 
2024-24-Sep 20:19:12 Getting arrow table file
2024-24-Sep 20:19:12 Inference time.  Displaying results after.
2024-24-Sep 20:19:12                      time  ... anomaly.count
2024-24-Sep 20:19:12 0 2024-09-24 20:19:12.748  ...             0
2024-24-Sep 20:19:12 1 2024-09-24 20:19:12.748  ...             0
2024-24-Sep 20:19:12 2 2024-09-24 20:19:12.748  ...             0
2024-24-Sep 20:19:12 3 2024-09-24 20:19:12.748  ...             0
2024-24-Sep 20:19:12 4 2024-09-24 20:19:12.748  ...             0
2024-24-Sep 20:19:12 
2024-24-Sep 20:19:12 [5 rows x 4 columns]

2024-24-Sep 20:17:10 Getting the workspace continuoustutorial
2024-24-Sep 20:17:10 Getting the pipeline continuouspipeline
2024-24-Sep 20:17:11 Deploy the pipeline.
2024-24-Sep 20:17:11 Deployment initiated for continuouspipeline. Please check pipeline status.
2024-24-Sep 20:17:11 {'status': 'Running', 'details': [], 'engines': [{'ip': '10.28.2.58', 'name': 'engine-77cb97f568-lpgk4', 'status': 'Running', 'reason': None, 'details': [], 'pipeline_statuses': {'pipelines': [{'id': 'continuouspipeline', 'status': 'Running', 'version': '0bb33df8-532b-46f6-990f-5bc3e69dd04e'}]}, 'model_statuses': {'models': [{'model_version_id': 7, 'name': 'houseprice', 'sha': 'e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6', 'status': 'Running', 'version': '5b7438ae-04f0-4398-8dae-3d7a0370e746'}]}}], 'engine_lbs': [{'ip': '10.28.2.57', 'name': 'engine-lb-6b59985857-tkx7c', 'status': 'Running', 'reason': None, 'details': []}], 'sidekicks': []}
2024-24-Sep 20:17:11 Inference time.  Displaying results after.
2024-24-Sep 20:17:11 Getting arrow table file
2024-24-Sep 20:17:11                      time  ... anomaly.count
2024-24-Sep 20:17:11 1 2024-09-24 20:17:11.906  ...             0
2024-24-Sep 20:17:11 0 2024-09-24 20:17:11.906  ...             0
2024-24-Sep 20:17:11 3 2024-09-24 20:17:11.906  ...             0
2024-24-Sep 20:17:11 2 2024-09-24 20:17:11.906  ...             0
2024-24-Sep 20:17:11 4 2024-09-24 20:17:11.906  ...             0
2024-24-Sep 20:17:11 
2024-24-Sep 20:17:11 [5 rows x 4 columns]
2024-24-Sep 20:18:11 Getting arrow table file
2024-24-Sep 20:18:12 Inference time.  Displaying results after.
2024-24-Sep 20:18:12                      time  ... anomaly.count
2024-24-Sep 20:18:12 0 2024-09-24 20:18:12.353  ...             0
2024-24-Sep 20:18:12 1 2024-09-24 20:18:12.353  ...             0
2024-24-Sep 20:18:12 2 2024-09-24 20:18:12.353  ...             0
2024-24-Sep 20:18:12 3 2024-09-24 20:18:12.353  ...             0
2024-24-Sep 20:18:12 4 2024-09-24 20:18:12.353  ...             0
2024-24-Sep 20:18:12 [5 rows x 4 columns]
2024-24-Sep 20:18:12 
2024-24-Sep 20:19:12 Getting arrow table file
2024-24-Sep 20:19:12 Inference time.  Displaying results after.
2024-24-Sep 20:19:12                      time  ... anomaly.count
2024-24-Sep 20:19:12 0 2024-09-24 20:19:12.748  ...             0
2024-24-Sep 20:19:12 1 2024-09-24 20:19:12.748  ...             0
2024-24-Sep 20:19:12 2 2024-09-24 20:19:12.748  ...             0
2024-24-Sep 20:19:12 3 2024-09-24 20:19:12.748  ...             0
2024-24-Sep 20:19:12 4 2024-09-24 20:19:12.748  ...             0
2024-24-Sep 20:19:12 
2024-24-Sep 20:19:12 [5 rows x 4 columns]

2024-24-Sep 20:17:10 Getting the workspace continuoustutorial
2024-24-Sep 20:17:10 Getting the pipeline continuouspipeline
2024-24-Sep 20:17:11 Deploy the pipeline.
2024-24-Sep 20:17:11 Deployment initiated for continuouspipeline. Please check pipeline status.
2024-24-Sep 20:17:11 {'status': 'Running', 'details': [], 'engines': [{'ip': '10.28.2.58', 'name': 'engine-77cb97f568-lpgk4', 'status': 'Running', 'reason': None, 'details': [], 'pipeline_statuses': {'pipelines': [{'id': 'continuouspipeline', 'status': 'Running', 'version': '0bb33df8-532b-46f6-990f-5bc3e69dd04e'}]}, 'model_statuses': {'models': [{'model_version_id': 7, 'name': 'houseprice', 'sha': 'e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6', 'status': 'Running', 'version': '5b7438ae-04f0-4398-8dae-3d7a0370e746'}]}}], 'engine_lbs': [{'ip': '10.28.2.57', 'name': 'engine-lb-6b59985857-tkx7c', 'status': 'Running', 'reason': None, 'details': []}], 'sidekicks': []}
2024-24-Sep 20:17:11 Inference time.  Displaying results after.
2024-24-Sep 20:17:11 Getting arrow table file
2024-24-Sep 20:17:11                      time  ... anomaly.count
2024-24-Sep 20:17:11 1 2024-09-24 20:17:11.906  ...             0
2024-24-Sep 20:17:11 0 2024-09-24 20:17:11.906  ...             0
2024-24-Sep 20:17:11 2 2024-09-24 20:17:11.906  ...             0
2024-24-Sep 20:17:11 3 2024-09-24 20:17:11.906  ...             0
2024-24-Sep 20:17:11 4 2024-09-24 20:17:11.906  ...             0
2024-24-Sep 20:17:11 
2024-24-Sep 20:17:11 [5 rows x 4 columns]
2024-24-Sep 20:18:11 Getting arrow table file
2024-24-Sep 20:18:12 Inference time.  Displaying results after.
2024-24-Sep 20:18:12                      time  ... anomaly.count
2024-24-Sep 20:18:12 0 2024-09-24 20:18:12.353  ...             0
2024-24-Sep 20:18:12 1 2024-09-24 20:18:12.353  ...             0
2024-24-Sep 20:18:12 2 2024-09-24 20:18:12.353  ...             0
2024-24-Sep 20:18:12 3 2024-09-24 20:18:12.353  ...             0
2024-24-Sep 20:18:12 4 2024-09-24 20:18:12.353  ...             0
2024-24-Sep 20:18:12 
2024-24-Sep 20:18:12 [5 rows x 4 columns]
2024-24-Sep 20:19:12 Getting arrow table file
2024-24-Sep 20:19:12 Inference time.  Displaying results after.
2024-24-Sep 20:19:12                      time  ... anomaly.count
2024-24-Sep 20:19:12 0 2024-09-24 20:19:12.748  ...             0
2024-24-Sep 20:19:12 1 2024-09-24 20:19:12.748  ...             0
2024-24-Sep 20:19:12 2 2024-09-24 20:19:12.748  ...             0
2024-24-Sep 20:19:12 3 2024-09-24 20:19:12.748  ...             0
2024-24-Sep 20:19:12 4 2024-09-24 20:19:12.748  ...             0
2024-24-Sep 20:19:12 
2024-24-Sep 20:19:12 [5 rows x 4 columns]
2024-24-Sep 20:20:12 Getting arrow table file
2024-24-Sep 20:20:13 Inference time.  Displaying results after.
2024-24-Sep 20:20:13                      time  ... anomaly.count
2024-24-Sep 20:20:13 0 2024-09-24 20:20:13.119  ...             0
2024-24-Sep 20:20:13 1 2024-09-24 20:20:13.119  ...             0
2024-24-Sep 20:20:13 2 2024-09-24 20:20:13.119  ...             0
2024-24-Sep 20:20:13 3 2024-09-24 20:20:13.119  ...             0
2024-24-Sep 20:20:13 4 2024-09-24 20:20:13.119  ...             0
2024-24-Sep 20:20:13 
2024-24-Sep 20:20:13 [5 rows x 4 columns]

2024-24-Sep 20:17:10 Getting the workspace continuoustutorial
2024-24-Sep 20:17:10 Getting the pipeline continuouspipeline
2024-24-Sep 20:17:11 Deploy the pipeline.
2024-24-Sep 20:17:11 Deployment initiated for continuouspipeline. Please check pipeline status.
2024-24-Sep 20:17:11 {'status': 'Running', 'details': [], 'engines': [{'ip': '10.28.2.58', 'name': 'engine-77cb97f568-lpgk4', 'status': 'Running', 'reason': None, 'details': [], 'pipeline_statuses': {'pipelines': [{'id': 'continuouspipeline', 'status': 'Running', 'version': '0bb33df8-532b-46f6-990f-5bc3e69dd04e'}]}, 'model_statuses': {'models': [{'model_version_id': 7, 'name': 'houseprice', 'sha': 'e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6', 'status': 'Running', 'version': '5b7438ae-04f0-4398-8dae-3d7a0370e746'}]}}], 'engine_lbs': [{'ip': '10.28.2.57', 'name': 'engine-lb-6b59985857-tkx7c', 'status': 'Running', 'reason': None, 'details': []}], 'sidekicks': []}
2024-24-Sep 20:17:11 Inference time.  Displaying results after.
2024-24-Sep 20:17:11 Getting arrow table file
2024-24-Sep 20:17:11                      time  ... anomaly.count
2024-24-Sep 20:17:11 1 2024-09-24 20:17:11.906  ...             0
2024-24-Sep 20:17:11 0 2024-09-24 20:17:11.906  ...             0
2024-24-Sep 20:17:11 2 2024-09-24 20:17:11.906  ...             0
2024-24-Sep 20:17:11 3 2024-09-24 20:17:11.906  ...             0
2024-24-Sep 20:17:11 4 2024-09-24 20:17:11.906  ...             0
2024-24-Sep 20:17:11 
2024-24-Sep 20:17:11 [5 rows x 4 columns]
2024-24-Sep 20:18:11 Getting arrow table file
2024-24-Sep 20:18:12 Inference time.  Displaying results after.
2024-24-Sep 20:18:12                      time  ... anomaly.count
2024-24-Sep 20:18:12 0 2024-09-24 20:18:12.353  ...             0
2024-24-Sep 20:18:12 1 2024-09-24 20:18:12.353  ...             0
2024-24-Sep 20:18:12 2 2024-09-24 20:18:12.353  ...             0
2024-24-Sep 20:18:12 3 2024-09-24 20:18:12.353  ...             0
2024-24-Sep 20:18:12 4 2024-09-24 20:18:12.353  ...             0
2024-24-Sep 20:18:12 
2024-24-Sep 20:18:12 [5 rows x 4 columns]
2024-24-Sep 20:19:12 Getting arrow table file
2024-24-Sep 20:19:12 Inference time.  Displaying results after.
2024-24-Sep 20:19:12                      time  ... anomaly.count
2024-24-Sep 20:19:12 0 2024-09-24 20:19:12.748  ...             0
2024-24-Sep 20:19:12 1 2024-09-24 20:19:12.748  ...             0
2024-24-Sep 20:19:12 2 2024-09-24 20:19:12.748  ...             0
2024-24-Sep 20:19:12 3 2024-09-24 20:19:12.748  ...             0
2024-24-Sep 20:19:12 4 2024-09-24 20:19:12.748  ...             0
2024-24-Sep 20:19:12 
2024-24-Sep 20:19:12 [5 rows x 4 columns]
2024-24-Sep 20:20:12 Getting arrow table file
2024-24-Sep 20:20:13 Inference time.  Displaying results after.
2024-24-Sep 20:20:13                      time  ... anomaly.count
2024-24-Sep 20:20:13 0 2024-09-24 20:20:13.119  ...             0
2024-24-Sep 20:20:13 1 2024-09-24 20:20:13.119  ...             0
2024-24-Sep 20:20:13 2 2024-09-24 20:20:13.119  ...             0
2024-24-Sep 20:20:13 3 2024-09-24 20:20:13.119  ...             0
2024-24-Sep 20:20:13 4 2024-09-24 20:20:13.119  ...             0
2024-24-Sep 20:20:13 
2024-24-Sep 20:20:13 [5 rows x 4 columns]

Kill Task

With our example complete, we will kill the task and undeploy the pipeline.

task_continuously.kill()

<ArbexStatus.PENDING_KILL: ‘pending_kill’>

pipeline.undeploy()

Waiting for undeployment - this will take up to 45s .................................... ok