The following tutorial is available on the Wallaroo Github Repository.
This notebook will walk through building Wallaroo pipeline with a a Classification model deployed to detect the likelihood of credit card fraud, then publishing that pipeline to an Open Container Initiative (OCI) Registry where it can be deployed in other Docker and Kubernetes environments. This example uses the Wallaroo SDK.
This demonstration will focus on deployment to the edge.
This demonstration performs the following:
The first step is to import the libraries used in this notebook.
import wallaroo
from wallaroo.object import EntityNotFoundError
import pyarrow as pa
import pandas as pd
import requests
# used to display dataframe information without truncating
from IPython.display import display
pd.set_option('display.max_colwidth', None)
The next step is to connect to Wallaroo through the Wallaroo client. The Python library is included in the Wallaroo install and available through the Jupyter Hub interface provided with your Wallaroo environment.
This is accomplished using the wallaroo.Client()
command, which provides a URL to grant the SDK permission to your specific Wallaroo environment. When displayed, enter the URL into a browser and confirm permissions. Store the connection into a variable that can be referenced later.
If logging into the Wallaroo instance through the internal JupyterHub service, use wl = wallaroo.Client()
. For more information on Wallaroo Client settings, see the Client Connection guide.
wl = wallaroo.Client()
We’ll use the SDK below to create our workspace , assign as our current workspace, then display all of the workspaces we have at the moment. We’ll also set up variables for our models and pipelines down the road, so we have one spot to change names to whatever fits your organization’s standards best.
To allow this tutorial to be run by multiple users in the same Wallaroo instance, a random 4 character prefix will be added to the workspace, pipeline, and model. Feel free to set suffix=''
if this is not required.
workspace_name = f'edge-publish-demo'
pipeline_name = 'edge-pipeline'
xgboost_model_name = 'ccfraud-xgboost'
xgboost_model_file_name = './models/xgboost_ccfraud.onnx'
keras_model_name = 'ccfraud-keras'
keras_model_file_name = './models/keras_ccfraud.onnx'
workspace = wl.get_workspace(name=workspace_name, create_if_not_exist=True)
wl.set_current_workspace(workspace)
{'name': 'edge-publish-demo', 'id': 9, 'archived': False, 'created_by': '76b893ff-5c30-4f01-bd9e-9579a20fc4ea', 'created_at': '2024-04-22T18:08:18.961066+00:00', 'models': [], 'pipelines': []}
When a model is uploaded to a Wallaroo cluster, it is optimized and packaged to make it ready to run as part of a pipeline. In many times, the Wallaroo Server can natively run a model without any Python overhead. In other cases, such as a Python script, a custom Python environment will be automatically generated. This is comparable to the process of “containerizing” a model by adding a small HTTP server and other wrapping around it.
Our pretrained model is in ONNX format, which is specified in the framework
parameter.
xgboost_edge_demo_model = wl.upload_model(
xgboost_model_name,
xgboost_model_file_name,
framework=wallaroo.framework.Framework.ONNX,
).configure(tensor_fields=["tensor"])
keras_edge_demo_model = wl.upload_model(
keras_model_name,
keras_model_file_name,
framework=wallaroo.framework.Framework.ONNX,
).configure(tensor_fields=["tensor"])
Before deploying an inference engine we need to tell wallaroo what resources it will need.
To do this we will use the wallaroo DeploymentConfigBuilder() and fill in the options listed below to determine what the properties of our inference engine will be.
We will be testing this deployment for an edge scenario, so the resource specifications are kept small – what’s the minimum needed to meet the expected load on the planned hardware.
deploy_config = (wallaroo
.DeploymentConfigBuilder()
.replica_count(1)
.cpus(1)
.memory("900Mi")
.build()
)
We will now deploy our pipeline into the current Kubernetes environment using the specified resource constraints. This is a “simulated edge” deploy in that we try to mimic the edge hardware as closely as possible.
pipeline = wl.build_pipeline(pipeline_name)
# clear the pipeline if previously run
pipeline.clear()
pipeline.add_model_step(xgboost_edge_demo_model)
_ = pipeline.deploy(deployment_config = deploy_config)
# get this pipeline version
pipeline_xgb_version = wl.get_pipeline(pipeline_name)
pipeline_xgb_version
name | edge-pipeline |
---|---|
created | 2024-04-22 18:08:21.762528+00:00 |
last_updated | 2024-04-22 18:11:45.853553+00:00 |
deployed | True |
arch | x86 |
accel | none |
tags | |
versions | e5964f75-ed49-4115-afa0-a9cca6ee50af, c52a053f-8ff8-47f1-aa13-a2c1711ec773, 068e0163-0ec5-4a5f-9ab8-f1939b743142, 0a81ccbe-84aa-4230-a7d2-ab079b6be605, 8975c2ec-ddfa-4cf5-afa4-538a2b3fddfd, 575ed34b-5435-4cf3-80be-99c0f832d83a, ee5783ac-a620-4e3e-885a-ed92718a00ef |
steps | ccfraud-xgboost |
published | True |
A single image, encoded using the Apache Arrow format, is sent to the deployed pipeline. Arrow is used here because, as a binary protocol, there is far lower network and compute overhead than using JSON. The Wallaroo Server engine accepts both JSON, pandas DataFrame, and Apache Arrow formats.
The sample DataFrames and arrow tables are in the ./data
directory. We’ll use the Apache Arrow table cc_data_10k.arrow
.
deploy_url = pipeline._deployment._url()
headers = wl.auth.auth_header()
headers['Content-Type']='application/vnd.apache.arrow.file'
# headers['Content-Type']='application/json; format=pandas-records'
headers['Accept']='application/json; format=pandas-records'
dataFile = './data/cc_data_10k.arrow'
!curl -X POST {deploy_url} \
-H "Authorization:{headers['Authorization']}" \
-H "Content-Type:{headers['Content-Type']}" \
-H "Accept:{headers['Accept']}" \
--data-binary @{dataFile} > curl_response_xgboost.df.json
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 8192k 100 7029k 100 1162k 2634k 435k 0:00:02 0:00:02 --:--:-- 3070k6k 865k 0:00:04 0:00:01 0:00:03 2272k00:02 0:00:02 --:--:-- 2846k
We’ll retrieve the DataFrame from our recent inference and display the first 5 rows.
df = pd.read_json('curl_response_xgboost.df.json', orient="records")
df.head(5)
time | in | out | anomaly | metadata | |
---|---|---|---|---|---|
0 | 1713809545130 | {'tensor': [-1.0603298, 2.3544967, -3.5638788, 5.138735, -1.2308457, -0.7687824400000001, -3.5881228, 1.8880838, -3.2789674, -3.9563255, 4.099344, -5.653918, -0.8775733, -9.131571, -0.6093538, -3.7480276, -5.0309124, -0.8748149, 1.9870535, 0.7005486, 0.9204422999999999, -0.10414918000000001, 0.32295644, -0.74181414, 0.038412016, 1.0993439, 1.2603409, -0.14662448, -1.4463212]} | {'variable': [1.0094898]} | {'count': 0} | {'last_model': '{"model_name":"ccfraud-xgboost","model_sha":"054810e3e3ebbdd34438d9c1a08ed6a6680ef10bf97b9223f78ebf38e14b3b52"}', 'pipeline_version': 'e5964f75-ed49-4115-afa0-a9cca6ee50af', 'elapsed': [783790, 8065979], 'dropped': [], 'partition': 'engine-b4d4c476d-jqhw7'} |
1 | 1713809545130 | {'tensor': [-1.0603298, 2.3544967, -3.5638788, 5.138735, -1.2308457, -0.7687824400000001, -3.5881228, 1.8880838, -3.2789674, -3.9563255, 4.099344, -5.653918, -0.8775733, -9.131571, -0.6093538, -3.7480276, -5.0309124, -0.8748149, 1.9870535, 0.7005486, 0.9204422999999999, -0.10414918000000001, 0.32295644, -0.74181414, 0.038412016, 1.0993439, 1.2603409, -0.14662448, -1.4463212]} | {'variable': [1.0094898]} | {'count': 0} | {'last_model': '{"model_name":"ccfraud-xgboost","model_sha":"054810e3e3ebbdd34438d9c1a08ed6a6680ef10bf97b9223f78ebf38e14b3b52"}', 'pipeline_version': 'e5964f75-ed49-4115-afa0-a9cca6ee50af', 'elapsed': [783790, 8065979], 'dropped': [], 'partition': 'engine-b4d4c476d-jqhw7'} |
2 | 1713809545130 | {'tensor': [-1.0603298, 2.3544967, -3.5638788, 5.138735, -1.2308457, -0.7687824400000001, -3.5881228, 1.8880838, -3.2789674, -3.9563255, 4.099344, -5.653918, -0.8775733, -9.131571, -0.6093538, -3.7480276, -5.0309124, -0.8748149, 1.9870535, 0.7005486, 0.9204422999999999, -0.10414918000000001, 0.32295644, -0.74181414, 0.038412016, 1.0993439, 1.2603409, -0.14662448, -1.4463212]} | {'variable': [1.0094898]} | {'count': 0} | {'last_model': '{"model_name":"ccfraud-xgboost","model_sha":"054810e3e3ebbdd34438d9c1a08ed6a6680ef10bf97b9223f78ebf38e14b3b52"}', 'pipeline_version': 'e5964f75-ed49-4115-afa0-a9cca6ee50af', 'elapsed': [783790, 8065979], 'dropped': [], 'partition': 'engine-b4d4c476d-jqhw7'} |
3 | 1713809545130 | {'tensor': [-1.0603298, 2.3544967, -3.5638788, 5.138735, -1.2308457, -0.7687824400000001, -3.5881228, 1.8880838, -3.2789674, -3.9563255, 4.099344, -5.653918, -0.8775733, -9.131571, -0.6093538, -3.7480276, -5.0309124, -0.8748149, 1.9870535, 0.7005486, 0.9204422999999999, -0.10414918000000001, 0.32295644, -0.74181414, 0.038412016, 1.0993439, 1.2603409, -0.14662448, -1.4463212]} | {'variable': [1.0094898]} | {'count': 0} | {'last_model': '{"model_name":"ccfraud-xgboost","model_sha":"054810e3e3ebbdd34438d9c1a08ed6a6680ef10bf97b9223f78ebf38e14b3b52"}', 'pipeline_version': 'e5964f75-ed49-4115-afa0-a9cca6ee50af', 'elapsed': [783790, 8065979], 'dropped': [], 'partition': 'engine-b4d4c476d-jqhw7'} |
4 | 1713809545130 | {'tensor': [0.5817662, 0.09788155, 0.15468194, 0.4754102, -0.19788623, -0.45043448, 0.016654044, -0.025607055, 0.09205616, -0.27839172, 0.059329946, -0.019658541, -0.42250833, -0.12175389, 1.5473095, 0.23916228, 0.3553975, -0.76851654, -0.7000849, -0.11900433, -0.34505169999999996, -1.1065114, 0.25234112000000003, 0.020944182000000002, 0.21992674, 0.25406894, -0.04502251, 0.10867739, 0.25471792]} | {'variable': [-1.9073485999999998e-06]} | {'count': 0} | {'last_model': '{"model_name":"ccfraud-xgboost","model_sha":"054810e3e3ebbdd34438d9c1a08ed6a6680ef10bf97b9223f78ebf38e14b3b52"}', 'pipeline_version': 'e5964f75-ed49-4115-afa0-a9cca6ee50af', 'elapsed': [783790, 8065979], 'dropped': [], 'partition': 'engine-b4d4c476d-jqhw7'} |
We will now redeploy the pipeline with the converted keras version of our model, then set that version of the pipeline.
# clear the pipeline if previously run
pipeline.clear()
pipeline.add_model_step(keras_edge_demo_model)
_ = pipeline.deploy(deployment_config = deploy_config)
# give a moment to verify the model swap
import time
time.sleep(10)
# get this version
keras_pipeline_version = wl.get_pipeline(pipeline_name)
keras_pipeline_version
name | edge-pipeline |
---|---|
created | 2024-04-22 18:08:21.762528+00:00 |
last_updated | 2024-04-22 18:13:19.120099+00:00 |
deployed | True |
arch | x86 |
accel | none |
tags | |
versions | 5f492ec9-feb9-497d-933c-8f1018e29d8d, 2e75cba0-12df-45d2-8386-3f8c3e236b0d, e5964f75-ed49-4115-afa0-a9cca6ee50af, c52a053f-8ff8-47f1-aa13-a2c1711ec773, 068e0163-0ec5-4a5f-9ab8-f1939b743142, 0a81ccbe-84aa-4230-a7d2-ab079b6be605, 8975c2ec-ddfa-4cf5-afa4-538a2b3fddfd, 575ed34b-5435-4cf3-80be-99c0f832d83a, ee5783ac-a620-4e3e-885a-ed92718a00ef |
steps | ccfraud-keras |
published | True |
We now run the same inference request
deploy_url = pipeline._deployment._url()
headers = wl.auth.auth_header()
headers['Content-Type']='application/vnd.apache.arrow.file'
headers['Accept']='application/json; format=pandas-records'
dataFile = './data/cc_data_10k.arrow'
import datetime
local_inference_start = datetime.datetime.now()
!curl -X POST {deploy_url} \
-H "Authorization:{headers['Authorization']}" \
-H "Content-Type:{headers['Content-Type']}" \
-H "Accept:{headers['Accept']}" \
--data-binary @{dataFile} > curl_response_keras.df.json
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 8161k 100 6999k 100 1162k 4754k 789k 0:00:01 0:00:01 --:--:-- 5544k
We’ll retrieve the DataFrame from our recent inference and display the first 5 rows.
df = pd.read_json('curl_response_keras.df.json', orient="records")
df.head(5)
time | in | out | anomaly | metadata | |
---|---|---|---|---|---|
0 | 1713809331155 | {'tensor': [-1.0603298, 2.3544967, -3.5638788, 5.138735, -1.2308457, -0.7687824400000001, -3.5881228, 1.8880838, -3.2789674, -3.9563255, 4.099344, -5.653918, -0.8775733, -9.131571, -0.6093538, -3.7480276, -5.0309124, -0.8748149, 1.9870535, 0.7005486, 0.9204422999999999, -0.10414918000000001, 0.32295644, -0.74181414, 0.038412016, 1.0993439, 1.2603409, -0.14662448, -1.4463212]} | {'dense_1': [0.99300325]} | {'count': 0} | {'last_model': '{"model_name":"ccfraud-keras","model_sha":"bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507"}', 'pipeline_version': '8975c2ec-ddfa-4cf5-afa4-538a2b3fddfd', 'elapsed': [672579, 1339770], 'dropped': [], 'partition': 'engine-5cf5d688cf-8rpqc'} |
1 | 1713809331155 | {'tensor': [-1.0603298, 2.3544967, -3.5638788, 5.138735, -1.2308457, -0.7687824400000001, -3.5881228, 1.8880838, -3.2789674, -3.9563255, 4.099344, -5.653918, -0.8775733, -9.131571, -0.6093538, -3.7480276, -5.0309124, -0.8748149, 1.9870535, 0.7005486, 0.9204422999999999, -0.10414918000000001, 0.32295644, -0.74181414, 0.038412016, 1.0993439, 1.2603409, -0.14662448, -1.4463212]} | {'dense_1': [0.99300325]} | {'count': 0} | {'last_model': '{"model_name":"ccfraud-keras","model_sha":"bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507"}', 'pipeline_version': '8975c2ec-ddfa-4cf5-afa4-538a2b3fddfd', 'elapsed': [672579, 1339770], 'dropped': [], 'partition': 'engine-5cf5d688cf-8rpqc'} |
2 | 1713809331155 | {'tensor': [-1.0603298, 2.3544967, -3.5638788, 5.138735, -1.2308457, -0.7687824400000001, -3.5881228, 1.8880838, -3.2789674, -3.9563255, 4.099344, -5.653918, -0.8775733, -9.131571, -0.6093538, -3.7480276, -5.0309124, -0.8748149, 1.9870535, 0.7005486, 0.9204422999999999, -0.10414918000000001, 0.32295644, -0.74181414, 0.038412016, 1.0993439, 1.2603409, -0.14662448, -1.4463212]} | {'dense_1': [0.99300325]} | {'count': 0} | {'last_model': '{"model_name":"ccfraud-keras","model_sha":"bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507"}', 'pipeline_version': '8975c2ec-ddfa-4cf5-afa4-538a2b3fddfd', 'elapsed': [672579, 1339770], 'dropped': [], 'partition': 'engine-5cf5d688cf-8rpqc'} |
3 | 1713809331155 | {'tensor': [-1.0603298, 2.3544967, -3.5638788, 5.138735, -1.2308457, -0.7687824400000001, -3.5881228, 1.8880838, -3.2789674, -3.9563255, 4.099344, -5.653918, -0.8775733, -9.131571, -0.6093538, -3.7480276, -5.0309124, -0.8748149, 1.9870535, 0.7005486, 0.9204422999999999, -0.10414918000000001, 0.32295644, -0.74181414, 0.038412016, 1.0993439, 1.2603409, -0.14662448, -1.4463212]} | {'dense_1': [0.99300325]} | {'count': 0} | {'last_model': '{"model_name":"ccfraud-keras","model_sha":"bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507"}', 'pipeline_version': '8975c2ec-ddfa-4cf5-afa4-538a2b3fddfd', 'elapsed': [672579, 1339770], 'dropped': [], 'partition': 'engine-5cf5d688cf-8rpqc'} |
4 | 1713809331155 | {'tensor': [0.5817662, 0.09788155, 0.15468194, 0.4754102, -0.19788623, -0.45043448, 0.016654044, -0.025607055, 0.09205616, -0.27839172, 0.059329946, -0.019658541, -0.42250833, -0.12175389, 1.5473095, 0.23916228, 0.3553975, -0.76851654, -0.7000849, -0.11900433, -0.34505169999999996, -1.1065114, 0.25234112000000003, 0.020944182000000002, 0.21992674, 0.25406894, -0.04502251, 0.10867739, 0.25471792]} | {'dense_1': [0.0010916889]} | {'count': 0} | {'last_model': '{"model_name":"ccfraud-keras","model_sha":"bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507"}', 'pipeline_version': '8975c2ec-ddfa-4cf5-afa4-538a2b3fddfd', 'elapsed': [672579, 1339770], 'dropped': [], 'partition': 'engine-5cf5d688cf-8rpqc'} |
With our testing complete, we will undeploy the pipeline and return the resources back to the cluster.
pipeline.undeploy()
name | edge-pipeline |
---|---|
created | 2024-04-22 18:08:21.762528+00:00 |
last_updated | 2024-04-22 18:13:19.120099+00:00 |
deployed | False |
arch | x86 |
accel | none |
tags | |
versions | 5f492ec9-feb9-497d-933c-8f1018e29d8d, 2e75cba0-12df-45d2-8386-3f8c3e236b0d, e5964f75-ed49-4115-afa0-a9cca6ee50af, c52a053f-8ff8-47f1-aa13-a2c1711ec773, 068e0163-0ec5-4a5f-9ab8-f1939b743142, 0a81ccbe-84aa-4230-a7d2-ab079b6be605, 8975c2ec-ddfa-4cf5-afa4-538a2b3fddfd, 575ed34b-5435-4cf3-80be-99c0f832d83a, ee5783ac-a620-4e3e-885a-ed92718a00ef |
steps | ccfraud-keras |
published | True |
It worked! For a demo, we’ll take working once as “tested”. So now that we’ve tested our pipeline, we are ready to publish it for edge deployment.
Publishing it means assembling all of the configuration files and model assets and pushing them to an Open Container Initiative (OCI) repository set in the Wallaroo instance as the Edge Registry service. DevOps engineers then retrieve that image and deploy it through Docker, Kubernetes, or similar deployments.
See Edge Deployment Registry Guide for details on adding an OCI Registry Service to Wallaroo as the Edge Deployment Registry.
This is done through the SDK command wallaroo.pipeline.publish(deployment_config)
which has the following parameters and returns.
The publish
method takes the following parameters. The containerized pipeline will be pushed to the Edge registry service with the model, pipeline configurations, and other artifacts needed to deploy the pipeline.
Parameter | Type | Description |
---|---|---|
deployment_config | wallaroo.deployment_config.DeploymentConfig (Optional) | Sets the pipeline deployment configuration. For example: For more information on pipeline deployment configuration, see the Wallaroo SDK Essentials Guide: Pipeline Deployment Configuration. |
Field | Type | Description |
---|---|---|
id | integer | Numerical Wallaroo id of the published pipeline. |
pipeline version id | integer | Numerical Wallaroo id of the pipeline version published. |
status | string | The status of the pipeline publication. Values include:
|
Engine URL | string | The URL of the published pipeline engine in the edge registry. |
Pipeline URL | string | The URL of the published pipeline in the edge registry. |
Helm Chart URL | string | The URL of the helm chart for the published pipeline in the edge registry. |
Helm Chart Reference | string | The help chart reference. |
Helm Chart Version | string | The version of the Helm Chart of the published pipeline. This is also used as the Docker tag. |
Engine Config | wallaroo.deployment_config.DeploymentConfig | The pipeline configuration included with the published pipeline. |
Created At | DateTime | When the published pipeline was created. |
Updated At | DateTime | When the published pipeline was updated. |
We will now publish the pipeline to our Edge Deployment Registry with the pipeline.publish(deployment_config)
command. deployment_config
is an optional field that specifies the pipeline deployment. This can be overridden by the DevOps engineer during deployment.
xgb_pub=pipeline_xgb_version.publish(deploy_config)
display(xgb_pub)
keras_pub=keras_pipeline_version.publish(deploy_config)
display(keras_pub)
Waiting for pipeline publish... It may take up to 600 sec.
Pipeline is publishing...... Published.
ID | 9 | |
Pipeline Name | edge-pipeline | |
Pipeline Version | 98795ff5-2e56-465a-9041-77e426d8fd19 | |
Status | Published | |
Engine URL | ghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/fitzroy-mini:v2024.1.0-main-4963 | |
Pipeline URL | ghcr.io/wallaroolabs/doc-samples/pipelines/edge-pipeline:98795ff5-2e56-465a-9041-77e426d8fd19 | |
Helm Chart URL | oci://ghcr.io/wallaroolabs/doc-samples/charts/edge-pipeline | |
Helm Chart Reference | ghcr.io/wallaroolabs/doc-samples/charts@sha256:c2839e79777cb68d9683a85f890464f7430e277368bcf1629dca338c635015cc | |
Helm Chart Version | 0.0.1-98795ff5-2e56-465a-9041-77e426d8fd19 | |
Engine Config | {'engine': {'resources': {'limits': {'cpu': 1.0, 'memory': '512Mi'}, 'requests': {'cpu': 1.0, 'memory': '512Mi'}, 'accel': 'none', 'arch': 'x86', 'gpu': False}}, 'engineAux': {'autoscale': {'type': 'none'}, 'images': {}}} | |
User Images | [] | |
Created By | john.hummel@wallaroo.ai | |
Created At | 2024-04-22 18:16:33.020137+00:00 | |
Updated At | 2024-04-22 18:16:33.020137+00:00 | |
Replaces | ||
Docker Run Command |
Note: Please set the EDGE_PORT , OCI_USERNAME , and OCI_PASSWORD environment variables. | |
Helm Install Command |
Note: Please set the HELM_INSTALL_NAME , HELM_INSTALL_NAMESPACE ,
OCI_USERNAME , and OCI_PASSWORD environment variables. |
Waiting for pipeline publish... It may take up to 600 sec.
Pipeline is publishing....... Published.
ID | 10 | |
Pipeline Name | edge-pipeline | |
Pipeline Version | 5a85f1c8-2a56-4887-9b1e-13f44968d390 | |
Status | Published | |
Engine URL | ghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/fitzroy-mini:v2024.1.0-main-4963 | |
Pipeline URL | ghcr.io/wallaroolabs/doc-samples/pipelines/edge-pipeline:5a85f1c8-2a56-4887-9b1e-13f44968d390 | |
Helm Chart URL | oci://ghcr.io/wallaroolabs/doc-samples/charts/edge-pipeline | |
Helm Chart Reference | ghcr.io/wallaroolabs/doc-samples/charts@sha256:e7da76da3996d2993171e9bdc23f71fd92fb92848bbca3aa8bd93ea4c62ef31f | |
Helm Chart Version | 0.0.1-5a85f1c8-2a56-4887-9b1e-13f44968d390 | |
Engine Config | {'engine': {'resources': {'limits': {'cpu': 1.0, 'memory': '512Mi'}, 'requests': {'cpu': 1.0, 'memory': '512Mi'}, 'accel': 'none', 'arch': 'x86', 'gpu': False}}, 'engineAux': {'autoscale': {'type': 'none'}, 'images': {}}} | |
User Images | [] | |
Created By | john.hummel@wallaroo.ai | |
Created At | 2024-04-22 18:17:02.879740+00:00 | |
Updated At | 2024-04-22 18:17:02.879740+00:00 | |
Replaces | ||
Docker Run Command |
Note: Please set the EDGE_PORT , OCI_USERNAME , and OCI_PASSWORD environment variables. | |
Helm Install Command |
Note: Please set the HELM_INSTALL_NAME , HELM_INSTALL_NAMESPACE ,
OCI_USERNAME , and OCI_PASSWORD environment variables. |
The method wallaroo.client.list_pipelines()
shows a list of all pipelines in the Wallaroo instance, and includes the published
field that indicates whether the pipeline was published to the registry (True
), or has not yet been published (False
).
wl.list_pipelines()
name | created | last_updated | deployed | arch | tags | versions | steps | published | |
---|---|---|---|---|---|---|---|---|---|
edge-pipeline | 2024-22-Apr 18:08:21 | 2024-22-Apr 18:17:01 | False | x86 | none | 5a85f1c8-2a56-4887-9b1e-13f44968d390, 98795ff5-2e56-465a-9041-77e426d8fd19, 5f492ec9-feb9-497d-933c-8f1018e29d8d, 2e75cba0-12df-45d2-8386-3f8c3e236b0d, e5964f75-ed49-4115-afa0-a9cca6ee50af, c52a053f-8ff8-47f1-aa13-a2c1711ec773, 068e0163-0ec5-4a5f-9ab8-f1939b743142, 0a81ccbe-84aa-4230-a7d2-ab079b6be605, 8975c2ec-ddfa-4cf5-afa4-538a2b3fddfd, 575ed34b-5435-4cf3-80be-99c0f832d83a, ee5783ac-a620-4e3e-885a-ed92718a00ef | ccfraud-keras | True | |
edge-low-connection-demo | 2024-22-Apr 17:53:56 | 2024-22-Apr 17:54:12 | True | x86 | none | cc4a10a2-4866-4ed6-a1d8-8b731e7bf8f8, 758d7e59-38c7-4fe7-b9b6-0b94bb902312, 0488841d-02d4-42e6-8b8f-387a5d42cc9d | rf-house-price-estimator | True | |
edge-pipeline-classification-cybersecurity | 2024-22-Apr 17:04:23 | 2024-22-Apr 17:05:19 | False | x86 | none | 21d4aa33-3a9a-4e55-9feb-6689eb822b7a, 93ca84ca-9121-4933-b44c-95539fe5c0ee, 3f585deb-71bd-4a0a-b08b-4970d4e60a77 | aloha | True | |
vgg16-clustering-pipeline | 2024-22-Apr 16:17:08 | 2024-22-Apr 16:48:01 | False | x86 | none | 7a7509d5-c30b-4f33-82ae-49deaf79dbd1, 29d94f80-3c21-44fb-9e71-a5498c3bce3d, 412b8da5-ad4c-417c-9f6e-ad79d71522a4, 4233c4e7-517a-48e8-807a-b626834f45ec, 4ca1d45a-507d-42e2-8038-d608c543681a, a99f0a28-ad9e-4db3-9eea-113bdd9ca1cd, be19886c-3896-47d5-9935-35592f44ad7c | vgg16-clustering | True | |
new-edge-inline-replacement | 2024-22-Apr 15:43:04 | 2024-22-Apr 15:44:46 | False | x86 | none | 455a7840-08c3-43bb-b6a6-7535894e6055, 5210e01e-6d0f-4cdc-92ea-d3499bcc42fc, d61fcf2d-95ad-41e7-9e53-50610d9e0419 | gbr-house-price-estimator | True | |
edge-inline-replacement-demon | 2024-22-Apr 15:27:36 | 2024-22-Apr 15:40:50 | False | x86 | none | c6a1e945-7de0-4f2c-addb-4f4746114a86, 0ad2e53e-6c00-4949-8bd4-08ae289430d5, 7c702eca-8acf-45e0-bdcd-bbb48a5102e5, 2ef51c5c-bc58-49b3-9ecf-9aa4bb0a0bae, fbc4bf00-d97f-4be1-a47c-85c788dd90d5 | xgb-house-price-estimator | True |
All publishes created from a pipeline are displayed with the wallaroo.pipeline.publishes
method. The pipeline_version_id
is used to know what version of the pipeline was used in that specific publish. This allows for pipelines to be updated over time, and newer versions to be sent and tracked to the Edge Deployment Registry service.
N/A
A List of the following fields:
Field | Type | Description |
---|---|---|
id | integer | Numerical Wallaroo id of the published pipeline. |
pipeline_version_id | integer | Numerical Wallaroo id of the pipeline version published. |
engine_url | string | The URL of the published pipeline engine in the edge registry. |
pipeline_url | string | The URL of the published pipeline in the edge registry. |
created_by | string | The email address of the user that published the pipeline. |
Created At | DateTime | When the published pipeline was created. |
Updated At | DateTime | When the published pipeline was updated. |
pipeline.publishes()
Once a pipeline is deployed to the Edge Registry service, it can be deployed in environments such as Docker, Kubernetes, or similar container running services by a DevOps engineer as a Wallaroo Server.
The following guides will demonstrate publishing a Wallaroo Pipeline as a Wallaroo Server.
Wallaroo Servers can optionally connect to the Wallaroo Ops instance and transmit their inference results. These are added to the pipeline logs for the published pipeline the Wallaroo Server is associated with.
Wallaroo Servers are added with the wallaroo.pipeline_publish.add_edge(name: string)
method. The name
is the unique primary key for each edge added to the pipeline publish and must be unique.
This returns a Publish Edge with the following fields:
Field | Type | Description |
---|---|---|
id | Integer | The integer ID of the pipeline publish. |
created_at | DateTime | The DateTime of the pipeline publish. |
docker_run_variables | String | The Docker variables in UUID format that include the following: The BUNDLE_VERSION , EDGE_NAME , JOIN_TOKEN_ , OPSCENTER_HOST , PIPELINE_URL , and WORKSPACE_ID . |
engine_config | String | The Wallaroo wallaroo.deployment_config.DeploymentConfig for the pipeline. |
pipeline_version_id | Integer | The integer identifier of the pipeline version published. |
status | String | The status of the publish. Published is a successful publish. |
updated_at | DateTime | The DateTime when the pipeline publish was updated. |
user_images | List(String) | User images used in the pipeline publish. |
created_by | String | The UUID of the Wallaroo user that created the pipeline publish. |
engine_url | String | The URL for the published pipeline’s Wallaroo engine in the OCI registry. |
error | String | Any errors logged. |
helm | String | The helm chart, helm reference and helm version. |
pipeline_url | String | The URL for the published pipeline’s container in the OCI registry. |
pipeline_version_name | String | The UUID identifier of the pipeline version published. |
additional_properties | String | Any other identities. |
xgb_edge = xgb_pub.add_edge("xgb-ccfraud-edge-publish")
display(xgb_edge)
keras_edge = keras_pub.add_edge("keras-ccfraud-publish")
display(keras_edge)
ID | 9 | |
Pipeline Name | edge-pipeline | |
Pipeline Version | 98795ff5-2e56-465a-9041-77e426d8fd19 | |
Status | Published | |
Engine URL | ghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/fitzroy-mini:v2024.1.0-main-4963 | |
Pipeline URL | ghcr.io/wallaroolabs/doc-samples/pipelines/edge-pipeline:98795ff5-2e56-465a-9041-77e426d8fd19 | |
Helm Chart URL | oci://ghcr.io/wallaroolabs/doc-samples/charts/edge-pipeline | |
Helm Chart Reference | ghcr.io/wallaroolabs/doc-samples/charts@sha256:c2839e79777cb68d9683a85f890464f7430e277368bcf1629dca338c635015cc | |
Helm Chart Version | 0.0.1-98795ff5-2e56-465a-9041-77e426d8fd19 | |
Engine Config | {'engine': {'resources': {'limits': {'cpu': 1.0, 'memory': '512Mi'}, 'requests': {'cpu': 1.0, 'memory': '512Mi'}, 'accel': 'none', 'arch': 'x86', 'gpu': False}}, 'engineAux': {'autoscale': {'type': 'none'}, 'images': {}}} | |
User Images | [] | |
Created By | john.hummel@wallaroo.ai | |
Created At | 2024-04-22 18:16:33.020137+00:00 | |
Updated At | 2024-04-22 18:16:33.020137+00:00 | |
Replaces | ||
Docker Run Command |
Note: Please set the PERSISTENT_VOLUME_DIR , EDGE_PORT , OCI_USERNAME , and OCI_PASSWORD environment variables. | |
Helm Install Command |
Note: Please set the HELM_INSTALL_NAME , HELM_INSTALL_NAMESPACE ,
OCI_USERNAME , and OCI_PASSWORD environment variables. |
ID | 10 | |
Pipeline Name | edge-pipeline | |
Pipeline Version | 5a85f1c8-2a56-4887-9b1e-13f44968d390 | |
Status | Published | |
Engine URL | ghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/fitzroy-mini:v2024.1.0-main-4963 | |
Pipeline URL | ghcr.io/wallaroolabs/doc-samples/pipelines/edge-pipeline:5a85f1c8-2a56-4887-9b1e-13f44968d390 | |
Helm Chart URL | oci://ghcr.io/wallaroolabs/doc-samples/charts/edge-pipeline | |
Helm Chart Reference | ghcr.io/wallaroolabs/doc-samples/charts@sha256:e7da76da3996d2993171e9bdc23f71fd92fb92848bbca3aa8bd93ea4c62ef31f | |
Helm Chart Version | 0.0.1-5a85f1c8-2a56-4887-9b1e-13f44968d390 | |
Engine Config | {'engine': {'resources': {'limits': {'cpu': 1.0, 'memory': '512Mi'}, 'requests': {'cpu': 1.0, 'memory': '512Mi'}, 'accel': 'none', 'arch': 'x86', 'gpu': False}}, 'engineAux': {'autoscale': {'type': 'none'}, 'images': {}}} | |
User Images | [] | |
Created By | john.hummel@wallaroo.ai | |
Created At | 2024-04-22 18:17:02.879740+00:00 | |
Updated At | 2024-04-22 18:17:02.879740+00:00 | |
Replaces | ||
Docker Run Command |
Note: Please set the PERSISTENT_VOLUME_DIR , EDGE_PORT , OCI_USERNAME , and OCI_PASSWORD environment variables. | |
Helm Install Command |
Note: Please set the HELM_INSTALL_NAME , HELM_INSTALL_NAMESPACE ,
OCI_USERNAME , and OCI_PASSWORD environment variables. |
We now have our pipeline published to our Edge Registry service. We can deploy this in a x86 environment running Docker that is logged into the same registry service that we deployed to.
For more details, check with the documentation on your artifact service. The following are provided for the three major cloud services:
Once a pipeline is deployed to the Edge Registry service, it can be deployed in environments such as Docker, Kubernetes, or similar container running services by a DevOps engineer. For full details, see How to Publish and Deploy AI Workloads for For Edge/Multicloud Model Deployments. The pipeline publishes Docker Run Command
and Helm Install Command
provide templates for deployment.
For our examples, we will deploy the XGB version of the pipeline to port 8080
, and the Keras version to port 8081
.
Once deployed, we can check the pipelines and models available. We’ll use a curl
command, but any HTTP based request will work the same way.
The endpoint /pipelines
returns:
Running
, or Error
if there are any issues.For this example, the deployment is made on a machine called localhost
. Replace this URL with the URL of you edge deployment.
!curl testboy.local:8080/pipelines
{"pipelines":[{"id":"edge-pipeline","version":"98795ff5-2e56-465a-9041-77e426d8fd19","status":"Running"}]}
!curl testboy.local:8081/pipelines
{"pipelines":[{"id":"edge-pipeline","version":"5a85f1c8-2a56-4887-9b1e-13f44968d390","status":"Running"}]}
The endpoint /models
returns a List of models with the following fields:
!curl testboy.local:8080/models
{"models":[{"sha":"054810e3e3ebbdd34438d9c1a08ed6a6680ef10bf97b9223f78ebf38e14b3b52","name":"ccfraud-xgboost","version":"a09efb31-1e8d-4e72-8f30-948fff078fb3","status":"Running"}]}
!curl testboy.local:8081/models
{"models":[{"sha":"bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507","name":"ccfraud-keras","version":"cbffbf65-e3cd-4b16-bcf8-c862baa43adf","status":"Running"}]}
The inference endpoint takes the following pattern:
/infer
: The inference endpoint remains the same regardless of the pipeline or models deployed. This allows publish replacements without altering the inference endpoint used by other applications.Wallaroo inference endpoint URLs accept the following data inputs through the Content-Type
header:
Content-Type: application/vnd.apache.arrow.file
: For Apache Arrow tables.Content-Type: application/json; format=pandas-records
: For pandas DataFrame in record format.Once deployed, we can perform an inference through the deployment URL.
The endpoint returns Content-Type: application/json; format=pandas-records
by default with the following fields:
null
if the input may be too long for a proper return.!curl -X POST testboy.local:8080/pipelines/edge-pipeline \
-H "Content-Type: application/vnd.apache.arrow.file" \
--data-binary @./data/cc_data_10k.arrow > xgb-edge-infer.df.json
df_xgb = pd.read_json("xgb-edge-infer.df.json", orient="records")
df_xgb.head(5)
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 8232k 100 7069k 100 1162k 1366k 224k 0:00:05 0:00:05 --:--:-- 1635k0k 367k 0:00:06 0:00:03 0:00:03 1497k
time | in | out | anomaly | metadata | |
---|---|---|---|---|---|
0 | 1713810175387 | {'tensor': [-1.0603298, 2.3544967, -3.5638788, 5.138735, -1.2308457, -0.7687824400000001, -3.5881228, 1.8880838, -3.2789674, -3.9563255, 4.099344, -5.653918, -0.8775733, -9.131571, -0.6093538, -3.7480276, -5.0309124, -0.8748149, 1.9870535, 0.7005486, 0.9204422999999999, -0.10414918000000001, 0.32295644, -0.74181414, 0.038412016, 1.0993439, 1.2603409, -0.14662448, -1.4463212]} | {'variable': [1.0094898]} | {'count': 0} | {'last_model': '{"model_name":"ccfraud-xgboost","model_sha":"054810e3e3ebbdd34438d9c1a08ed6a6680ef10bf97b9223f78ebf38e14b3b52"}', 'pipeline_version': '98795ff5-2e56-465a-9041-77e426d8fd19', 'elapsed': [1943497, 25420785], 'dropped': [], 'partition': 'xgb-ccfraud-edge-publish'} |
1 | 1713810175387 | {'tensor': [-1.0603298, 2.3544967, -3.5638788, 5.138735, -1.2308457, -0.7687824400000001, -3.5881228, 1.8880838, -3.2789674, -3.9563255, 4.099344, -5.653918, -0.8775733, -9.131571, -0.6093538, -3.7480276, -5.0309124, -0.8748149, 1.9870535, 0.7005486, 0.9204422999999999, -0.10414918000000001, 0.32295644, -0.74181414, 0.038412016, 1.0993439, 1.2603409, -0.14662448, -1.4463212]} | {'variable': [1.0094898]} | {'count': 0} | {'last_model': '{"model_name":"ccfraud-xgboost","model_sha":"054810e3e3ebbdd34438d9c1a08ed6a6680ef10bf97b9223f78ebf38e14b3b52"}', 'pipeline_version': '98795ff5-2e56-465a-9041-77e426d8fd19', 'elapsed': [1943497, 25420785], 'dropped': [], 'partition': 'xgb-ccfraud-edge-publish'} |
2 | 1713810175387 | {'tensor': [-1.0603298, 2.3544967, -3.5638788, 5.138735, -1.2308457, -0.7687824400000001, -3.5881228, 1.8880838, -3.2789674, -3.9563255, 4.099344, -5.653918, -0.8775733, -9.131571, -0.6093538, -3.7480276, -5.0309124, -0.8748149, 1.9870535, 0.7005486, 0.9204422999999999, -0.10414918000000001, 0.32295644, -0.74181414, 0.038412016, 1.0993439, 1.2603409, -0.14662448, -1.4463212]} | {'variable': [1.0094898]} | {'count': 0} | {'last_model': '{"model_name":"ccfraud-xgboost","model_sha":"054810e3e3ebbdd34438d9c1a08ed6a6680ef10bf97b9223f78ebf38e14b3b52"}', 'pipeline_version': '98795ff5-2e56-465a-9041-77e426d8fd19', 'elapsed': [1943497, 25420785], 'dropped': [], 'partition': 'xgb-ccfraud-edge-publish'} |
3 | 1713810175387 | {'tensor': [-1.0603298, 2.3544967, -3.5638788, 5.138735, -1.2308457, -0.7687824400000001, -3.5881228, 1.8880838, -3.2789674, -3.9563255, 4.099344, -5.653918, -0.8775733, -9.131571, -0.6093538, -3.7480276, -5.0309124, -0.8748149, 1.9870535, 0.7005486, 0.9204422999999999, -0.10414918000000001, 0.32295644, -0.74181414, 0.038412016, 1.0993439, 1.2603409, -0.14662448, -1.4463212]} | {'variable': [1.0094898]} | {'count': 0} | {'last_model': '{"model_name":"ccfraud-xgboost","model_sha":"054810e3e3ebbdd34438d9c1a08ed6a6680ef10bf97b9223f78ebf38e14b3b52"}', 'pipeline_version': '98795ff5-2e56-465a-9041-77e426d8fd19', 'elapsed': [1943497, 25420785], 'dropped': [], 'partition': 'xgb-ccfraud-edge-publish'} |
4 | 1713810175387 | {'tensor': [0.5817662, 0.09788155, 0.15468194, 0.4754102, -0.19788623, -0.45043448, 0.016654044, -0.025607055, 0.09205616, -0.27839172, 0.059329946, -0.019658541, -0.42250833, -0.12175389, 1.5473095, 0.23916228, 0.3553975, -0.76851654, -0.7000849, -0.11900433, -0.34505169999999996, -1.1065114, 0.25234112000000003, 0.020944182000000002, 0.21992674, 0.25406894, -0.04502251, 0.10867739, 0.25471792]} | {'variable': [-1.9073485999999998e-06]} | {'count': 0} | {'last_model': '{"model_name":"ccfraud-xgboost","model_sha":"054810e3e3ebbdd34438d9c1a08ed6a6680ef10bf97b9223f78ebf38e14b3b52"}', 'pipeline_version': '98795ff5-2e56-465a-9041-77e426d8fd19', 'elapsed': [1943497, 25420785], 'dropped': [], 'partition': 'xgb-ccfraud-edge-publish'} |
!curl -X POST testboy.local:8081/pipelines/edge-pipeline \
-H "Content-Type: application/vnd.apache.arrow.file" \
--data-binary @./data/cc_data_10k.arrow > xgb-edge-infer.df.json
df_keras = pd.read_json("xgb-edge-infer.df.json", orient="records")
df_keras.head(5)
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 8161k 100 6999k 100 1162k 1712k 284k 0:00:04 0:00:04 --:--:-- 1997k
time | in | out | anomaly | metadata | |
---|---|---|---|---|---|
0 | 1713810199329 | {'tensor': [-1.0603298, 2.3544967, -3.5638788, 5.138735, -1.2308457, -0.7687824400000001, -3.5881228, 1.8880838, -3.2789674, -3.9563255, 4.099344, -5.653918, -0.8775733, -9.131571, -0.6093538, -3.7480276, -5.0309124, -0.8748149, 1.9870535, 0.7005486, 0.9204422999999999, -0.10414918000000001, 0.32295644, -0.74181414, 0.038412016, 1.0993439, 1.2603409, -0.14662448, -1.4463212]} | {'dense_1': [0.99300325]} | {'count': 0} | {'last_model': '{"model_name":"ccfraud-keras","model_sha":"bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507"}', 'pipeline_version': '5a85f1c8-2a56-4887-9b1e-13f44968d390', 'elapsed': [1882610, 29810875], 'dropped': [], 'partition': 'keras-ccfraud-publish'} |
1 | 1713810199329 | {'tensor': [-1.0603298, 2.3544967, -3.5638788, 5.138735, -1.2308457, -0.7687824400000001, -3.5881228, 1.8880838, -3.2789674, -3.9563255, 4.099344, -5.653918, -0.8775733, -9.131571, -0.6093538, -3.7480276, -5.0309124, -0.8748149, 1.9870535, 0.7005486, 0.9204422999999999, -0.10414918000000001, 0.32295644, -0.74181414, 0.038412016, 1.0993439, 1.2603409, -0.14662448, -1.4463212]} | {'dense_1': [0.99300325]} | {'count': 0} | {'last_model': '{"model_name":"ccfraud-keras","model_sha":"bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507"}', 'pipeline_version': '5a85f1c8-2a56-4887-9b1e-13f44968d390', 'elapsed': [1882610, 29810875], 'dropped': [], 'partition': 'keras-ccfraud-publish'} |
2 | 1713810199329 | {'tensor': [-1.0603298, 2.3544967, -3.5638788, 5.138735, -1.2308457, -0.7687824400000001, -3.5881228, 1.8880838, -3.2789674, -3.9563255, 4.099344, -5.653918, -0.8775733, -9.131571, -0.6093538, -3.7480276, -5.0309124, -0.8748149, 1.9870535, 0.7005486, 0.9204422999999999, -0.10414918000000001, 0.32295644, -0.74181414, 0.038412016, 1.0993439, 1.2603409, -0.14662448, -1.4463212]} | {'dense_1': [0.99300325]} | {'count': 0} | {'last_model': '{"model_name":"ccfraud-keras","model_sha":"bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507"}', 'pipeline_version': '5a85f1c8-2a56-4887-9b1e-13f44968d390', 'elapsed': [1882610, 29810875], 'dropped': [], 'partition': 'keras-ccfraud-publish'} |
3 | 1713810199329 | {'tensor': [-1.0603298, 2.3544967, -3.5638788, 5.138735, -1.2308457, -0.7687824400000001, -3.5881228, 1.8880838, -3.2789674, -3.9563255, 4.099344, -5.653918, -0.8775733, -9.131571, -0.6093538, -3.7480276, -5.0309124, -0.8748149, 1.9870535, 0.7005486, 0.9204422999999999, -0.10414918000000001, 0.32295644, -0.74181414, 0.038412016, 1.0993439, 1.2603409, -0.14662448, -1.4463212]} | {'dense_1': [0.99300325]} | {'count': 0} | {'last_model': '{"model_name":"ccfraud-keras","model_sha":"bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507"}', 'pipeline_version': '5a85f1c8-2a56-4887-9b1e-13f44968d390', 'elapsed': [1882610, 29810875], 'dropped': [], 'partition': 'keras-ccfraud-publish'} |
4 | 1713810199329 | {'tensor': [0.5817662, 0.09788155, 0.15468194, 0.4754102, -0.19788623, -0.45043448, 0.016654044, -0.025607055, 0.09205616, -0.27839172, 0.059329946, -0.019658541, -0.42250833, -0.12175389, 1.5473095, 0.23916228, 0.3553975, -0.76851654, -0.7000849, -0.11900433, -0.34505169999999996, -1.1065114, 0.25234112000000003, 0.020944182000000002, 0.21992674, 0.25406894, -0.04502251, 0.10867739, 0.25471792]} | {'dense_1': [0.0010916889]} | {'count': 0} | {'last_model': '{"model_name":"ccfraud-keras","model_sha":"bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507"}', 'pipeline_version': '5a85f1c8-2a56-4887-9b1e-13f44968d390', 'elapsed': [1882610, 29810875], 'dropped': [], 'partition': 'keras-ccfraud-publish'} |