Wallaroo provides the ability to publish a pipelines, which automatically packages pipelines with their models and models steps and publish the containerized pipelines with the inference engine to an Open Container Initiative (OCI) Registry as a pipeline publish.
Edge locations are added to pipeline publishes associated with the pipeline, and provides:
/logs
endpoint./logs
endpoint, and are transmitted bck to the Wallaroo Ops instance when connectivity is restored.metadata.partition
labeled with the edge location’s name. For example, inference requests to the the edge location named us-west1
would have its inference request logs stored in the pipeline logs metadata.partition
as us-west1
.When connectivity is available, edge inference logs are synced with the pipeline they are referenced from. For more details on adding an edge location to a pipeline publish, see How to Publish and Deploy AI Workloads for For Edge and Multicloud Model Deployments.
Inference logs from an edge location can be available for analysis in the Wallaroo Model Ops Center pipeline are retrieved by one of the two methods:
wallaroo.pipeline.Pipeline.logs
: Returns the pipeline inference logs.wallaroo.pipeline.Pipeline.export_logs
: Creates a subdirectory and JSON files in pandas Record Format of the pipeline inference logs.The field metadata.partition
displays which location the inference was performed through. This is returned with the dataset
parameter and requesting the metadata
dataset for each pipeline log request. For example:
pipeline.log(dataset=["time", "in", "out", "metadata"]) # returns the time, in, out and metadata datasets
pipeline.export_logs(dataset=["time", "in", "out", "metadata"]) # returns the time, in, out and metadata datasets
If latency between the model’s edge deployment and the Wallaroo Ops Instance is higher than 0.5, the connection may time out and the inference results are not transmitted back to the Wallaroo Ops Center. To check the latency between the deployed model and the Wallaroo Ops Center, use the following command from the edge device:
curl -o /dev/null -s -w "%{time_total}\n" {Wallaroo Domain}:8443
For example, if the Wallaroo Domain of the Wallaroo Ops Center is example.wallaroo.ai
, the command is:
curl -o /dev/null -s -w "%{time_total}\n" example.wallaroo.ai:8443
0.089316
For docker run
deployments, increase the timeout with the option -e MTLS_CONNECTION_TIMEOUT=3.0
. Increase the MTLS_CONNECTION_TIMEOUT
further as needed for extreme low latency scenarios.
For example:
docker run -p 8080:8080 \
-v ./data:/persist \
-e DEBUG=true \
-e MTLS_CONNECTION_TIMEOUT=3.0 \
-e OCI_REGISTRY=$OCI_REGISTRY \
-e EDGE_BUNDLE=abc123 \
-e CONFIG_CPUS=1 \
-e OCI_USERNAME=$OCI_USERNAME \
-e OCI_PASSWORD=$OCI_PASSWORD \
-e PIPELINE_URL=example.registry.io/wallaroolabs/doc-samples/pipelines/edge-observability-pipeline:3b49fba8-94d8-4fca-aecc-c75257fd16c6 \
example.registry.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/standalone-mini:v2023.4.0-main-4079
In scenarios where connectivity is limited or unavailable for centralized Edge observability, Edge model deployments with Wallaroo can be configured to store inference logs locally in a “first in, first deleted” format: the oldest inference request logs are the first deleted when storage capacity runs out. Inference logs stored at the edge will be available in the model operations center, once connectivity is restored, following an automated retry mechanism.
Edge location inference log storage capacity is set with the LOCAL_INFERENCE_STORAGE
and PLATEAU_PAGE_SIZE
.
LOCAL_INFERENCE_STORAGE
(Optional): Sets amount of storage to allocate for the edge deployments inference log storage capacity. This is in the format {size as number}{unit value}
. If used, must be used with PLATEAU_PAGE_SIZE
. The values are similar to the Kubernetes memory resource units format. This is configurable based on the amount of storage needed during low/no connectivity periods before connectivity is restored for logs to be available in the Wallaroo model Ops center. The accepted unit values are:Ki
(for KiloBytes)Mi
(for MegaBytes)Gi
(for GigaBytes)Ti
(for TeraBytes)PLATEAU_PAGE_SIZE
(Optional): How many inference log rows to upload from the edge deployment at a time. Must be used with LOCAL_INFERENCE_STORAGE
.The following example shows a standard deployment with the edge observability configured for 512Mi
of log storage.
docker run -v $PERSISTENT_VOLUME_DIR:/persist \
-p $EDGE_PORT:8080 \
-e OCI_USERNAME=$OCI_USERNAME \
-e OCI_PASSWORD=$OCI_PASSWORD \
-e LOCAL_INFERENCE_STORAGE=512Mi \
-e PLATEAU_PAGE_SIZE=100 \
-e PIPELINE_URL=registry.example.com/pipelines/edge-low-connection-demo:cc4a10a2-4866-4ed6-a1d8-8b731e7bf8f8\
-e EDGE_BUNDLE=abc123 \
-e CONFIG_CPUS=4 registry.example.com/engines/proxy/wallaroo/ghcr.io/wallaroolabs/fitzroy-mini:v2024.1.0-main-4963
For full details of model deployment in edge locations, see How to Publish and Deploy AI Workloads for For Edge/Multicloud Model Deployments.
Inference logs are retrieved from edge location deployments through the /logs
endpoint.
/logs
POST
Content-Type: application/json
: Submissions to the /logs
endpoint in JSON text format.Accept: application/json; format=pandas-records
: The /logs
endpoint returns JSON in pandas Record format.{}
: An empty set.Inference logs are returned as JSON in pandas Record Format with the following fields:
Field | Type | Description |
---|---|---|
time | DateTime | DateTime field in Epoch format. |
in | Dict | The inputs in Dict format. |
out | Dict | The outputs in Dict format with the model field outputs and values. |
anomaly | Dict | Any anomalies detected; the field count is reserved for the total number of validations derived as True . See anomalies for more details. |
metadata | Dict | Metadata of the transaction that includes:
|
The following example demonstrates the following:
For this example, we have deployed a house price model to an edge location. We perform an inference request and submit JSON in pandas Record Format.
import datetime
import time
!curl -X POST http://HOSTNAME:8080/infer \
-H "Content-Type: Content-Type: application/json; format=pandas-records" \
--data @./data/smallinputs.df.json > ./inferenceoutput2.df.json
# get the dataframe from what we just did
df_result = pd.read_json('./inferenceoutput2.df.json', orient="records")
df_result.tail(10).loc[:, ['time', 'out', 'metadata']]
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 624k 100 502k 100 121k 209k 51771 0:00:02 0:00:02 --:--:-- 260k
time | out | metadata | |
---|---|---|---|
990 | 1713892195526 | {'variable': [437177.97]} | {'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'} |
991 | 1713892195526 | {'variable': [236238.67]} | {'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'} |
992 | 1713892195526 | {'variable': [276046.53]} | {'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'} |
993 | 1713892195526 | {'variable': [758714.3]} | {'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'} |
994 | 1713892195526 | {'variable': [627853.3]} | {'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'} |
995 | 1713892195526 | {'variable': [450867.7]} | {'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'} |
996 | 1713892195526 | {'variable': [553463.25]} | {'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'} |
997 | 1713892195526 | {'variable': [241330.17]} | {'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'} |
998 | 1713892195526 | {'variable': [1295531.8]} | {'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'} |
999 | 1713892195526 | {'variable': [712309.9]} | {'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'} |
We now show the logs from the edge location via the /logs
endpoint.
# local logs - show the last 10 log records to show the updates
!curl -X POST http://HOSTNAME:8080/logs \
-H "Content-Type: Content-Type: application/json; format=pandas-records" \
--data '{ }' > ./edge-logs.df.json
df_logs = pd.read_json("./edge-logs.df.json", orient="records")
df_logs.tail(10)
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 502k 100 502k 100 3 264k 1 0:00:03 0:00:01 0:00:02 264k
time | in | out | anomaly | metadata | |
---|---|---|---|---|---|
990 | 1713892195526 | {'tensor': [2.0, 1.0, 1410.0, 5000.0, 1.0, 0.0, 2.0, 4.0, 7.0, 940.0, 470.0, 47.5531005859, -122.3789978027, 1450.0, 5000.0, 97.0, 0.0, 0.0]} | {'variable': [437929.84]} | {'count': 0} | {'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'} |
991 | 1713892195526 | {'tensor': [4.0, 1.5, 1720.0, 6417.0, 1.0, 0.0, 0.0, 3.0, 7.0, 1720.0, 0.0, 47.7267990112, -122.3109970093, 1430.0, 6240.0, 61.0, 0.0, 0.0]} | {'variable': [437177.97]} | {'count': 0} | {'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'} |
992 | 1713892195526 | {'tensor': [3.0, 1.75, 1050.0, 9871.0, 1.0, 0.0, 0.0, 5.0, 7.0, 1050.0, 0.0, 47.3815994263, -122.0869979858, 1300.0, 10794.0, 46.0, 0.0, 0.0]} | {'variable': [236238.67]} | {'count': 0} | {'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'} |
993 | 1713892195526 | {'tensor': [3.0, 2.25, 1590.0, 11745.0, 1.0, 0.0, 0.0, 3.0, 7.0, 1090.0, 500.0, 47.3553009033, -122.2799987793, 1540.0, 12530.0, 36.0, 0.0, 0.0]} | {'variable': [276046.53]} | {'count': 0} | {'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'} |
994 | 1713892195526 | {'tensor': [4.0, 2.5, 2500.0, 8540.0, 2.0, 0.0, 0.0, 3.0, 9.0, 2500.0, 0.0, 47.5759010315, -121.9940032959, 2560.0, 8475.0, 24.0, 0.0, 0.0]} | {'variable': [758714.3]} | {'count': 0} | {'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'} |
995 | 1713892195526 | {'tensor': [4.0, 2.5, 2040.0, 9225.0, 1.0, 0.0, 0.0, 5.0, 8.0, 1610.0, 430.0, 47.6360015869, -122.0970001221, 1730.0, 9225.0, 46.0, 0.0, 0.0]} | {'variable': [627853.3]} | {'count': 0} | {'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'} |
996 | 1713892195526 | {'tensor': [3.0, 3.0, 1330.0, 1379.0, 2.0, 0.0, 0.0, 4.0, 8.0, 1120.0, 210.0, 47.6125984192, -122.31300354, 1810.0, 1770.0, 9.0, 0.0, 0.0]} | {'variable': [450867.7]} | {'count': 0} | {'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'} |
997 | 1713892195526 | {'tensor': [3.0, 2.5, 1880.0, 4499.0, 2.0, 0.0, 0.0, 3.0, 8.0, 1880.0, 0.0, 47.5663986206, -121.9990005493, 2130.0, 5114.0, 22.0, 0.0, 0.0]} | {'variable': [553463.25]} | {'count': 0} | {'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'} |
998 | 1713892195526 | {'tensor': [4.0, 1.5, 1200.0, 10890.0, 1.0, 0.0, 0.0, 5.0, 7.0, 1200.0, 0.0, 47.342300415, -122.0879974365, 1250.0, 10139.0, 42.0, 0.0, 0.0]} | {'variable': [241330.17]} | {'count': 0} | {'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'} |
999 | 1713892195526 | {'tensor': [4.0, 3.25, 5180.0, 19850.0, 2.0, 0.0, 3.0, 3.0, 12.0, 3540.0, 1640.0, 47.5620002747, -122.1620025635, 3160.0, 9750.0, 9.0, 0.0, 0.0]} | {'variable': [1295531.8]} | {'count': 0} | {'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'} |
Get the most recent logs to verify the edge records were transmitted. We’ll see in the metadata.partition
section the name matching our edge location.
pipeline.logs(dataset=['time', 'out', 'metadata'])
Warning: There are more logs available. Please set a larger limit or request a file using export_logs.
time | out.variable | metadata.last_model | metadata.pipeline_version | metadata.elapsed | metadata.dropped | metadata.partition | |
---|---|---|---|---|---|---|---|
0 | 2024-04-23 17:09:55.526 | [712309.9] | {"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"} | 226508c6-7205-4847-afe3-3d0f62c73096 | [3660658, 3177750] | [] | houseprice-low-connection-demonstration |
1 | 2024-04-23 17:09:55.526 | [1295531.8] | {"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"} | 226508c6-7205-4847-afe3-3d0f62c73096 | [3660658, 3177750] | [] | houseprice-low-connection-demonstration |
2 | 2024-04-23 17:09:55.526 | [241330.17] | {"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"} | 226508c6-7205-4847-afe3-3d0f62c73096 | [3660658, 3177750] | [] | houseprice-low-connection-demonstration |
3 | 2024-04-23 17:09:55.526 | [553463.25] | {"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"} | 226508c6-7205-4847-afe3-3d0f62c73096 | [3660658, 3177750] | [] | houseprice-low-connection-demonstration |
4 | 2024-04-23 17:09:55.526 | [450867.7] | {"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"} | 226508c6-7205-4847-afe3-3d0f62c73096 | [3660658, 3177750] | [] | houseprice-low-connection-demonstration |
... | ... | ... | ... | ... | ... | ... | ... |
95 | 2024-04-23 17:09:55.526 | [450867.7] | {"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"} | 226508c6-7205-4847-afe3-3d0f62c73096 | [3660658, 3177750] | [] | houseprice-low-connection-demonstration |
96 | 2024-04-23 17:09:55.526 | [715530.06] | {"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"} | 226508c6-7205-4847-afe3-3d0f62c73096 | [3660658, 3177750] | [] | houseprice-low-connection-demonstration |
97 | 2024-04-23 17:09:55.526 | [837085.5] | {"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"} | 226508c6-7205-4847-afe3-3d0f62c73096 | [3660658, 3177750] | [] | houseprice-low-connection-demonstration |
98 | 2024-04-23 17:09:55.526 | [630865.5] | {"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"} | 226508c6-7205-4847-afe3-3d0f62c73096 | [3660658, 3177750] | [] | houseprice-low-connection-demonstration |
99 | 2024-04-23 17:09:55.526 | [256845.58] | {"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"} | 226508c6-7205-4847-afe3-3d0f62c73096 | [3660658, 3177750] | [] | houseprice-low-connection-demonstration |
100 rows × 7 columns
For this example, we’ll simulate a connection loss, then perform another set of inferences, this time from the ./data/biginputs.df.json
list. When we retrieve the pipeline logs again from the Wallaroo Ops instance, we should not see any additional records.
!curl -X POST HOSTNAME:8080/infer \
-H "Content-Type: Content-Type: application/json; format=pandas-records" \
--data @./data/biginputs.df.json > ./inferenceoutput2.df.json
# get the dataframe from what we just did
df_result = pd.read_json('./inferenceoutput2.df.json', orient="records")
df_result.tail(10).loc[:, ['time', 'out', 'metadata']]
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 5786 100 4639 100 1147 13402 3313 --:--:-- --:--:-- --:--:-- 16722
time | out | metadata | |
---|---|---|---|
0 | 1713892238998 | {'variable': [1514079.4]} | {'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [78802, 229304], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'} |
1 | 1713892238998 | {'variable': [1967344.1]} | {'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [78802, 229304], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'} |
2 | 1713892238998 | {'variable': [2002393.6]} | {'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [78802, 229304], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'} |
3 | 1713892238998 | {'variable': [1886959.3]} | {'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [78802, 229304], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'} |
4 | 1713892238998 | {'variable': [1689843.1]} | {'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [78802, 229304], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'} |
5 | 1713892238998 | {'variable': [1946437.8]} | {'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [78802, 229304], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'} |
6 | 1713892238998 | {'variable': [2005883.1]} | {'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [78802, 229304], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'} |
7 | 1713892238998 | {'variable': [1910824.0]} | {'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [78802, 229304], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'} |
8 | 1713892238998 | {'variable': [2016006.1]} | {'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [78802, 229304], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'} |
These are the logs retrieved from the Wallaroo Ops center during the connection loss. We can see the new records from the edge deployment have not been added.
pipeline.logs(dataset=['time', 'out', 'metadata'])
Warning: There are more logs available. Please set a larger limit or request a file using export_logs.
time | out.variable | metadata.last_model | metadata.pipeline_version | metadata.elapsed | metadata.dropped | metadata.partition | |
---|---|---|---|---|---|---|---|
0 | 2024-04-23 17:09:55.526 | [712309.9] | {"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"} | 226508c6-7205-4847-afe3-3d0f62c73096 | [3660658, 3177750] | [] | houseprice-low-connection-demonstration |
1 | 2024-04-23 17:09:55.526 | [1295531.8] | {"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"} | 226508c6-7205-4847-afe3-3d0f62c73096 | [3660658, 3177750] | [] | houseprice-low-connection-demonstration |
2 | 2024-04-23 17:09:55.526 | [241330.17] | {"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"} | 226508c6-7205-4847-afe3-3d0f62c73096 | [3660658, 3177750] | [] | houseprice-low-connection-demonstration |
3 | 2024-04-23 17:09:55.526 | [553463.25] | {"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"} | 226508c6-7205-4847-afe3-3d0f62c73096 | [3660658, 3177750] | [] | houseprice-low-connection-demonstration |
4 | 2024-04-23 17:09:55.526 | [450867.7] | {"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"} | 226508c6-7205-4847-afe3-3d0f62c73096 | [3660658, 3177750] | [] | houseprice-low-connection-demonstration |
... | ... | ... | ... | ... | ... | ... | ... |
95 | 2024-04-23 17:09:55.526 | [450867.7] | {"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"} | 226508c6-7205-4847-afe3-3d0f62c73096 | [3660658, 3177750] | [] | houseprice-low-connection-demonstration |
96 | 2024-04-23 17:09:55.526 | [715530.06] | {"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"} | 226508c6-7205-4847-afe3-3d0f62c73096 | [3660658, 3177750] | [] | houseprice-low-connection-demonstration |
97 | 2024-04-23 17:09:55.526 | [837085.5] | {"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"} | 226508c6-7205-4847-afe3-3d0f62c73096 | [3660658, 3177750] | [] | houseprice-low-connection-demonstration |
98 | 2024-04-23 17:09:55.526 | [630865.5] | {"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"} | 226508c6-7205-4847-afe3-3d0f62c73096 | [3660658, 3177750] | [] | houseprice-low-connection-demonstration |
99 | 2024-04-23 17:09:55.526 | [256845.58] | {"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"} | 226508c6-7205-4847-afe3-3d0f62c73096 | [3660658, 3177750] | [] | houseprice-low-connection-demonstration |
100 rows × 7 columns
The logs retrieved from the local Edge location, which is not in communication with the Wallaroo Ops instance.
# local logs
!curl -X POST http://HOSTNAME:8080/logs \
-H "Content-Type: Content-Type: application/json; format=pandas-records" \
--data '{}' > ./edge-logs.df.json
df_logs = pd.read_json("./edge-logs.df.json", orient="records")
df_logs
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 502k 100 502k 100 2 344k 1 0:00:02 0:00:01 0:00:01 344k
time | in | out | anomaly | metadata | |
---|---|---|---|---|---|
0 | 1713892183771 | {'tensor': [4.0, 2.5, 2900.0, 5505.0, 2.0, 0.0, 0.0, 3.0, 8.0, 2900.0, 0.0, 47.6063, -122.02, 2970.0, 5251.0, 12.0, 0.0, 0.0]} | {'variable': [718013.7]} | {'count': 0} | {'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [149203, 331606], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'} |
1 | 1713892195526 | {'tensor': [4.0, 2.75, 3010.0, 7215.0, 2.0, 0.0, 0.0, 3.0, 9.0, 3010.0, 0.0, 47.6952018738, -122.1780014038, 3010.0, 7215.0, 0.0, 0.0, 0.0]} | {'variable': [795841.06]} | {'count': 0} | {'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'} |
2 | 1713892195526 | {'tensor': [4.0, 1.75, 1400.0, 7920.0, 1.0, 0.0, 0.0, 3.0, 7.0, 1400.0, 0.0, 47.465801239, -122.1839981079, 1910.0, 7700.0, 52.0, 0.0, 0.0]} | {'variable': [267013.97]} | {'count': 0} | {'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'} |
3 | 1713892195526 | {'tensor': [4.0, 2.5, 3130.0, 13202.0, 2.0, 0.0, 0.0, 3.0, 10.0, 3130.0, 0.0, 47.5877990723, -121.9759979248, 2840.0, 10470.0, 19.0, 0.0, 0.0]} | {'variable': [879083.56]} | {'count': 0} | {'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'} |
4 | 1713892195526 | {'tensor': [3.0, 2.25, 1620.0, 997.0, 2.5, 0.0, 0.0, 3.0, 8.0, 1540.0, 80.0, 47.5400009155, -122.0260009766, 1620.0, 1068.0, 4.0, 0.0, 0.0]} | {'variable': [544392.06]} | {'count': 0} | {'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'} |
... | ... | ... | ... | ... | ... |
995 | 1713892195526 | {'tensor': [4.0, 2.5, 2040.0, 9225.0, 1.0, 0.0, 0.0, 5.0, 8.0, 1610.0, 430.0, 47.6360015869, -122.0970001221, 1730.0, 9225.0, 46.0, 0.0, 0.0]} | {'variable': [627853.3]} | {'count': 0} | {'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'} |
996 | 1713892195526 | {'tensor': [3.0, 3.0, 1330.0, 1379.0, 2.0, 0.0, 0.0, 4.0, 8.0, 1120.0, 210.0, 47.6125984192, -122.31300354, 1810.0, 1770.0, 9.0, 0.0, 0.0]} | {'variable': [450867.7]} | {'count': 0} | {'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'} |
997 | 1713892195526 | {'tensor': [3.0, 2.5, 1880.0, 4499.0, 2.0, 0.0, 0.0, 3.0, 8.0, 1880.0, 0.0, 47.5663986206, -121.9990005493, 2130.0, 5114.0, 22.0, 0.0, 0.0]} | {'variable': [553463.25]} | {'count': 0} | {'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'} |
998 | 1713892195526 | {'tensor': [4.0, 1.5, 1200.0, 10890.0, 1.0, 0.0, 0.0, 5.0, 7.0, 1200.0, 0.0, 47.342300415, -122.0879974365, 1250.0, 10139.0, 42.0, 0.0, 0.0]} | {'variable': [241330.17]} | {'count': 0} | {'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'} |
999 | 1713892195526 | {'tensor': [4.0, 3.25, 5180.0, 19850.0, 2.0, 0.0, 3.0, 3.0, 12.0, 3540.0, 1640.0, 47.5620002747, -122.1620025635, 3160.0, 9750.0, 9.0, 0.0, 0.0]} | {'variable': [1295531.8]} | {'count': 0} | {'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'} |
1000 rows × 5 columns
We’ll now reenable communication, then wait 60 seconds and check the pipeline inference logs to see if communication is restored.
import time
time.sleep(60)
import json
df = pd.DataFrame.from_records({"tensor": [[4.0,
2.5,
2900.0,
5505.0,
2.0,
0.0,
0.0,
3.0,
8.0,
2900.0,
0.0,
47.6063,
-122.02,
2970.0,
5251.0,
12.0,
0.0,
0.0]]})
data = df.to_dict(orient="records")
!curl -X POST HOSTNAME:8080/infer \
-H "Content-Type: Content-Type: application/json; format=pandas-records" \
--data '{json.dumps(data)}' > ./inferenceoutput.df.json
# get the dataframe from what we just did
df_result = pd.read_json('./inferenceoutput.df.json', orient="records")
df_result.loc[:, ['time', 'out', 'metadata']]
pipeline.logs(dataset=['time', 'out', 'metadata'])
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 625 100 497 100 128 1242 319 --:--:-- --:--:-- --:--:-- 1562
Warning: There are more logs available. Please set a larger limit or request a file using export_logs.
time | out.variable | metadata.last_model | metadata.pipeline_version | metadata.elapsed | metadata.dropped | metadata.partition | |
---|---|---|---|---|---|---|---|
0 | 2024-04-23 17:12:55.904 | [718013.7] | {"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"} | 226508c6-7205-4847-afe3-3d0f62c73096 | [46701, 199803] | [] | houseprice-low-connection-demonstration |
1 | 2024-04-23 17:10:38.998 | [2016006.1] | {"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"} | 226508c6-7205-4847-afe3-3d0f62c73096 | [78802, 229304] | [] | houseprice-low-connection-demonstration |
2 | 2024-04-23 17:10:38.998 | [1910824.0] | {"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"} | 226508c6-7205-4847-afe3-3d0f62c73096 | [78802, 229304] | [] | houseprice-low-connection-demonstration |
3 | 2024-04-23 17:10:38.998 | [2005883.1] | {"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"} | 226508c6-7205-4847-afe3-3d0f62c73096 | [78802, 229304] | [] | houseprice-low-connection-demonstration |
4 | 2024-04-23 17:10:38.998 | [1946437.8] | {"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"} | 226508c6-7205-4847-afe3-3d0f62c73096 | [78802, 229304] | [] | houseprice-low-connection-demonstration |
... | ... | ... | ... | ... | ... | ... | ... |
95 | 2024-04-23 17:09:55.526 | [726181.75] | {"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"} | 226508c6-7205-4847-afe3-3d0f62c73096 | [3660658, 3177750] | [] | houseprice-low-connection-demonstration |
96 | 2024-04-23 17:09:55.526 | [836230.2] | {"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"} | 226508c6-7205-4847-afe3-3d0f62c73096 | [3660658, 3177750] | [] | houseprice-low-connection-demonstration |
97 | 2024-04-23 17:09:55.526 | [291903.97] | {"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"} | 226508c6-7205-4847-afe3-3d0f62c73096 | [3660658, 3177750] | [] | houseprice-low-connection-demonstration |
98 | 2024-04-23 17:09:55.526 | [448627.8] | {"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"} | 226508c6-7205-4847-afe3-3d0f62c73096 | [3660658, 3177750] | [] | houseprice-low-connection-demonstration |
99 | 2024-04-23 17:09:55.526 | [1060847.5] | {"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"} | 226508c6-7205-4847-afe3-3d0f62c73096 | [3660658, 3177750] | [] | houseprice-low-connection-demonstration |
100 rows × 7 columns