Model Observability for Edge Deployments with Low or No Connectivity

How to configure and get observability in edge model deployments where there is low or no connectivity from the edge locations to the Wallaroo Ops instance.

Wallaroo provides the ability to publish a pipelines, which automatically packages pipelines with their models and models steps and publish the containerized pipelines with the inference engine to an Open Container Initiative (OCI) Registry as a pipeline publish.

Edge locations are added to pipeline publishes associated with the pipeline, and provides:

  • Authentication credentials to allow edge deployments to connect back to the Wallaroo Ops instance.
  • Inference results from the edge deployments are stored in the local configured storage with configurable storage size in a “first in first out” basis; the oldest inference results are deleted first. These records are stored locally even when connection to the Wallaroo Ops instance is lost.
  • Edge locations store their inference logs locally through a persistent volume.
    • These logs are retrieved through the edge deployments /logs endpoint.
    • When a connection is not available from a Wallaroo Ops instance, or an “air gap” deployment, the logs stored in the local edge deployments are still available through the /logs endpoint, and are transmitted bck to the Wallaroo Ops instance when connectivity is restored.
    • Logs are stored in a First In-First Out method. When the storage capacity is reached, the oldest logs are deleted first.
  • When the edge location is connected with the Wallaroo instance, inference result logs are transmitted back to the Wallaroo Ops instance and added to the pipeline the edge location is associated with, based on the DateTime stamp of when the inference occurred, not when the logs were transmitted back to the Wallaroo ops instance.
    • This allows operations such as model drift detection with assays and other observability operations to sync inference times across multiple locations.
    • Pipeline logs store the inference location in the field metadata.partition labeled with the edge location’s name. For example, inference requests to the the edge location named us-west1 would have its inference request logs stored in the pipeline logs metadata.partition as us-west1.
  • Pipeline publishes replaced from new pipeline versions or entirely new pipelines allocate the edge locations associated with the pipeline to the new pipeline. Previous inference results and logs remain with the previous pipeline, and new inference records are added to the pipeline the edge locations are now associated with.

Edge Observability via the Wallaroo SDK

When connectivity is available, edge inference logs are synced with the pipeline they are referenced from. For more details on adding an edge location to a pipeline publish, see How to Publish and Deploy AI Workloads for For Edge and Multicloud Model Deployments.

Retrieve Edge Location Inference Logs from a Pipeline via the Wallaroo SDK

Inference logs from an edge location can be available for analysis in the Wallaroo Model Ops Center pipeline are retrieved by one of the two methods:

The field metadata.partition displays which location the inference was performed through. This is returned with the dataset parameter and requesting the metadata dataset for each pipeline log request. For example:

pipeline.log(dataset=["time", "in", "out", "metadata"]) # returns the time, in, out and metadata datasets
pipeline.export_logs(dataset=["time", "in", "out", "metadata"]) # returns the time, in, out and metadata datasets

Airgapped Edge Observability from Edge Deployments

In scenarios where connectivity is limited or unavailable for centralized Edge observability, Edge model deployments with Wallaroo can be configured to store inference logs locally in a “first in, first deleted” format: the oldest inference request logs are the first deleted when storage capacity runs out. Inference logs stored at the edge will be available in the model operations center, once connectivity is restored, following an automated retry mechanism.

Edge Observability Storage Configuration

Edge location inference log storage capacity is set with the LOCAL_INFERENCE_STORAGE and PLATEAU_PAGE_SIZE.

  • LOCAL_INFERENCE_STORAGE (Optional): Sets amount of storage to allocate for the edge deployments inference log storage capacity. This is in the format {size as number}{unit value}. If used, must be used with PLATEAU_PAGE_SIZE. The values are similar to the Kubernetes memory resource units format. This is configurable based on the amount of storage needed during low/no connectivity periods before connectivity is restored for logs to be available in the Wallaroo model Ops center. The accepted unit values are:
    • Ki (for KiloBytes)
    • Mi (for MegaBytes)
    • Gi (for GigaBytes)
    • Ti (for TeraBytes)
  • PLATEAU_PAGE_SIZE (Optional): How many inference log rows to upload from the edge deployment at a time. Must be used with LOCAL_INFERENCE_STORAGE.

The following example shows a standard deployment with the edge observability configured for 512Mi of log storage.

docker run -v $PERSISTENT_VOLUME_DIR:/persist \
    -p $EDGE_PORT:8080 \
    -e OCI_USERNAME=$OCI_USERNAME \
    -e OCI_PASSWORD=$OCI_PASSWORD \
    -e LOCAL_INFERENCE_STORAGE=512Mi \
    -e PLATEAU_PAGE_SIZE=100 \
    -e PIPELINE_URL=registry.example.com/pipelines/edge-low-connection-demo:cc4a10a2-4866-4ed6-a1d8-8b731e7bf8f8\
    -e EDGE_BUNDLE=abc123 \
    -e CONFIG_CPUS=4 registry.example.com/engines/proxy/wallaroo/ghcr.io/wallaroolabs/fitzroy-mini:v2024.1.0-main-4963

For full details of model deployment in edge locations, see How to Publish and Deploy AI Workloads for For Edge/Multicloud Model Deployments.

Log Retrieval from Edge Locations

Inference logs are retrieved from edge location deployments through the /logs endpoint.

  • Endpoint: /logs
  • Type: POST
  • Headers:
    • Content-Type: application/json: Submissions to the /logs endpoint in JSON text format.
    • Accept: application/json; format=pandas-records: The /logs endpoint returns JSON in pandas Record format.
  • Parameters:
    • {}: An empty set.

Inference logs are returned as JSON in pandas Record Format with the following fields:

FieldTypeDescription
timeDateTimeDateTime field in Epoch format.
inDictThe inputs in Dict format.
outDictThe outputs in Dict format with the model field outputs and values.
anomalyDictAny anomalies detected; the field count is reserved for the total number of validations derived as True. See anomalies for more details.
metadataDictMetadata of the transaction that includes:
  • last_model (Dict): The last model used in the inference request that includes:
    • model_name (String): The name assigned to the model when uploaded to Wallaroo Ops.
    • model_sha (String): The sha hash of the model.
  • pipeline_version (String): The version of the pipeline in UUID format.
  • elasped (List[Integer]):
    • A list of time in nanoseconds for:
      • The time to serialize the input.
      • How long each step took.
  • dropped (List[String]): Any inference input fields dropped to reduce log storage. Inference results always return the entire inference inputs and outputs; inference logs may drop input fields for space purposes.
  • partition (String): The partition the inference logs are assigned. On edge deployments, the partition matches the name of the edge added to the pipeline.
  • Examples

    Edge Observability Low Connectivity Example

    The following example demonstrates the following:

    • From an edge location model deployment:
      • Perform inference requests and retrieving logs from both the edge location and the Wallaroo Ops center.
      • Simulate a disconnection between the edge location and the Wallaroo Ops center and showing how inference requests logs are available in the edge location.
      • Simulate a reconnection between the edge location and the Wallaroo Ops center and showing how inference request logs are added to the Wallaroo Ops instance.

    For this example, we have deployed a house price model to an edge location. We perform an inference request and submit JSON in pandas Record Format.

    import datetime
    import time
    
    !curl -X POST http://HOSTNAME:8080/infer \
        -H "Content-Type: Content-Type: application/json; format=pandas-records" \
        --data @./data/smallinputs.df.json > ./inferenceoutput2.df.json
    
    # get the dataframe from what we just did
    
    df_result = pd.read_json('./inferenceoutput2.df.json', orient="records")
    df_result.tail(10).loc[:, ['time', 'out', 'metadata']]
    
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
      0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
    
    100  624k  100  502k  100  121k   209k  51771  0:00:02  0:00:02 --:--:--  260k
    
    timeoutmetadata
    9901713892195526{'variable': [437177.97]}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'}
    9911713892195526{'variable': [236238.67]}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'}
    9921713892195526{'variable': [276046.53]}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'}
    9931713892195526{'variable': [758714.3]}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'}
    9941713892195526{'variable': [627853.3]}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'}
    9951713892195526{'variable': [450867.7]}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'}
    9961713892195526{'variable': [553463.25]}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'}
    9971713892195526{'variable': [241330.17]}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'}
    9981713892195526{'variable': [1295531.8]}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'}
    9991713892195526{'variable': [712309.9]}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'}

    We now show the logs from the edge location via the /logs endpoint.

    # local logs - show the last 10 log records to show the updates
    
    !curl -X POST http://HOSTNAME:8080/logs \
        -H "Content-Type: Content-Type: application/json; format=pandas-records" \
        --data '{ }' > ./edge-logs.df.json
    
    df_logs = pd.read_json("./edge-logs.df.json", orient="records")
    df_logs.tail(10)
    
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
      0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
    
    100  502k  100  502k  100     3   264k      1  0:00:03  0:00:01  0:00:02  264k
    
    timeinoutanomalymetadata
    9901713892195526{'tensor': [2.0, 1.0, 1410.0, 5000.0, 1.0, 0.0, 2.0, 4.0, 7.0, 940.0, 470.0, 47.5531005859, -122.3789978027, 1450.0, 5000.0, 97.0, 0.0, 0.0]}{'variable': [437929.84]}{'count': 0}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'}
    9911713892195526{'tensor': [4.0, 1.5, 1720.0, 6417.0, 1.0, 0.0, 0.0, 3.0, 7.0, 1720.0, 0.0, 47.7267990112, -122.3109970093, 1430.0, 6240.0, 61.0, 0.0, 0.0]}{'variable': [437177.97]}{'count': 0}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'}
    9921713892195526{'tensor': [3.0, 1.75, 1050.0, 9871.0, 1.0, 0.0, 0.0, 5.0, 7.0, 1050.0, 0.0, 47.3815994263, -122.0869979858, 1300.0, 10794.0, 46.0, 0.0, 0.0]}{'variable': [236238.67]}{'count': 0}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'}
    9931713892195526{'tensor': [3.0, 2.25, 1590.0, 11745.0, 1.0, 0.0, 0.0, 3.0, 7.0, 1090.0, 500.0, 47.3553009033, -122.2799987793, 1540.0, 12530.0, 36.0, 0.0, 0.0]}{'variable': [276046.53]}{'count': 0}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'}
    9941713892195526{'tensor': [4.0, 2.5, 2500.0, 8540.0, 2.0, 0.0, 0.0, 3.0, 9.0, 2500.0, 0.0, 47.5759010315, -121.9940032959, 2560.0, 8475.0, 24.0, 0.0, 0.0]}{'variable': [758714.3]}{'count': 0}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'}
    9951713892195526{'tensor': [4.0, 2.5, 2040.0, 9225.0, 1.0, 0.0, 0.0, 5.0, 8.0, 1610.0, 430.0, 47.6360015869, -122.0970001221, 1730.0, 9225.0, 46.0, 0.0, 0.0]}{'variable': [627853.3]}{'count': 0}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'}
    9961713892195526{'tensor': [3.0, 3.0, 1330.0, 1379.0, 2.0, 0.0, 0.0, 4.0, 8.0, 1120.0, 210.0, 47.6125984192, -122.31300354, 1810.0, 1770.0, 9.0, 0.0, 0.0]}{'variable': [450867.7]}{'count': 0}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'}
    9971713892195526{'tensor': [3.0, 2.5, 1880.0, 4499.0, 2.0, 0.0, 0.0, 3.0, 8.0, 1880.0, 0.0, 47.5663986206, -121.9990005493, 2130.0, 5114.0, 22.0, 0.0, 0.0]}{'variable': [553463.25]}{'count': 0}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'}
    9981713892195526{'tensor': [4.0, 1.5, 1200.0, 10890.0, 1.0, 0.0, 0.0, 5.0, 7.0, 1200.0, 0.0, 47.342300415, -122.0879974365, 1250.0, 10139.0, 42.0, 0.0, 0.0]}{'variable': [241330.17]}{'count': 0}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'}
    9991713892195526{'tensor': [4.0, 3.25, 5180.0, 19850.0, 2.0, 0.0, 3.0, 3.0, 12.0, 3540.0, 1640.0, 47.5620002747, -122.1620025635, 3160.0, 9750.0, 9.0, 0.0, 0.0]}{'variable': [1295531.8]}{'count': 0}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'}

    Get the most recent logs to verify the edge records were transmitted. We’ll see in the metadata.partition section the name matching our edge location.

    pipeline.logs(dataset=['time', 'out', 'metadata'])
    
    Warning: There are more logs available. Please set a larger limit or request a file using export_logs.
    
    timeout.variablemetadata.last_modelmetadata.pipeline_versionmetadata.elapsedmetadata.droppedmetadata.partition
    02024-04-23 17:09:55.526[712309.9]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}226508c6-7205-4847-afe3-3d0f62c73096[3660658, 3177750][]houseprice-low-connection-demonstration
    12024-04-23 17:09:55.526[1295531.8]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}226508c6-7205-4847-afe3-3d0f62c73096[3660658, 3177750][]houseprice-low-connection-demonstration
    22024-04-23 17:09:55.526[241330.17]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}226508c6-7205-4847-afe3-3d0f62c73096[3660658, 3177750][]houseprice-low-connection-demonstration
    32024-04-23 17:09:55.526[553463.25]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}226508c6-7205-4847-afe3-3d0f62c73096[3660658, 3177750][]houseprice-low-connection-demonstration
    42024-04-23 17:09:55.526[450867.7]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}226508c6-7205-4847-afe3-3d0f62c73096[3660658, 3177750][]houseprice-low-connection-demonstration
    ........................
    952024-04-23 17:09:55.526[450867.7]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}226508c6-7205-4847-afe3-3d0f62c73096[3660658, 3177750][]houseprice-low-connection-demonstration
    962024-04-23 17:09:55.526[715530.06]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}226508c6-7205-4847-afe3-3d0f62c73096[3660658, 3177750][]houseprice-low-connection-demonstration
    972024-04-23 17:09:55.526[837085.5]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}226508c6-7205-4847-afe3-3d0f62c73096[3660658, 3177750][]houseprice-low-connection-demonstration
    982024-04-23 17:09:55.526[630865.5]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}226508c6-7205-4847-afe3-3d0f62c73096[3660658, 3177750][]houseprice-low-connection-demonstration
    992024-04-23 17:09:55.526[256845.58]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}226508c6-7205-4847-afe3-3d0f62c73096[3660658, 3177750][]houseprice-low-connection-demonstration

    100 rows × 7 columns

    Simulate Connection Loss

    For this example, we’ll simulate a connection loss, then perform another set of inferences, this time from the ./data/biginputs.df.json list. When we retrieve the pipeline logs again from the Wallaroo Ops instance, we should not see any additional records.

    !curl -X POST HOSTNAME:8080/infer \
        -H "Content-Type: Content-Type: application/json; format=pandas-records" \
        --data @./data/biginputs.df.json > ./inferenceoutput2.df.json
    
    # get the dataframe from what we just did
    
    df_result = pd.read_json('./inferenceoutput2.df.json', orient="records")
    df_result.tail(10).loc[:, ['time', 'out', 'metadata']]
    
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100  5786  100  4639  100  1147  13402   3313 --:--:-- --:--:-- --:--:-- 16722
    
    timeoutmetadata
    01713892238998{'variable': [1514079.4]}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [78802, 229304], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'}
    11713892238998{'variable': [1967344.1]}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [78802, 229304], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'}
    21713892238998{'variable': [2002393.6]}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [78802, 229304], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'}
    31713892238998{'variable': [1886959.3]}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [78802, 229304], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'}
    41713892238998{'variable': [1689843.1]}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [78802, 229304], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'}
    51713892238998{'variable': [1946437.8]}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [78802, 229304], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'}
    61713892238998{'variable': [2005883.1]}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [78802, 229304], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'}
    71713892238998{'variable': [1910824.0]}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [78802, 229304], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'}
    81713892238998{'variable': [2016006.1]}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [78802, 229304], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'}

    These are the logs retrieved from the Wallaroo Ops center during the connection loss. We can see the new records from the edge deployment have not been added.

    pipeline.logs(dataset=['time', 'out', 'metadata'])
    
    Warning: There are more logs available. Please set a larger limit or request a file using export_logs.
    
    timeout.variablemetadata.last_modelmetadata.pipeline_versionmetadata.elapsedmetadata.droppedmetadata.partition
    02024-04-23 17:09:55.526[712309.9]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}226508c6-7205-4847-afe3-3d0f62c73096[3660658, 3177750][]houseprice-low-connection-demonstration
    12024-04-23 17:09:55.526[1295531.8]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}226508c6-7205-4847-afe3-3d0f62c73096[3660658, 3177750][]houseprice-low-connection-demonstration
    22024-04-23 17:09:55.526[241330.17]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}226508c6-7205-4847-afe3-3d0f62c73096[3660658, 3177750][]houseprice-low-connection-demonstration
    32024-04-23 17:09:55.526[553463.25]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}226508c6-7205-4847-afe3-3d0f62c73096[3660658, 3177750][]houseprice-low-connection-demonstration
    42024-04-23 17:09:55.526[450867.7]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}226508c6-7205-4847-afe3-3d0f62c73096[3660658, 3177750][]houseprice-low-connection-demonstration
    ........................
    952024-04-23 17:09:55.526[450867.7]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}226508c6-7205-4847-afe3-3d0f62c73096[3660658, 3177750][]houseprice-low-connection-demonstration
    962024-04-23 17:09:55.526[715530.06]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}226508c6-7205-4847-afe3-3d0f62c73096[3660658, 3177750][]houseprice-low-connection-demonstration
    972024-04-23 17:09:55.526[837085.5]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}226508c6-7205-4847-afe3-3d0f62c73096[3660658, 3177750][]houseprice-low-connection-demonstration
    982024-04-23 17:09:55.526[630865.5]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}226508c6-7205-4847-afe3-3d0f62c73096[3660658, 3177750][]houseprice-low-connection-demonstration
    992024-04-23 17:09:55.526[256845.58]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}226508c6-7205-4847-afe3-3d0f62c73096[3660658, 3177750][]houseprice-low-connection-demonstration

    100 rows × 7 columns

    The logs retrieved from the local Edge location, which is not in communication with the Wallaroo Ops instance.

    # local logs
    
    !curl -X POST http://HOSTNAME:8080/logs \
        -H "Content-Type: Content-Type: application/json; format=pandas-records" \
        --data '{}' > ./edge-logs.df.json
    
    df_logs = pd.read_json("./edge-logs.df.json", orient="records")
    df_logs
    
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100  502k  100  502k  100     2   344k      1  0:00:02  0:00:01  0:00:01  344k
    
    timeinoutanomalymetadata
    01713892183771{'tensor': [4.0, 2.5, 2900.0, 5505.0, 2.0, 0.0, 0.0, 3.0, 8.0, 2900.0, 0.0, 47.6063, -122.02, 2970.0, 5251.0, 12.0, 0.0, 0.0]}{'variable': [718013.7]}{'count': 0}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [149203, 331606], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'}
    11713892195526{'tensor': [4.0, 2.75, 3010.0, 7215.0, 2.0, 0.0, 0.0, 3.0, 9.0, 3010.0, 0.0, 47.6952018738, -122.1780014038, 3010.0, 7215.0, 0.0, 0.0, 0.0]}{'variable': [795841.06]}{'count': 0}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'}
    21713892195526{'tensor': [4.0, 1.75, 1400.0, 7920.0, 1.0, 0.0, 0.0, 3.0, 7.0, 1400.0, 0.0, 47.465801239, -122.1839981079, 1910.0, 7700.0, 52.0, 0.0, 0.0]}{'variable': [267013.97]}{'count': 0}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'}
    31713892195526{'tensor': [4.0, 2.5, 3130.0, 13202.0, 2.0, 0.0, 0.0, 3.0, 10.0, 3130.0, 0.0, 47.5877990723, -121.9759979248, 2840.0, 10470.0, 19.0, 0.0, 0.0]}{'variable': [879083.56]}{'count': 0}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'}
    41713892195526{'tensor': [3.0, 2.25, 1620.0, 997.0, 2.5, 0.0, 0.0, 3.0, 8.0, 1540.0, 80.0, 47.5400009155, -122.0260009766, 1620.0, 1068.0, 4.0, 0.0, 0.0]}{'variable': [544392.06]}{'count': 0}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'}
    ..................
    9951713892195526{'tensor': [4.0, 2.5, 2040.0, 9225.0, 1.0, 0.0, 0.0, 5.0, 8.0, 1610.0, 430.0, 47.6360015869, -122.0970001221, 1730.0, 9225.0, 46.0, 0.0, 0.0]}{'variable': [627853.3]}{'count': 0}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'}
    9961713892195526{'tensor': [3.0, 3.0, 1330.0, 1379.0, 2.0, 0.0, 0.0, 4.0, 8.0, 1120.0, 210.0, 47.6125984192, -122.31300354, 1810.0, 1770.0, 9.0, 0.0, 0.0]}{'variable': [450867.7]}{'count': 0}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'}
    9971713892195526{'tensor': [3.0, 2.5, 1880.0, 4499.0, 2.0, 0.0, 0.0, 3.0, 8.0, 1880.0, 0.0, 47.5663986206, -121.9990005493, 2130.0, 5114.0, 22.0, 0.0, 0.0]}{'variable': [553463.25]}{'count': 0}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'}
    9981713892195526{'tensor': [4.0, 1.5, 1200.0, 10890.0, 1.0, 0.0, 0.0, 5.0, 7.0, 1200.0, 0.0, 47.342300415, -122.0879974365, 1250.0, 10139.0, 42.0, 0.0, 0.0]}{'variable': [241330.17]}{'count': 0}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'}
    9991713892195526{'tensor': [4.0, 3.25, 5180.0, 19850.0, 2.0, 0.0, 3.0, 3.0, 12.0, 3540.0, 1640.0, 47.5620002747, -122.1620025635, 3160.0, 9750.0, 9.0, 0.0, 0.0]}{'variable': [1295531.8]}{'count': 0}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '226508c6-7205-4847-afe3-3d0f62c73096', 'elapsed': [3660658, 3177750], 'dropped': [], 'partition': 'houseprice-low-connection-demonstration'}

    1000 rows × 5 columns

    We’ll now reenable communication, then wait 60 seconds and check the pipeline inference logs to see if communication is restored.

    import time
    time.sleep(60)
    
    import json
    
    df = pd.DataFrame.from_records({"tensor": [[4.0, 
                                                2.5, 
                                                2900.0, 
                                                5505.0, 
                                                2.0, 
                                                0.0, 
                                                0.0, 
                                                3.0, 
                                                8.0, 
                                                2900.0, 
                                                0.0, 
                                                47.6063, 
                                                -122.02, 
                                                2970.0, 
                                                5251.0, 
                                                12.0, 
                                                0.0, 
                                                0.0]]})
    
    data = df.to_dict(orient="records")
    
    !curl -X POST HOSTNAME:8080/infer \
        -H "Content-Type: Content-Type: application/json; format=pandas-records" \
        --data '{json.dumps(data)}' > ./inferenceoutput.df.json
    
    # get the dataframe from what we just did
    
    df_result = pd.read_json('./inferenceoutput.df.json', orient="records")
    df_result.loc[:, ['time', 'out', 'metadata']]
    
    pipeline.logs(dataset=['time', 'out', 'metadata'])
    
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100   625  100   497  100   128   1242    319 --:--:-- --:--:-- --:--:--  1562
    
    Warning: There are more logs available. Please set a larger limit or request a file using export_logs.
    
    timeout.variablemetadata.last_modelmetadata.pipeline_versionmetadata.elapsedmetadata.droppedmetadata.partition
    02024-04-23 17:12:55.904[718013.7]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}226508c6-7205-4847-afe3-3d0f62c73096[46701, 199803][]houseprice-low-connection-demonstration
    12024-04-23 17:10:38.998[2016006.1]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}226508c6-7205-4847-afe3-3d0f62c73096[78802, 229304][]houseprice-low-connection-demonstration
    22024-04-23 17:10:38.998[1910824.0]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}226508c6-7205-4847-afe3-3d0f62c73096[78802, 229304][]houseprice-low-connection-demonstration
    32024-04-23 17:10:38.998[2005883.1]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}226508c6-7205-4847-afe3-3d0f62c73096[78802, 229304][]houseprice-low-connection-demonstration
    42024-04-23 17:10:38.998[1946437.8]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}226508c6-7205-4847-afe3-3d0f62c73096[78802, 229304][]houseprice-low-connection-demonstration
    ........................
    952024-04-23 17:09:55.526[726181.75]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}226508c6-7205-4847-afe3-3d0f62c73096[3660658, 3177750][]houseprice-low-connection-demonstration
    962024-04-23 17:09:55.526[836230.2]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}226508c6-7205-4847-afe3-3d0f62c73096[3660658, 3177750][]houseprice-low-connection-demonstration
    972024-04-23 17:09:55.526[291903.97]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}226508c6-7205-4847-afe3-3d0f62c73096[3660658, 3177750][]houseprice-low-connection-demonstration
    982024-04-23 17:09:55.526[448627.8]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}226508c6-7205-4847-afe3-3d0f62c73096[3660658, 3177750][]houseprice-low-connection-demonstration
    992024-04-23 17:09:55.526[1060847.5]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}226508c6-7205-4847-afe3-3d0f62c73096[3660658, 3177750][]houseprice-low-connection-demonstration

    100 rows × 7 columns

    Tutorials