Edge Observabilty with No/Low Connection Tutorial

A demonstration model deployments on edge location with low or no connectivity.
Features:
Models:

This tutorial and the assets can be downloaded as part of the Wallaroo Tutorials repository.

Airgapped Edge Observability with No/Low Connection Tutorial

Wallaroo pipeline publishes are packaged versions of the pipeline, models, and the inference engine published to an Open Container Initiative (OCI) Registry. Once published, they are used to deploy models to edge devices and used for inference requests.

Edge locations added to a pipeline published allow deployments on edge devices to connect with the Wallaroo Model Ops Center instance and transmit their inference results as part of the pipeline logs.

Edge Location Observability

Inference request logs are stored locally in the edge device. This storage space is configurable during model deployment on the edge devices via the LOCAL_INFERENCE_STORAGE flag. Inference logs are stored on the edge devices until the assigned storage capacity is reached. At that point, logs are pruned on a “FIFO” - First In First Out method, where the first inference logs stored are the one pruned first.

In conditions where the edge device is unable to connect with the Wallaroo Model Ops Center instance for an extended period of time, logs are stored until connection is restored to the Wallaroo Model Ops Center instance. When the connection is restored, logs are transmitted back to the Wallaroo Model Ops Center instance and added to the pipeline where the edge location is assigned to.

Edge inference logs have the inference DateTime is set to the DateTime the inference occurred on the edge device, not when the logs are uploaded to the Wallaroo Model Ops Center. Assays and other features that use inference logs are updated when they perform analyses with any new inference records uploaded by edge devices. This insures that inference log information is synced across different edge devices without concern for when the data was uploaded, just when the inference occurred.

Goals

This tutorial will perform the following:

  1. Upload and deploy a sample real estate house price estimation to a Wallaroo Model Ops Center instance.
  2. Publish the pipeline and deploy to an edge device.
  3. Perform sample inferences on the edge devices.
    1. Show inference logs from the local edge device.
    2. Show inference logs submitted to the Wallaroo Model Ops Center instance from the edge device.
  4. Simulate a loss of communication on the Wallaroo Model Ops Center instance from the edge device. Perform additional sample inferences on the edge device.
    1. Show new inference logs on the edge device.
    2. Show new inference logs are not yet transmitted to the Wallaroo Model Ops Center instance.
  5. Simulate a restoration of communication on the Wallaroo Model Ops Center instance and show the inference logs from the edge device are restored to the Wallaroo Model Ops Center instance.

Resources

This tutorial provides the following:

  • Models:
    • ./models/rf_model.onnx: The champion model that predicts house prices.
  • Data:
    • ./data/normal-inputs.df.json: A set of inference requests across a span of different house types and values.
    • ./data/smallinputs.df.json: A set of inference requests that tend to generate house price predictions with low values.
    • ./data/biginputs.df.json: A set of inference requests that tend to generate house price predictions with high values.

Prerequisites

Edge Deployment Preparation Steps

The following is used to set up the environment that is used for the pipeline publish replacement and in-line model updates on an edge tutorial.

In these steps, we will:

  • Deploying a sample ML model used to determine house prices based on a set of input parameters.
  • Publish the model deployment configuration to an OCI registry.
  • Use the publish and set edge locations.
  • Deploy the model in an edge location and perform sample inferences.

Import Libraries

The first step will be to import our libraries, and set variables used through this tutorial.

import wallaroo
from wallaroo.object import EntityNotFoundError
from wallaroo.framework import Framework

from IPython.display import display

# used to display DataFrame information without truncating
from IPython.display import display
import pandas as pd
pd.set_option('display.max_colwidth', None)
pd.set_option('display.max_columns', None)

import json
import time

workspace_name = 'edge-low-connection-demonstration'

# ignoring warnings for demonstration
import warnings
warnings.filterwarnings('ignore')

# used to display DataFrame information without truncating
from IPython.display import display
import pandas as pd
pd.set_option('display.max_colwidth', None)
pd.set_option('display.max_columns', None)

Connect to the Wallaroo Instance

The first step is to connect to Wallaroo through the Wallaroo client. The Python library is included in the Wallaroo install and available through the Jupyter Hub interface provided with your Wallaroo environment.

This is accomplished using the wallaroo.Client() command, which provides a URL to grant the SDK permission to your specific Wallaroo environment. When displayed, enter the URL into a browser and confirm permissions. Store the connection into a variable that can be referenced later.

If logging into the Wallaroo instance through the internal JupyterHub service, use wl = wallaroo.Client(). For more information on Wallaroo Client settings, see the Client Connection guide.

# Login through local Wallaroo instance

wl = wallaroo.Client()

Create Workspace

We will create a workspace to manage our pipeline and models. The following variables will set the name of our sample workspace then set it as the current workspace.

Workspace names are unique across the Wallaroo instance. If a user requests a workspace they have not been granted access to, an error message is returned.

workspace = wl.get_workspace(name=workspace_name, create_if_not_exist=True)

wl.set_current_workspace(workspace)

{'name': 'edge-low-connection-demonstration', 'id': 15, 'archived': False, 'created_by': '1a819833-f4ef-4298-8065-1785a7014681', 'created_at': '2025-07-15T20:46:01.008763+00:00', 'models': [], 'pipelines': []}

Upload The Champion Model

For our example, we will upload the champion model that has been trained to derive house prices from a variety of inputs. The model file is rf_model.onnx, and is uploaded with the name rf-house-price-estimator.

housing_model_control = (wl.upload_model("rf-house-price-estimator", 
                                        './models/rf_model.onnx', 
                                        framework=Framework.ONNX)
                                        .configure(tensor_fields=["tensor"])
                        )

Build the Pipeline

This pipeline is made to be an example of an existing situation where a model is deployed and being used for inferences in a production environment. We’ll call it edge-inline-replacement-demonstration, set housing_model_control as a pipeline step.

pipeline = wl.build_pipeline('edge-low-connection-example')

# clear the steps if used before
pipeline.clear()

_ = pipeline.add_model_step(housing_model_control)

Edge Deployment

We can now deploy the original pipeline to an edge device. This will require the following steps:

  • Publish the pipeline: Publishes the pipeline to the OCI registry.
  • Add Edge: Add the edge location to the pipeline publish.
  • Deploy Edge: Deploy the edge device with the edge location settings.

Publish Pipeline

Publishing the pipeline uses the pipeline wallaroo.pipeline.publish() command. This requires that the Wallaroo Model Ops Center instance have Edge Registry Services enabled.

The following publishes the pipeline to the OCI registry and displays the container details. For more information, see Wallaroo SDK Essentials Guide: Pipeline Edge Publication.

#minimum deployment config
deploy_config = wallaroo.DeploymentConfigBuilder() \
                    .replica_count(1) \
                    .cpus(0.5) \
                    .memory("1Gi") \
                    .build()

pub = pipeline.publish(deployment_config = deploy_config)

Waiting for pipeline publish... It may take up to 600 sec.
Pipeline is publishing..... Published.

Add Edge Location

The edge location is added with the wallaroo.pipeline_publish.add_edge(name) method. This returns the OCI registration information, and the EDGE_BUNDLE information. The EDGE_BUNDLE data is a base64 encoded set of parameters for the pipeline that the edge device is associated with, the workspace, and other data.

For full details, see Wallaroo SDK Essentials Guide: Pipeline Edge Publication: Airgapped Edge Observability.

For this example, we will add a location:

  • houseprice-low-connection-example-01
edge_name_01 = "houseprice-low-connection-example-01"
edge_publish_01 = pub.add_edge(edge_name_01)
display(edge_publish_01)
ID6
Pipeline Nameedge-low-connection-example
Pipeline Version93b30c7d-d8fb-43ea-82e4-db31179fce1a
StatusPublished
Workspace Id15
Workspace Nameedge-low-connection-demonstration
Edgeshouseprice-low-connection-example-01
Engine URLghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/fitzroy-mini:v2025.1.0-6250
Pipeline URLghcr.io/wallaroolabs/doc-samples/pipelines/edge-low-connection-example:93b30c7d-d8fb-43ea-82e4-db31179fce1a
Helm Chart URLoci://ghcr.io/wallaroolabs/doc-samples/charts/edge-low-connection-example
Helm Chart Referenceghcr.io/wallaroolabs/doc-samples/charts@sha256:7c2d774b96171351554b16d79c8cd400673fd4ea0df81c8bd78336e4bc21fa83
Helm Chart Version0.0.1-93b30c7d-d8fb-43ea-82e4-db31179fce1a
Engine Config{'engine': {'resources': {'limits': {'cpu': 0.5, 'memory': '1Gi'}, 'requests': {'cpu': 0.5, 'memory': '1Gi'}, 'accel': 'none', 'arch': 'x86', 'gpu': False}}, 'engineAux': {'autoscale': {'type': 'none', 'cpu_utilization': 50.0}, 'images': {}}}
User Images[]
Created Byjohn.hansarick@wallaroo.ai
Created At2025-07-16 15:50:43.705943+00:00
Updated At2025-07-16 15:50:43.705943+00:00
Replaces
Docker Run Command
docker run -v $PERSISTENT_VOLUME_DIR:/persist \
    -p $EDGE_PORT:8080 \
    -e OCI_USERNAME=$OCI_USERNAME \
    -e OCI_PASSWORD=$OCI_PASSWORD \
    -e PIPELINE_URL=ghcr.io/wallaroolabs/doc-samples/pipelines/edge-low-connection-example:93b30c7d-d8fb-43ea-82e4-db31179fce1a\
    -e EDGE_BUNDLE=ZXhwb3J0IEJVTkRMRV9WRVJTSU9OPTEKZXhwb3J0IENPTkZJR19DUFVTPTEKZXhwb3J0IEVER0VfTkFNRT1ob3VzZXByaWNlLWxvdy1jb25uZWN0aW9uLWV4YW1wbGUtMDEKZXhwb3J0IE9QU0NFTlRFUl9IT1NUPWRvYy10ZXN0LndhbGxhcm9vY29tbXVuaXR5Lm5pbmphCmV4cG9ydCBQSVBFTElORV9VUkw9Z2hjci5pby93YWxsYXJvb2xhYnMvZG9jLXNhbXBsZXMvcGlwZWxpbmVzL2VkZ2UtbG93LWNvbm5lY3Rpb24tZXhhbXBsZTo5M2IzMGM3ZC1kOGZiLTQzZWEtODJlNC1kYjMxMTc5ZmNlMWEKZXhwb3J0IEpPSU5fVE9LRU49MWFhMjU2ZjMtM2UzNy00YTMxLWJiM2YtODBiYTNlOTNjYjVmCmV4cG9ydCBPQ0lfUkVHSVNUUlk9Z2hjci5pbw== \
    -e CONFIG_CPUS=1.0 --cpus=0.5 --memory=1g \
    ghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/fitzroy-mini:v2025.1.0-6250

Note: Please set the PERSISTENT_VOLUME_DIR, EDGE_PORT, OCI_USERNAME, and OCI_PASSWORD environment variables.
Podman Run Command
podman run -v $PERSISTENT_VOLUME_DIR:/persist \
    -p $EDGE_PORT:8080 \
    -e OCI_USERNAME=$OCI_USERNAME \
    -e OCI_PASSWORD=$OCI_PASSWORD \
    -e PIPELINE_URL=ghcr.io/wallaroolabs/doc-samples/pipelines/edge-low-connection-example:93b30c7d-d8fb-43ea-82e4-db31179fce1a\
    -e EDGE_BUNDLE=ZXhwb3J0IEJVTkRMRV9WRVJTSU9OPTEKZXhwb3J0IENPTkZJR19DUFVTPTEKZXhwb3J0IEVER0VfTkFNRT1ob3VzZXByaWNlLWxvdy1jb25uZWN0aW9uLWV4YW1wbGUtMDEKZXhwb3J0IE9QU0NFTlRFUl9IT1NUPWRvYy10ZXN0LndhbGxhcm9vY29tbXVuaXR5Lm5pbmphCmV4cG9ydCBQSVBFTElORV9VUkw9Z2hjci5pby93YWxsYXJvb2xhYnMvZG9jLXNhbXBsZXMvcGlwZWxpbmVzL2VkZ2UtbG93LWNvbm5lY3Rpb24tZXhhbXBsZTo5M2IzMGM3ZC1kOGZiLTQzZWEtODJlNC1kYjMxMTc5ZmNlMWEKZXhwb3J0IEpPSU5fVE9LRU49MWFhMjU2ZjMtM2UzNy00YTMxLWJiM2YtODBiYTNlOTNjYjVmCmV4cG9ydCBPQ0lfUkVHSVNUUlk9Z2hjci5pbw== \
    -e CONFIG_CPUS=1.0 --cpus=0.5 --memory=1g \
    ghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/fitzroy-mini:v2025.1.0-6250

Note: Please set the PERSISTENT_VOLUME_DIR, EDGE_PORT, OCI_USERNAME, and OCI_PASSWORD environment variables.
Helm Install Command
helm install --atomic $HELM_INSTALL_NAME \
    oci://ghcr.io/wallaroolabs/doc-samples/charts/edge-low-connection-example \
    --namespace $HELM_INSTALL_NAMESPACE \
    --version 0.0.1-93b30c7d-d8fb-43ea-82e4-db31179fce1a \
    --set ociRegistry.username=$OCI_USERNAME \
    --set ociRegistry.password=$OCI_PASSWORD \
    --set edgeBundle=ZXhwb3J0IEJVTkRMRV9WRVJTSU9OPTEKZXhwb3J0IENPTkZJR19DUFVTPTEKZXhwb3J0IEVER0VfTkFNRT1ob3VzZXByaWNlLWxvdy1jb25uZWN0aW9uLWV4YW1wbGUtMDEKZXhwb3J0IE9QU0NFTlRFUl9IT1NUPWRvYy10ZXN0LndhbGxhcm9vY29tbXVuaXR5Lm5pbmphCmV4cG9ydCBQSVBFTElORV9VUkw9Z2hjci5pby93YWxsYXJvb2xhYnMvZG9jLXNhbXBsZXMvcGlwZWxpbmVzL2VkZ2UtbG93LWNvbm5lY3Rpb24tZXhhbXBsZTo5M2IzMGM3ZC1kOGZiLTQzZWEtODJlNC1kYjMxMTc5ZmNlMWEKZXhwb3J0IEpPSU5fVE9LRU49MWFhMjU2ZjMtM2UzNy00YTMxLWJiM2YtODBiYTNlOTNjYjVmCmV4cG9ydCBPQ0lfUkVHSVNUUlk9Z2hjci5pbw==

Note: Please set the HELM_INSTALL_NAME, HELM_INSTALL_NAMESPACE, OCI_USERNAME, and OCI_PASSWORD environment variables.

DevOps Deployment

The following shows two methods of model deployments on edge devices. For more details on model edge deployments with Wallaroo, see Model Operations: Run Anywhere.

Docker Run Based Deployment

For the model deployment to the edge location via docker run, use the Docker Run Command displayed with the pipeline publish. The following variables must be set first.

  • $EDGE_PORT: The external port to connect to the model deployment.
  • $OCI_USERNAME: The username for the OCI registry service.
  • $OCI_PASSWORD: The password or token for the OCI registry service.

Edge location inference log storage capacity is set with the LOCAL_INFERENCE_STORAGE. This sets amount of storage to allocate for the edge deployments inference log storage capacity. This is in the format {size as number}{unit value}. The accepted unit values are:

  • Ki (for KiloBytes)
  • Mi (for MegaBytes)
  • Gi (for GigaBytes)
  • Ti (for TeraBytes)

For example, to set the inference storage capacity to 512Mi, the following command would be used:

docker run -v $PERSISTENT_VOLUME_DIR:/persist \
    -p $EDGE_PORT:8080 \
    -e OCI_USERNAME=$OCI_USERNAME \
    -e OCI_PASSWORD=$OCI_PASSWORD \
    -e LOCAL_INFERENCE_STORAGE=512Mi \ # sets the log storage to 512MB
    -e PIPELINE_URL=ghcr.io/wallaroolabs/doc-samples/pipelines/edge-low-connection-demo:cc4a10a2-4866-4ed6-a1d8-8b731e7bf8f8\
    -e EDGE_BUNDLE=ZXhwb3J0IEJVTkRMRV9WRVJTSU9OPTEKZXhwb3J0IENPTkZJR19DUFVTPTQKZXhwb3J0IEVER0VfTkFNRT1ob3VzZXByaWNlLWxvdy1jb25uZWN0aW9uLWRlbW9uc3RyYXRpb24tMDEKZXhwb3J0IE9QU0NFTlRFUl9IT1NUPWRvYy10ZXN0LmVkZ2Uud2FsbGFyb29jb21tdW5pdHkubmluamEKZXhwb3J0IFBJUEVMSU5FX1VSTD1naGNyLmlvL3dhbGxhcm9vbGFicy9kb2Mtc2FtcGxlcy9waXBlbGluZXMvZWRnZS1sb3ctY29ubmVjdGlvbi1kZW1vOmNjNGExMGEyLTQ4NjYtNGVkNi1hMWQ4LThiNzMxZTdiZjhmOApleHBvcnQgSk9JTl9UT0tFTj1iMmEzZGVlMS0zYjQzLTRhNmQtOGEzMS00MDk1ZjE1YmY5MmIKZXhwb3J0IE9DSV9SRUdJU1RSWT1naGNyLmlv \
    -e CONFIG_CPUS=4 ghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/fitzroy-mini:v2024.1.0-main-4963

Edge Inference Example

We will perform sample inferences on the current model edge deployment to demonstrate how the inferences change as the models are updated through the in-line edge deployments.

With the model deployed on the edge location, we will perform sample inferences on the edge deployment. For this example, the hostname is testboy.local. Adjust the hostname to fit your edge location deployment.

The following endpoints are available.

EndpointTypeDescription
/modelsGETReturns the models and model versions used in the edge deployment.
/pipelinesGETReturns the pipeline ID and status of the pipeline used in the edge deployment.
/inferPOSTThe inference request endpoint. This accepts either an Apache Arrow table, or a JSON in pandas record format.
/inferPOSTThe inference result logs endpoint.

To enable the inference results logs endpoint, we add two additional environmental variables for the docker run command:

  • LOCAL_INFERENCE_STORAGE (Optional): Sets amount of storage to allocate for the edge deployments inference log storage capacity. This is in the format {size as number}{unit value}. The values are similar to the Kubernetes memory resource units format. If used, must be used with PLATEAU_PAGE_SIZE. The accepted unit values are:
    • Ki (for KiloBytes)
    • Mi (for MegaBytes)
    • Gi (for GigaBytes)
    • Ti (for TeraBytes)
  • PLATEAU_PAGE_SIZE (Optional): How many inference log rows to upload from the edge deployment at a time. Must be used with LOCAL_INFERENCE_STORAGE.

The deployment becomes:

docker run -v $PERSISTENT_VOLUME_DIR:/persist \
    -p $EDGE_PORT:8080 \
    -e OCI_USERNAME=$OCI_USERNAME \
    -e OCI_PASSWORD=$OCI_PASSWORD \
    -e PIPELINE_URL=ghcr.io/wallaroolabs/doc-samples/pipelines/edge-low-connection-demonstration:a8c50aab-5227-4a36-bb66-f34086ff65f4\
    -e EDGE_BUNDLE=ZXhwb3J0IEJVTkRMRV9WRVJTSU9OPTEKZXhwb3J0IENPTkZJR19DUFVTPTEKZXhwb3J0IEVER0VfTkFNRT1ob3VzZXByaWNlLWxvdy1jb25uZWN0aW9uLWRlbW9uc3RyYXRpb24KZXhwb3J0IE9QU0NFTlRFUl9IT1NUPWRvYy10ZXN0LndhbGxhcm9vY29tbXVuaXR5Lm5pbmphCmV4cG9ydCBQSVBFTElORV9VUkw9Z2hjci5pby93YWxsYXJvb2xhYnMvZG9jLXNhbXBsZXMvcGlwZWxpbmVzL2VkZ2UtbG93LWNvbm5lY3Rpb24tZGVtb25zdHJhdGlvbjphOGM1MGFhYi01MjI3LTRhMzYtYmI2Ni1mMzQwODZmZjY1ZjQKZXhwb3J0IEpPSU5fVE9LRU49YTk3YjAwOGItODUyZC00YzBiLWE3MzUtYjY0MDAwNzU0YWExCmV4cG9ydCBPQ0lfUkVHSVNUUlk9Z2hjci5pbw== \
    -e CONFIG_CPUS=1.0 --cpus=0.5 --memory=1g \
    -e LOCAL_INFERENCE_STORAGE=100m \
    -e PLATEAU_PAGE_SIZE=100 \
    ghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/fitzroy-mini:v2025.1.0-6250

For more details on model edge deployments with Wallaroo, see Model Operations: Run Anywhere.

First we’ll retrieve the model. We should see the same name we set when we uploaded it to the Wallaroo instance.

!curl testboy.lan:8081/models

{"models":[{"sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6","name":"rf-house-price-estimator","version":"bf054557-1503-4d01-88c3-07708e0340c9","status":"Running","model_version_id":24}]}

Now we’ll retrieve the pipeline details, and verify the name is the same as set earlier and it is running.

!curl testboy.lan:8081/pipelines

{"pipelines":[{"id":"edge-low-connection-example","version":"93b30c7d-d8fb-43ea-82e4-db31179fce1a","status":"Running"}]}

Now do the infer with the same DataFrame we used for our sample inference when the model was deployed in the Wallaroo instance.

import json

df = pd.DataFrame.from_records({"tensor": [[4.0, 
                                            2.5, 
                                            2900.0, 
                                            5505.0, 
                                            2.0, 
                                            0.0, 
                                            0.0, 
                                            3.0, 
                                            8.0, 
                                            2900.0, 
                                            0.0, 
                                            47.6063, 
                                            -122.02, 
                                            2970.0, 
                                            5251.0, 
                                            12.0, 
                                            0.0, 
                                            0.0]]})

data = df.to_dict(orient="records")

!curl -X POST testboy.lan:8081/infer \
    -H "Content-Type: Content-Type: application/json; format=pandas-records" \
    --data '{json.dumps(data)}' > ./inferenceoutput.df.json

# get the dataframe from what we just did

df_result = pd.read_json('./inferenceoutput.df.json', orient="records")
df_result.loc[:, ['time', 'out', 'metadata']]

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   623  100   495  100   128  34494   8919 --:--:-- --:--:-- --:--:-- 44500
timeoutmetadata
01752681130155{'variable': [718013.7]}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '93b30c7d-d8fb-43ea-82e4-db31179fce1a', 'elapsed': [305045, 818254], 'dropped': [], 'partition': 'houseprice-low-connection-example-01'}
pipeline.logs(dataset=['time', 'out', 'metadata'])
timeout.variablemetadata.last_modelmetadata.pipeline_versionmetadata.elapsedmetadata.droppedmetadata.partition
02025-07-16 15:52:10.155[718013.7]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}93b30c7d-d8fb-43ea-82e4-db31179fce1a[305045, 818254][]houseprice-low-connection-example-01

Let’s do a request with more records.

import datetime
import time

!curl -X POST http://testboy.lan:8081/infer \
    -H "Content-Type: Content-Type: application/json; format=pandas-records" \
    --data @./data/smallinputs.df.json > ./inferenceoutput2.df.json

# get the dataframe from what we just did

df_result = pd.read_json('./inferenceoutput2.df.json', orient="records")
df_result.tail(10).loc[:, ['time', 'out', 'metadata']]

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  622k  100  500k  100  121k  5358k      0 --:--:-- --:--:-- --:--:--     01297k --:--:-- --:--:-- --:--:-- 6688k
timeoutmetadata
9901752681140571{'variable': [437177.97]}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '93b30c7d-d8fb-43ea-82e4-db31179fce1a', 'elapsed': [7984271, 14221794], 'dropped': [], 'partition': 'houseprice-low-connection-example-01'}
9911752681140571{'variable': [236238.67]}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '93b30c7d-d8fb-43ea-82e4-db31179fce1a', 'elapsed': [7984271, 14221794], 'dropped': [], 'partition': 'houseprice-low-connection-example-01'}
9921752681140571{'variable': [276046.53]}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '93b30c7d-d8fb-43ea-82e4-db31179fce1a', 'elapsed': [7984271, 14221794], 'dropped': [], 'partition': 'houseprice-low-connection-example-01'}
9931752681140571{'variable': [758714.3]}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '93b30c7d-d8fb-43ea-82e4-db31179fce1a', 'elapsed': [7984271, 14221794], 'dropped': [], 'partition': 'houseprice-low-connection-example-01'}
9941752681140571{'variable': [627853.3]}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '93b30c7d-d8fb-43ea-82e4-db31179fce1a', 'elapsed': [7984271, 14221794], 'dropped': [], 'partition': 'houseprice-low-connection-example-01'}
9951752681140571{'variable': [450867.7]}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '93b30c7d-d8fb-43ea-82e4-db31179fce1a', 'elapsed': [7984271, 14221794], 'dropped': [], 'partition': 'houseprice-low-connection-example-01'}
9961752681140571{'variable': [553463.25]}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '93b30c7d-d8fb-43ea-82e4-db31179fce1a', 'elapsed': [7984271, 14221794], 'dropped': [], 'partition': 'houseprice-low-connection-example-01'}
9971752681140571{'variable': [241330.17]}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '93b30c7d-d8fb-43ea-82e4-db31179fce1a', 'elapsed': [7984271, 14221794], 'dropped': [], 'partition': 'houseprice-low-connection-example-01'}
9981752681140571{'variable': [1295531.8]}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '93b30c7d-d8fb-43ea-82e4-db31179fce1a', 'elapsed': [7984271, 14221794], 'dropped': [], 'partition': 'houseprice-low-connection-example-01'}
9991752681140571{'variable': [712309.9]}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '93b30c7d-d8fb-43ea-82e4-db31179fce1a', 'elapsed': [7984271, 14221794], 'dropped': [], 'partition': 'houseprice-low-connection-example-01'}

Get the most recent logs to verify the edge records were transmitted. We’ll see in the metadata.partition section the name matching our edge location.

pipeline.logs(dataset=['time', 'out', 'metadata'])

Warning: There are more logs available. Please set a larger limit or request a file using export_logs.
timeout.variablemetadata.last_modelmetadata.pipeline_versionmetadata.elapsedmetadata.droppedmetadata.partition
02025-07-16 15:52:55.743[2016006.1]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}93b30c7d-d8fb-43ea-82e4-db31179fce1a[117899, 522351][]houseprice-low-connection-example-01
12025-07-16 15:52:55.743[1910824.0]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}93b30c7d-d8fb-43ea-82e4-db31179fce1a[117899, 522351][]houseprice-low-connection-example-01
22025-07-16 15:52:55.743[2005883.1]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}93b30c7d-d8fb-43ea-82e4-db31179fce1a[117899, 522351][]houseprice-low-connection-example-01
32025-07-16 15:52:55.743[1946437.8]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}93b30c7d-d8fb-43ea-82e4-db31179fce1a[117899, 522351][]houseprice-low-connection-example-01
42025-07-16 15:52:55.743[1689843.1]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}93b30c7d-d8fb-43ea-82e4-db31179fce1a[117899, 522351][]houseprice-low-connection-example-01
........................
952025-07-16 15:52:20.571[836230.2]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}93b30c7d-d8fb-43ea-82e4-db31179fce1a[7984271, 14221794][]houseprice-low-connection-example-01
962025-07-16 15:52:20.571[291903.97]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}93b30c7d-d8fb-43ea-82e4-db31179fce1a[7984271, 14221794][]houseprice-low-connection-example-01
972025-07-16 15:52:20.571[448627.8]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}93b30c7d-d8fb-43ea-82e4-db31179fce1a[7984271, 14221794][]houseprice-low-connection-example-01
982025-07-16 15:52:20.571[1060847.5]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}93b30c7d-d8fb-43ea-82e4-db31179fce1a[7984271, 14221794][]houseprice-low-connection-example-01
992025-07-16 15:52:20.571[239734.08]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}93b30c7d-d8fb-43ea-82e4-db31179fce1a[7984271, 14221794][]houseprice-low-connection-example-01

100 rows × 7 columns

Simulate Connection Loss

For this example, we’ll simulate a connection loss, then perform another set of inferences, this time from the ./data/biginputs.df.json list. When we retrieve the pipeline logs again from the Wallaroo Model Ops Center instance, we should not see any additional records.

!curl -X POST testboy.lan:8081/infer \
    -H "Content-Type: Content-Type: application/json; format=pandas-records" \
    --data @./data/biginputs.df.json > ./inferenceoutput2.df.json

# get the dataframe from what we just did

df_result = pd.read_json('./inferenceoutput2.df.json', orient="records")
df_result.tail(10).loc[:, ['time', 'out', 'metadata']]

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  5768  100  4621  100  1147   154k  39263 --:--:-- --:--:-- --:--:--     0--:--:--  194k
timeoutmetadata
01752681347335{'variable': [1514079.4]}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '93b30c7d-d8fb-43ea-82e4-db31179fce1a', 'elapsed': [116252, 471814], 'dropped': [], 'partition': 'houseprice-low-connection-example-01'}
11752681347335{'variable': [1967344.1]}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '93b30c7d-d8fb-43ea-82e4-db31179fce1a', 'elapsed': [116252, 471814], 'dropped': [], 'partition': 'houseprice-low-connection-example-01'}
21752681347335{'variable': [2002393.6]}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '93b30c7d-d8fb-43ea-82e4-db31179fce1a', 'elapsed': [116252, 471814], 'dropped': [], 'partition': 'houseprice-low-connection-example-01'}
31752681347335{'variable': [1886959.3]}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '93b30c7d-d8fb-43ea-82e4-db31179fce1a', 'elapsed': [116252, 471814], 'dropped': [], 'partition': 'houseprice-low-connection-example-01'}
41752681347335{'variable': [1689843.1]}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '93b30c7d-d8fb-43ea-82e4-db31179fce1a', 'elapsed': [116252, 471814], 'dropped': [], 'partition': 'houseprice-low-connection-example-01'}
51752681347335{'variable': [1946437.8]}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '93b30c7d-d8fb-43ea-82e4-db31179fce1a', 'elapsed': [116252, 471814], 'dropped': [], 'partition': 'houseprice-low-connection-example-01'}
61752681347335{'variable': [2005883.1]}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '93b30c7d-d8fb-43ea-82e4-db31179fce1a', 'elapsed': [116252, 471814], 'dropped': [], 'partition': 'houseprice-low-connection-example-01'}
71752681347335{'variable': [1910824.0]}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '93b30c7d-d8fb-43ea-82e4-db31179fce1a', 'elapsed': [116252, 471814], 'dropped': [], 'partition': 'houseprice-low-connection-example-01'}
81752681347335{'variable': [2016006.1]}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '93b30c7d-d8fb-43ea-82e4-db31179fce1a', 'elapsed': [116252, 471814], 'dropped': [], 'partition': 'houseprice-low-connection-example-01'}

These are the logs retrieved from the Wallaroo Model Ops Center center during the connection loss. We can see the new records from the edge deployment have not been added.

pipeline.logs(dataset=['time', 'out', 'metadata'])

ERROR:root:Keycloak token refresh got error: 400 - {"error":"invalid_grant","error_description":"Session not active"}
Warning: There are more logs available. Please set a larger limit or request a file using export_logs.
timeout.variablemetadata.last_modelmetadata.pipeline_versionmetadata.elapsedmetadata.droppedmetadata.partition
02025-07-16 15:52:55.743[2016006.1]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}93b30c7d-d8fb-43ea-82e4-db31179fce1a[117899, 522351][]houseprice-low-connection-example-01
12025-07-16 15:52:55.743[1910824.0]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}93b30c7d-d8fb-43ea-82e4-db31179fce1a[117899, 522351][]houseprice-low-connection-example-01
22025-07-16 15:52:55.743[2005883.1]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}93b30c7d-d8fb-43ea-82e4-db31179fce1a[117899, 522351][]houseprice-low-connection-example-01
32025-07-16 15:52:55.743[1946437.8]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}93b30c7d-d8fb-43ea-82e4-db31179fce1a[117899, 522351][]houseprice-low-connection-example-01
42025-07-16 15:52:55.743[1689843.1]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}93b30c7d-d8fb-43ea-82e4-db31179fce1a[117899, 522351][]houseprice-low-connection-example-01
........................
952025-07-16 15:52:20.571[836230.2]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}93b30c7d-d8fb-43ea-82e4-db31179fce1a[7984271, 14221794][]houseprice-low-connection-example-01
962025-07-16 15:52:20.571[291903.97]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}93b30c7d-d8fb-43ea-82e4-db31179fce1a[7984271, 14221794][]houseprice-low-connection-example-01
972025-07-16 15:52:20.571[448627.8]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}93b30c7d-d8fb-43ea-82e4-db31179fce1a[7984271, 14221794][]houseprice-low-connection-example-01
982025-07-16 15:52:20.571[1060847.5]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}93b30c7d-d8fb-43ea-82e4-db31179fce1a[7984271, 14221794][]houseprice-low-connection-example-01
992025-07-16 15:52:20.571[239734.08]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}93b30c7d-d8fb-43ea-82e4-db31179fce1a[7984271, 14221794][]houseprice-low-connection-example-01

100 rows × 7 columns

The logs retrieved from the local Edge location, which is not in communication with the Wallaroo Model Ops Center instance.

# local logs

!curl -X POST http://testboy.lan:8081/logs \
    -H "Content-Type: Content-Type: application/json; format=pandas-records" \
    --data '{}' > ./edge-logs.df.json

df_logs = pd.read_json("./edge-logs.df.json", orient="records")
df_logs

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  500k  100  500k  100     2  7418k     29 --:--:-- --:--:-- --:--:-- 7474k
timeinoutanomalymetadata
01752681130155{'tensor': [4.0, 2.5, 2900.0, 5505.0, 2.0, 0.0, 0.0, 3.0, 8.0, 2900.0, 0.0, 47.6063, -122.02, 2970.0, 5251.0, 12.0, 0.0, 0.0]}{'variable': [718013.7]}{'count': 0}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '93b30c7d-d8fb-43ea-82e4-db31179fce1a', 'elapsed': [305045, 818254], 'dropped': [], 'partition': 'houseprice-low-connection-example-01'}
11752681140571{'tensor': [4.0, 2.75, 3010.0, 7215.0, 2.0, 0.0, 0.0, 3.0, 9.0, 3010.0, 0.0, 47.6952018738, -122.1780014038, 3010.0, 7215.0, 0.0, 0.0, 0.0]}{'variable': [795841.06]}{'count': 0}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '93b30c7d-d8fb-43ea-82e4-db31179fce1a', 'elapsed': [7984271, 14221794], 'dropped': [], 'partition': 'houseprice-low-connection-example-01'}
21752681140571{'tensor': [4.0, 1.75, 1400.0, 7920.0, 1.0, 0.0, 0.0, 3.0, 7.0, 1400.0, 0.0, 47.465801239, -122.1839981079, 1910.0, 7700.0, 52.0, 0.0, 0.0]}{'variable': [267013.97]}{'count': 0}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '93b30c7d-d8fb-43ea-82e4-db31179fce1a', 'elapsed': [7984271, 14221794], 'dropped': [], 'partition': 'houseprice-low-connection-example-01'}
31752681140571{'tensor': [4.0, 2.5, 3130.0, 13202.0, 2.0, 0.0, 0.0, 3.0, 10.0, 3130.0, 0.0, 47.5877990723, -121.9759979248, 2840.0, 10470.0, 19.0, 0.0, 0.0]}{'variable': [879083.56]}{'count': 0}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '93b30c7d-d8fb-43ea-82e4-db31179fce1a', 'elapsed': [7984271, 14221794], 'dropped': [], 'partition': 'houseprice-low-connection-example-01'}
41752681140571{'tensor': [3.0, 2.25, 1620.0, 997.0, 2.5, 0.0, 0.0, 3.0, 8.0, 1540.0, 80.0, 47.5400009155, -122.0260009766, 1620.0, 1068.0, 4.0, 0.0, 0.0]}{'variable': [544392.06]}{'count': 0}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '93b30c7d-d8fb-43ea-82e4-db31179fce1a', 'elapsed': [7984271, 14221794], 'dropped': [], 'partition': 'houseprice-low-connection-example-01'}
..................
9951752681140571{'tensor': [4.0, 2.5, 2040.0, 9225.0, 1.0, 0.0, 0.0, 5.0, 8.0, 1610.0, 430.0, 47.6360015869, -122.0970001221, 1730.0, 9225.0, 46.0, 0.0, 0.0]}{'variable': [627853.3]}{'count': 0}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '93b30c7d-d8fb-43ea-82e4-db31179fce1a', 'elapsed': [7984271, 14221794], 'dropped': [], 'partition': 'houseprice-low-connection-example-01'}
9961752681140571{'tensor': [3.0, 3.0, 1330.0, 1379.0, 2.0, 0.0, 0.0, 4.0, 8.0, 1120.0, 210.0, 47.6125984192, -122.31300354, 1810.0, 1770.0, 9.0, 0.0, 0.0]}{'variable': [450867.7]}{'count': 0}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '93b30c7d-d8fb-43ea-82e4-db31179fce1a', 'elapsed': [7984271, 14221794], 'dropped': [], 'partition': 'houseprice-low-connection-example-01'}
9971752681140571{'tensor': [3.0, 2.5, 1880.0, 4499.0, 2.0, 0.0, 0.0, 3.0, 8.0, 1880.0, 0.0, 47.5663986206, -121.9990005493, 2130.0, 5114.0, 22.0, 0.0, 0.0]}{'variable': [553463.25]}{'count': 0}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '93b30c7d-d8fb-43ea-82e4-db31179fce1a', 'elapsed': [7984271, 14221794], 'dropped': [], 'partition': 'houseprice-low-connection-example-01'}
9981752681140571{'tensor': [4.0, 1.5, 1200.0, 10890.0, 1.0, 0.0, 0.0, 5.0, 7.0, 1200.0, 0.0, 47.342300415, -122.0879974365, 1250.0, 10139.0, 42.0, 0.0, 0.0]}{'variable': [241330.17]}{'count': 0}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '93b30c7d-d8fb-43ea-82e4-db31179fce1a', 'elapsed': [7984271, 14221794], 'dropped': [], 'partition': 'houseprice-low-connection-example-01'}
9991752681140571{'tensor': [4.0, 3.25, 5180.0, 19850.0, 2.0, 0.0, 3.0, 3.0, 12.0, 3540.0, 1640.0, 47.5620002747, -122.1620025635, 3160.0, 9750.0, 9.0, 0.0, 0.0]}{'variable': [1295531.8]}{'count': 0}{'last_model': '{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}', 'pipeline_version': '93b30c7d-d8fb-43ea-82e4-db31179fce1a', 'elapsed': [7984271, 14221794], 'dropped': [], 'partition': 'houseprice-low-connection-example-01'}

1000 rows × 5 columns

We’ll now reenable communication, then wait 60 seconds and check the pipeline inference logs to see if communication is restored.

import time
inference_start = datetime.datetime.now(datetime.timezone.utc)
time.sleep(60)
import json

df = pd.DataFrame.from_records({"tensor": [[4.0, 
                                            2.5, 
                                            2900.0, 
                                            5505.0, 
                                            2.0, 
                                            0.0, 
                                            0.0, 
                                            3.0, 
                                            8.0, 
                                            2900.0, 
                                            0.0, 
                                            47.6063, 
                                            -122.02, 
                                            2970.0, 
                                            5251.0, 
                                            12.0, 
                                            0.0, 
                                            0.0]]})

data = df.to_dict(orient="records")

!curl -X POST testboy.lan:8081/infer \
    -H "Content-Type: Content-Type: application/json; format=pandas-records" \
    --data @./data/biginputs.df.json > inferenceresultsreconnect.df.json

# get the dataframe from what we just did

df_result = pd.read_json('./inferenceoutput.df.json', orient="records")
df_result.loc[:, ['time', 'out', 'metadata']]

# an extra 10 seconds to make sure logs are uploaded
time.sleep(10)
inference_end = datetime.datetime.now(datetime.timezone.utc)

# pipeline.logs(start_datetime=inference_start, end_datetime=inference_end)
 
pipeline.logs(dataset=['time', 'out', 'metadata'])

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  5768  100  4621  100  1147   144k  36784 --:--:-- --:--:-- --:--:--  181k

Warning: There are more logs available. Please set a larger limit or request a file using export_logs.
timeout.variablemetadata.last_modelmetadata.pipeline_versionmetadata.elapsedmetadata.droppedmetadata.partition
02025-07-16 15:57:54.572[2016006.1]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}93b30c7d-d8fb-43ea-82e4-db31179fce1a[131009, 502531][]houseprice-low-connection-example-01
12025-07-16 15:57:54.572[1910824.0]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}93b30c7d-d8fb-43ea-82e4-db31179fce1a[131009, 502531][]houseprice-low-connection-example-01
22025-07-16 15:57:54.572[2005883.1]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}93b30c7d-d8fb-43ea-82e4-db31179fce1a[131009, 502531][]houseprice-low-connection-example-01
32025-07-16 15:57:54.572[1946437.8]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}93b30c7d-d8fb-43ea-82e4-db31179fce1a[131009, 502531][]houseprice-low-connection-example-01
42025-07-16 15:57:54.572[1689843.1]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}93b30c7d-d8fb-43ea-82e4-db31179fce1a[131009, 502531][]houseprice-low-connection-example-01
........................
952025-07-16 15:52:20.571[323856.28]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}93b30c7d-d8fb-43ea-82e4-db31179fce1a[7984271, 14221794][]houseprice-low-connection-example-01
962025-07-16 15:52:20.571[536388.2]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}93b30c7d-d8fb-43ea-82e4-db31179fce1a[7984271, 14221794][]houseprice-low-connection-example-01
972025-07-16 15:52:20.571[502022.94]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}93b30c7d-d8fb-43ea-82e4-db31179fce1a[7984271, 14221794][]houseprice-low-connection-example-01
982025-07-16 15:52:20.571[544392.06]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}93b30c7d-d8fb-43ea-82e4-db31179fce1a[7984271, 14221794][]houseprice-low-connection-example-01
992025-07-16 15:52:20.571[435628.72]{"model_name":"rf-house-price-estimator","model_sha":"e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6"}93b30c7d-d8fb-43ea-82e4-db31179fce1a[7984271, 14221794][]houseprice-low-connection-example-01

100 rows × 7 columns