.

.

Wallaroo MLOps API Essentials Guide

Basic Guide for the Wallaroo API

Retrieve Credentials

Through Keycloak

Wallaroo comes pre-installed with a confidential OpenID Connect client. The default client is api-client, but other clients may be created and configured.

Confidential clients require its secret to be supplied when requesting a token. Administrators may obtain their API client credentials from Keycloak from the Keycloak Service URL as listed above and the prefix /auth/admin/master/console/#/realms/master/clients.

For example, if the Wallaroo DNS address is in the format https://{WALLAROO PREFIX.}{WALLAROO SUFFIX}, then the direct path to the Keycloak API client credentials would be:

https://{WALLAROO PREFIX.}keycloak.{WALLAROO SUFFIX}/auth/admin/master/console/#/realms/master/clients

If the there is no prefix, then the address would simply be:

https://keycloak.{WALLAROO SUFFIX}/auth/admin/master/console/#/realms/master/clients

Then select the client, in this case api-client, then Credentials.

By default, tokens issued for api-client are valid for up to 60 minutes. Refresh tokens are supported.

Token Types

There are two tokens used with Wallaroo API services:

  • MLOps tokens: User tokens are generated with the confidential client credentials and the username/password of the Wallaroo user making the MLOps API request and requires:

    • The Wallaroo instance Keycloak address.

    • The confidential client, api-client by default.

    • The confidential client secret.

    • The Wallaroo username making the MLOps API request.

    • The Wallaroo user’s password.

      This request return includes the access_token and the refresh_token. The access_token is used to authenticate. The refresh_token can be used to create a new token without submitting the original username and password.

      A sample curl version of that request is:

      eval $(curl "https://${URL_PREFIX}keycloak.${URL_SUFFIX}/auth/realms/master/protocol/openid-connect/token" -u "${CONFIDENTIAL_CLIENT}:${CONFIDENTIAL_CLIENT_SECRET}" -d "grant_type=password&username=${USERNAME}&password=${PASSWORD}&scope=offline_access' -s | jq -r '"TOKEN=\(.access_token) REFRESH=\(.refresh_token)"')
      
      • Tokens can be refreshed via a refresh request and require:
        • The confidential client, api-client by default.

        • The confidential client secret.

        • The refresh token retrieved from the initial access token request. A curl version of that request is:

          TOKEN=$(curl "https://${URL_PREFIX}keycloak.${URL_SUFFIX}/auth/realms/master/protocol/openid-connect/token" -u "${CONFIDENTIAL_CLIENT}:${CONFIDENTIAL_CLIENT_SECRET}" -d "grant_type=refresh_token&refresh_token=${REFRESH}" -s | jq -r '.access_token')
          
  • Inference Token: Tokens used as part of a Pipeline Inference URL request. These do not require a Wallaroo user credentials. Inference token request require the following:

    • The Wallaroo instance Keycloak address.

    • The confidential client, api-client by default.

    • The confidential client secret.

      A curl version of that command is:

      TOKEN=$(curl "https://${URL_PREFIX}keycloak.${URL_SUFFIX}/auth/realms/master/protocol/openid-connect/token" -u "${CONFIDENTIAL_CLIENT}:${CONFIDENTIAL_CLIENT_SECRET}" -d 'grant_type=client_credentials' -s | jq -r '.access_token')
      

The following examples demonstrate:

  • Generating a MLOps API token with the confidential client, client secret, username, and password.
  • Refreshing a MLOps API token with the confidential client and client secret (the username and password are not required for refreshing the token).
  • Generate a Pipeline Inference URl token with the confidential client and client secret (username and password are not required).

The username and password for the user are stored in the file ./creds.json to prevent them from being displayed in a demonstration.

## Generating token with confidential client, client secret, username, password

TOKENURL=f'https://{wallarooPrefix}keycloak.{wallarooSuffix}/auth/realms/master/protocol/openid-connect/token'

USERNAME = login_data["username"]
PASSWORD = login_data["password"]
CONFIDENTIAL_CLIENT=login_data["confidentialClient"]
CONFIDENTIAL_CLIENT_SECRET=login_data["confidentialPassword"]

auth = HTTPBasicAuth(CONFIDENTIAL_CLIENT, CONFIDENTIAL_CLIENT_SECRET)
data = {
    'grant_type': 'password',
    'username': USERNAME,
    'password': PASSWORD
}
response = requests.post(TOKENURL, auth=auth, data=data, verify=True)
access_token = response.json()['access_token']
refresh_token = response.json()['refresh_token']
display(access_token)
'eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJDYkFqN19QY0xCWTFkWmJiUDZ6Q3BsbkNBYTd6US0tRHlyNy0yLXlQb25nIn0.eyJleHAiOjE2ODQzNjAxNjUsImlhdCI6MTY4NDM1NjU2NSwianRpIjoiZGQxMDFkODMtMzk5ZC00N2M2LThlZDMtNjQxMGRmNThhYmViIiwiaXNzIjoiaHR0cHM6Ly9kb2MtdGVzdC5rZXljbG9hay53YWxsYXJvb2NvbW11bml0eS5uaW5qYS9hdXRoL3JlYWxtcy9tYXN0ZXIiLCJhdWQiOlsibWFzdGVyLXJlYWxtIiwiYWNjb3VudCJdLCJzdWIiOiIwMjhjOGI0OC1jMzliLTQ1NzgtOTExMC0wYjViZGQzODI0ZGEiLCJ0eXAiOiJCZWFyZXIiLCJhenAiOiJhcGktY2xpZW50Iiwic2Vzc2lvbl9zdGF0ZSI6Ijk4MDhkZTA5LWU2NjYtNGIyNC05ZWQ4LTc2MmUxZjllODk0ZSIsImFjciI6IjEiLCJhbGxvd2VkLW9yaWdpbnMiOlsiKiJdLCJyZWFsbV9hY2Nlc3MiOnsicm9sZXMiOlsiZGVmYXVsdC1yb2xlcy1tYXN0ZXIiLCJvZmZsaW5lX2FjY2VzcyIsInVtYV9hdXRob3JpemF0aW9uIl19LCJyZXNvdXJjZV9hY2Nlc3MiOnsibWFzdGVyLXJlYWxtIjp7InJvbGVzIjpbIm1hbmFnZS11c2VycyIsInZpZXctdXNlcnMiLCJxdWVyeS1ncm91cHMiLCJxdWVyeS11c2VycyJdfSwiYWNjb3VudCI6eyJyb2xlcyI6WyJtYW5hZ2UtYWNjb3VudCIsIm1hbmFnZS1hY2NvdW50LWxpbmtzIiwidmlldy1wcm9maWxlIl19fSwic2NvcGUiOiJwcm9maWxlIGVtYWlsIiwic2lkIjoiOTgwOGRlMDktZTY2Ni00YjI0LTllZDgtNzYyZTFmOWU4OTRlIiwiZW1haWxfdmVyaWZpZWQiOmZhbHNlLCJodHRwczovL2hhc3VyYS5pby9qd3QvY2xhaW1zIjp7IngtaGFzdXJhLXVzZXItaWQiOiIwMjhjOGI0OC1jMzliLTQ1NzgtOTExMC0wYjViZGQzODI0ZGEiLCJ4LWhhc3VyYS1kZWZhdWx0LXJvbGUiOiJ1c2VyIiwieC1oYXN1cmEtYWxsb3dlZC1yb2xlcyI6WyJ1c2VyIl0sIngtaGFzdXJhLXVzZXItZ3JvdXBzIjoie30ifSwibmFtZSI6IkpvaG4gSGFuc2FyaWNrIiwicHJlZmVycmVkX3VzZXJuYW1lIjoiam9obi5odW1tZWxAd2FsbGFyb28uYWkiLCJnaXZlbl9uYW1lIjoiSm9obiIsImZhbWlseV9uYW1lIjoiSGFuc2FyaWNrIiwiZW1haWwiOiJqb2huLmh1bW1lbEB3YWxsYXJvby5haSJ9.lfesLVvWM21kbWIhQtT2ap-ruT5_qt7CVcPUt1mAS8KoksuiJIb4QxPV9FwGB1I7sPiWjXR60cR-cjNLLoTgCX9GZZbfISDR4NvqN5ZBANDzYCx64WtTZCaDPeWROClHvmmE6Mfs1mAdgC3fIxkDe6Ns5-S6wnDqW7v6-yaNo5gBywftaCFyD3lFsmpmBvcyXphtn7sUlX_W4Ku9xmaalUkLv1F8528thZAARN5Jl-_uTHNKCe5wYGiEpQkbeIZ_Rjzqnctx-onw3cVKgbS6_wATr0TZQxgR2AY459OkCJ3rcuJTTTI5PihEGKlQUX5GmDIGG8DqE3iAPJ-xCY-OBQ'
## Refresh the token

TOKENURL=f'https://{wallarooPrefix}keycloak.{wallarooSuffix}/auth/realms/master/protocol/openid-connect/token'

# Retrieving through os environmental variables 
f = open('./creds.json')
login_data = json.load(f)

CONFIDENTIAL_CLIENT=login_data["confidentialClient"]
CONFIDENTIAL_CLIENT_SECRET=login_data["confidentialPassword"]

auth = HTTPBasicAuth(CONFIDENTIAL_CLIENT, CONFIDENTIAL_CLIENT_SECRET)
data = {
    'grant_type': 'refresh_token',
    'refresh_token': refresh_token
}
response = requests.post(TOKENURL, auth=auth, data=data, verify=True)
access_token = response.json()['access_token']
refresh_token = response.json()['refresh_token']
display(access_token)
'eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJDYkFqN19QY0xCWTFkWmJiUDZ6Q3BsbkNBYTd6US0tRHlyNy0yLXlQb25nIn0.eyJleHAiOjE2ODQzNjAxNjcsImlhdCI6MTY4NDM1NjU2NywianRpIjoiZDJlNTNlNzEtYjYzMi00MzNmLThjY2UtOGIxMDI0ZjFmYzliIiwiaXNzIjoiaHR0cHM6Ly9kb2MtdGVzdC5rZXljbG9hay53YWxsYXJvb2NvbW11bml0eS5uaW5qYS9hdXRoL3JlYWxtcy9tYXN0ZXIiLCJhdWQiOlsibWFzdGVyLXJlYWxtIiwiYWNjb3VudCJdLCJzdWIiOiIwMjhjOGI0OC1jMzliLTQ1NzgtOTExMC0wYjViZGQzODI0ZGEiLCJ0eXAiOiJCZWFyZXIiLCJhenAiOiJhcGktY2xpZW50Iiwic2Vzc2lvbl9zdGF0ZSI6Ijk4MDhkZTA5LWU2NjYtNGIyNC05ZWQ4LTc2MmUxZjllODk0ZSIsImFjciI6IjEiLCJhbGxvd2VkLW9yaWdpbnMiOlsiKiJdLCJyZWFsbV9hY2Nlc3MiOnsicm9sZXMiOlsiZGVmYXVsdC1yb2xlcy1tYXN0ZXIiLCJvZmZsaW5lX2FjY2VzcyIsInVtYV9hdXRob3JpemF0aW9uIl19LCJyZXNvdXJjZV9hY2Nlc3MiOnsibWFzdGVyLXJlYWxtIjp7InJvbGVzIjpbIm1hbmFnZS11c2VycyIsInZpZXctdXNlcnMiLCJxdWVyeS1ncm91cHMiLCJxdWVyeS11c2VycyJdfSwiYWNjb3VudCI6eyJyb2xlcyI6WyJtYW5hZ2UtYWNjb3VudCIsIm1hbmFnZS1hY2NvdW50LWxpbmtzIiwidmlldy1wcm9maWxlIl19fSwic2NvcGUiOiJwcm9maWxlIGVtYWlsIiwic2lkIjoiOTgwOGRlMDktZTY2Ni00YjI0LTllZDgtNzYyZTFmOWU4OTRlIiwiZW1haWxfdmVyaWZpZWQiOmZhbHNlLCJodHRwczovL2hhc3VyYS5pby9qd3QvY2xhaW1zIjp7IngtaGFzdXJhLXVzZXItaWQiOiIwMjhjOGI0OC1jMzliLTQ1NzgtOTExMC0wYjViZGQzODI0ZGEiLCJ4LWhhc3VyYS1kZWZhdWx0LXJvbGUiOiJ1c2VyIiwieC1oYXN1cmEtYWxsb3dlZC1yb2xlcyI6WyJ1c2VyIl0sIngtaGFzdXJhLXVzZXItZ3JvdXBzIjoie30ifSwibmFtZSI6IkpvaG4gSGFuc2FyaWNrIiwicHJlZmVycmVkX3VzZXJuYW1lIjoiam9obi5odW1tZWxAd2FsbGFyb28uYWkiLCJnaXZlbl9uYW1lIjoiSm9obiIsImZhbWlseV9uYW1lIjoiSGFuc2FyaWNrIiwiZW1haWwiOiJqb2huLmh1bW1lbEB3YWxsYXJvby5haSJ9.Va5OiLkedj-mFuI7UhsxXshmaSbhXthLv-PU56f2JiQi5-wiWXXxRk3pUioavIzKbi-VjmYQfbR95VY5QSWpUuD3scPuRhHDkSeslz6390phYiFygK_PmXMviQnL2q1mwdGzwh69htOjUWLf7MGWjNmkNdzjYyBy8gfD3V2O2MCfN3onVVCqr1aA1aAQXe9y_JswhjooxAQGit1xzNicvm3IW3QhHtOrDKj7gXNuSlc5vKqe52RQYEgElltqIOV4PVe12UGthKMfvdlDIeUEpTzXVFRH8XHJCrO_YQ_W9m-Rt1_9kelBl3SksdYKOisZaGwo6lv7hhapembH0iD29Q'
## Pipeline Inference URL token - does not require Wallaroo username/password.

TOKENURL=f'https://{wallarooPrefix}keycloak.{wallarooSuffix}/auth/realms/master/protocol/openid-connect/token'

# Retrieving through os environmental variables 
f = open('./creds.json')
login_data = json.load(f)

CONFIDENTIAL_CLIENT=login_data["confidentialClient"]
CLIENT_SECRET=login_data["confidentialPassword"]

auth = HTTPBasicAuth(CONFIDENTIAL_CLIENT, CLIENT_SECRET)
data = {
    'grant_type': 'client_credentials'
}
response = requests.post(TOKENURL, auth=auth, data=data, verify=True)
inference_access_token = response.json()['access_token']
display(inference_access_token)
'eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJDYkFqN19QY0xCWTFkWmJiUDZ6Q3BsbkNBYTd6US0tRHlyNy0yLXlQb25nIn0.eyJleHAiOjE2ODQzNjAxNjgsImlhdCI6MTY4NDM1NjU2OCwianRpIjoiNjIyOTViYmUtYzVlMi00NDQ2LWFmMDctNzY5MDAwNmI2NzI3IiwiaXNzIjoiaHR0cHM6Ly9kb2MtdGVzdC5rZXljbG9hay53YWxsYXJvb2NvbW11bml0eS5uaW5qYS9hdXRoL3JlYWxtcy9tYXN0ZXIiLCJhdWQiOlsibWFzdGVyLXJlYWxtIiwiYWNjb3VudCJdLCJzdWIiOiJmNmI5ODg5NC1iZTVjLTQyZDUtYTZhNS02ZjE5ZTY1YmNiNGEiLCJ0eXAiOiJCZWFyZXIiLCJhenAiOiJhcGktY2xpZW50IiwiYWNyIjoiMSIsImFsbG93ZWQtb3JpZ2lucyI6WyIqIl0sInJlYWxtX2FjY2VzcyI6eyJyb2xlcyI6WyJkZWZhdWx0LXJvbGVzLW1hc3RlciIsIm9mZmxpbmVfYWNjZXNzIiwidW1hX2F1dGhvcml6YXRpb24iXX0sInJlc291cmNlX2FjY2VzcyI6eyJtYXN0ZXItcmVhbG0iOnsicm9sZXMiOlsiaW1wZXJzb25hdGlvbiIsIm1hbmFnZS11c2VycyIsInZpZXctdXNlcnMiLCJxdWVyeS1ncm91cHMiLCJxdWVyeS11c2VycyJdfSwiYWNjb3VudCI6eyJyb2xlcyI6WyJtYW5hZ2UtYWNjb3VudCIsIm1hbmFnZS1hY2NvdW50LWxpbmtzIiwidmlldy1wcm9maWxlIl19fSwic2NvcGUiOiJwcm9maWxlIGVtYWlsIiwiZW1haWxfdmVyaWZpZWQiOmZhbHNlLCJjbGllbnRJZCI6ImFwaS1jbGllbnQiLCJjbGllbnRIb3N0IjoiMTAuMjQ0LjEuNzQiLCJodHRwczovL2hhc3VyYS5pby9qd3QvY2xhaW1zIjp7IngtaGFzdXJhLXVzZXItaWQiOiJmNmI5ODg5NC1iZTVjLTQyZDUtYTZhNS02ZjE5ZTY1YmNiNGEiLCJ4LWhhc3VyYS1kZWZhdWx0LXJvbGUiOiJ1c2VyIiwieC1oYXN1cmEtYWxsb3dlZC1yb2xlcyI6WyJ1c2VyIl0sIngtaGFzdXJhLXVzZXItZ3JvdXBzIjoie30ifSwicHJlZmVycmVkX3VzZXJuYW1lIjoic2VydmljZS1hY2NvdW50LWFwaS1jbGllbnQiLCJjbGllbnRBZGRyZXNzIjoiMTAuMjQ0LjEuNzQifQ.fde1-NsmXqCen71sRcIarscK1j4oFGATf8jh834aAUSb_UGXEmxEnUDDGMegu7KmpbeOi2ogIGY0ndACaZqS21lvVpzWHyVdsQGXCtl1mjwgLt0kzq6U5uR8znMIV-2Babw-9eE65F9I3TdUKRlnh8J5SAPvbOj_Hv_Y3u4cNj1b_Hk_o9lAEg-m2V0ZL7UDxgnVyitbWChiP4DE3q6yBBSVoORiBXDrfUiwIpCXyVKJIO_HrowEA8bYVOhh8PcbywmVa1kZaPcMuAOzsaysE361NCvJqbikVf4KX5Ii9k7lk90v3c-9VX24bIC67HFG8TwvWVnKRBAawwXcQ2ZTIA'

Through the Wallaroo SDK

The Wallaroo SDK method Wallaroo Client wl.auth.auth_header() method provides the token with the Authorization header.

# Retrieve the token
headers = wl.auth.auth_header()
display(headers)

{'Authorization': 'Bearer abcdefg'}

Connect to Wallaroo

For this example, a connection to the Wallaroo SDK is used. This will be used to retrieve the JWT token for the MLOps API calls.

This example will store the user’s credentials into the file ./creds.json which contains the following:

{
    "username": "{Connecting User's Username}", 
    "password": "{Connecting User's Password}", 
    "email": "{Connecting User's Email Address}"
}

Replace the username, password, and email fields with the user account connecting to the Wallaroo instance. This allows a seamless connection to the Wallaroo instance and bypasses the standard browser based confirmation link. For more information, see the Wallaroo SDK Essentials Guide: Client Connection.

For wallarooPrefix = "YOUR PREFIX." and wallarooSuffix = "YOUR SUFFIX", enter the prefix and suffix for your Wallaroo instance DNS name. If the prefix instance is blank, then it can be wallarooPrefix = "". Note that the prefix includes the . for proper formatting.

# Retrieve the login credentials.
os.environ["WALLAROO_SDK_CREDENTIALS"] = './creds.json'

# Client connection from local Wallaroo instance

wallarooPrefix = "YOUR PREFIX."
wallarooSuffix = "YOUR SUFFIX"

wl = wallaroo.Client(api_endpoint=f"https://{wallarooPrefix}api.{wallarooSuffix}", 
                     auth_endpoint=f"https://{wallarooPrefix}keycloak.{wallarooSuffix}", 
                     auth_type="user_password")

API URL

The variable APIURL is used to specify the connection to the Wallaroo instance’s MLOps API URL.

APIURL=f"https://{wallarooPrefix}api.{wallarooSuffix}"

API Request Methods

This tutorial relies on the Python requests library, and the Wallaroo Wallaroo Client wl.auth.auth_header() method.

MLOps API requests are always POST. Most are submitted with the header 'Content-Type':'application/json' unless specified otherwise.

1 - Wallaroo MLOps API Essentials Guide: User Management

How to use the Wallaroo API for User Management

Users

User management provides the ability to create new users, activate/deactivate them and other tasks.

Example Notes

For these examples, we will rely on the wallaroo SDK and requests library for making connections to our sample Wallaroo Ops instance. Examples are provided for both the Python requests library and curl.

import wallaroo

import requests

import json

The Wallaroo SDK provides the API endpoint through the wallaroo.client.api_endpoint variable. This is derived from the Wallaroo OPs DNS settings.

The method wallaroo.client.auth.auth_header() retrieves the HTTP authorization headers for the API connection.

Both of these are used to authenticate for the Wallaroo MLOps API calls used in the future examples. For these examples, the Wallaroo Client is stored in the variable wl.

User Endpoints

Get Users

  • Endpoint: /v1/api/users/query

Users are retrieved either by their Keycloak user id, or return all users if an empty set {} is submitted.

Get Users Parameters

FieldTypeDescription
user_idsList[Keycloak user ids] (Optional)An array of Keycloak user ids, typically in UUID format.

If an empty set {} is submitted as a parameter, then all users are returned.

Get Users Returns

Full details are available from the Keycloak UserRepresentation site. The following represents the most relevant values.

Field  TypeDescription
usersList[user]A list of users and their information with the Keycloak ID as the primary key.
 {keycloak id}userUser details.
  createdTimeTampIntegerThe Unix Epoc Timestamp of when the user was created.
  emailStringThe user’s email address.
  enabledBooleanWhether the user is verified or not.
  firstNameStringThe user’s first name.
  lastNameStringThe user’s last name.
  idStringThe user’s keycloak id in UUID format.
  usernameStringThe user’s username as an email address.

Get Users Examples

The first example will submit an empty set {} to return all users, then submit the first user’s user id and request only that user’s details.

Get all users via Requests.


# Retrieve the token 
headers = wl.auth.auth_header()

endpoint = f"{wl.api_endpoint}/v1/api/users/query"
data = {
}

response = requests.post(endpoint, 
                         json=data, 
                         headers=headers, 
                         verify=True).json()
response
{'users': {'12ea09d1-0f49-405e-bed1-27eb6d10fde4': {'access': {'view': True,
        'manage': True,
        'manageGroupMembership': True,
        'mapRoles': True,
        'impersonate': False},
       'createdTimestamp': 1700496282637,
       'disableableCredentialTypes': [],
       'email': 'john.hummel@wallaroo.ai',
       'emailVerified': False,
       'enabled': True,
       'firstName': 'John',
       'id': '12ea09d1-0f49-405e-bed1-27eb6d10fde4',
       'lastName': 'Hansarick',
       'notBefore': 0,
       'requiredActions': [],
       'username': 'john.hummel@wallaroo.ai'},
      '57d61aed-3058-4327-9e65-a5d39a9718c0': {'access': {'view': True,
        'manage': True,
        'impersonate': False,
        'manageGroupMembership': True,
        'mapRoles': True},
       'createdTimestamp': 1701202186276,
       'disableableCredentialTypes': [],
       'email': 'john.hansarick@wallaroo.ai',
       'emailVerified': True,
       'enabled': True,
       'firstName': 'John',
       'id': '57d61aed-3058-4327-9e65-a5d39a9718c0',
       'lastName': 'Hansarick',
       'notBefore': 0,
       'requiredActions': [],
       'username': 'john.hansarick@wallaroo.ai'},
      '1bc88c6d-b476-4e4d-aa1a-a5a3554591d3': {'access': {'view': True,
        'mapRoles': True,
        'manage': True,
        'manageGroupMembership': True,
        'impersonate': False},
       'createdTimestamp': 1700495081278,
       'disableableCredentialTypes': [],
       'emailVerified': False,
       'enabled': True,
       'id': '1bc88c6d-b476-4e4d-aa1a-a5a3554591d3',
       'notBefore': 0,
       'requiredActions': [],
       'username': 'admin'}}}

Get All Users via curl


curl {wl.api_endpoint}/v1/api/users/query \
    -H "Authorization: {wl.auth.auth_header()['Authorization']}" \
    -H "Content-Type: application/json" \
    --data '{{}}'
{
  "users": {
    "57d61aed-3058-4327-9e65-a5d39a9718c0": {
      "access": {
        "manage": true,
        "impersonate": false,
        "view": true,
        "manageGroupMembership": true,
        "mapRoles": true
      },
      "createdTimestamp": 1701202186276,
      "disableableCredentialTypes": [],
      "email": "john.hansarick@wallaroo.ai",
      "emailVerified": true,
      "enabled": true,
      "firstName": "John",
      "id": "57d61aed-3058-4327-9e65-a5d39a9718c0",
      "lastName": "Hansarick",
      "notBefore": 0,
      "requiredActions": [],
      "username": "john.hansarick@wallaroo.ai"
    },
    "12ea09d1-0f49-405e-bed1-27eb6d10fde4": {
      "access": {
        "impersonate": false,
        "manageGroupMembership": true,
        "view": true,
        "manage": true,
        "mapRoles": true
      },
      "createdTimestamp": 1700496282637,
      "disableableCredentialTypes": [],
      "email": "john.hummel@wallaroo.ai",
      "emailVerified": false,
      "enabled": true,
      "firstName": "John",
      "id": "12ea09d1-0f49-405e-bed1-27eb6d10fde4",
      "lastName": "Hansarick",
      "notBefore": 0,
      "requiredActions": [],
      "username": "john.hummel@wallaroo.ai"
    },
    "1bc88c6d-b476-4e4d-aa1a-a5a3554591d3": {
      "access": {
        "manage": true,
        "manageGroupMembership": true,
        "mapRoles": true,
        "view": true,
        "impersonate": false
      },
      "createdTimestamp": 1700495081278,
      "disableableCredentialTypes": [],
      "emailVerified": false,
      "enabled": true,
      "id": "1bc88c6d-b476-4e4d-aa1a-a5a3554591d3",
      "notBefore": 0,
      "requiredActions": [],
      "username": "admin"
    }
  }
}

Get first user via Keycloak ID.


# Retrieve the token 
headers = wl.auth.auth_header()

endpoint = f"{wl.api_endpoint}/v1/api/users/query"

# retrieved from the previous request - get the 2nd user since the first will always be `admin`
first_user_keycloak = list(response['users'])[1]

data = {
  "user_ids": [
    first_user_keycloak
  ]
}

user_response = requests.post(endpoint, 
                         json=data, 
                         headers=headers, 
                         verify=True).json()
user_response
{
    'users': {
        '57d61aed-3058-4327-9e65-a5d39a9718c0': {
            'access': {
                'manageGroupMembership': True,
                'mapRoles': True,
                'manage': True,
                'impersonate': False,
                'view': True},
            'createdTimestamp': 1701202186276,
            'disableableCredentialTypes': [],
            'email': 'john.hansarick@wallaroo.ai',
            'emailVerified': True,
            'enabled': True,
            'federatedIdentities': [],
            'firstName': 'John',
            'id': '57d61aed-3058-4327-9e65-a5d39a9718c0',
            'lastName': 'Hansarick',
            'notBefore': 0,
            'requiredActions': [],
            'username': 'john.hansarick@wallaroo.ai'
        }
    }
}

Get first user via curl.


curl {wl.api_endpoint}/v1/api/users/query \
    -H "Authorization: {wl.auth.auth_header()['Authorization']}" \
    -H "Content-Type: application/json" \
    --data '{json.dumps(data)}'
{
  "users": {
    "57d61aed-3058-4327-9e65-a5d39a9718c0": {
      "access": {
        "manageGroupMembership": true,
        "mapRoles": true,
        "impersonate": false,
        "view": true,
        "manage": true
      },
      "createdTimestamp": 1701202186276,
      "disableableCredentialTypes": [],
      "email": "john.hansarick@wallaroo.ai",
      "emailVerified": true,
      "enabled": true,
      "federatedIdentities": [],
      "firstName": "John",
      "id": "57d61aed-3058-4327-9e65-a5d39a9718c0",
      "lastName": "Hansarick",
      "notBefore": 0,
      "requiredActions": [],
      "username": "john.hansarick@wallaroo.ai"
    }
  }
}

Invite Users

  • Endpoint: /v1/api/users/invite

IMPORTANT NOTE: This command is for Wallaroo Community only. For more details on user management, see Wallaroo User Management.

Users are invited through /users/invite. When using Wallaroo Community, this will send an invitation email to the email address listed. Note that the user must not already be a member of the Wallaroo instance, and email addresses must be unique. If the email address is already in use for another user, the request will generate an error.

Invite Users Parameters

FieldTypeDescription
emailString (Required)The email address of the new user to invite.
passwordString (Optional)The assigned password of the new user to invite. If not provided, the Wallaroo instance will provide the new user a temporary password that must be changed upon initial login.

Invite Users Returns

FieldTypeDescription
idStringThe email address of the new user to invite.
passwordStringThe assigned password of the new user.

Invite Users Examples

Example: In this example, a new user will be invited to the Wallaroo instance and assigned a password.

Invite users via Requests.

headers = wl.auth.auth_header()

endpoint = f"{wl.api_endpoint}/v1/api/users/invite"

data = {
    "email": "example.person@wallaroo.ai",
    "password":"Example-Password"
}

response = requests.post(endpoint, 
                         json=data, 
                         headers=headers, 
                         verify=True).json()
response

Invite users via curl.

curl {wl.api_endpoint}/v1/api/users/invite \
    -H "Authorization: {wl.auth.auth_header()['Authorization']}" \
    -H "Content-Type: application/json" \
    --data '{{"email": "example.person@wallaroo.ai","password":"Example-Password"}}'

Deactivate User

  • Endpoint: /v1/api/users/deactivate

Users can be deactivated so they can not login to their Wallaroo instance. Deactivated users do not count against the Wallaroo license count.

Deactivate User Parameters

FieldTypeDescription
emailString (Required)The email address user to deactivate.

Deactivate User Returns

{} on a successful request.

Deactivate User Examples

Example: In this example, a user will be deactivated.

Deactivate user via Requests.

# Retrieve the token 
headers = wl.auth.auth_header()

endpoint = f"{wl.api_endpoint}/v1/api/users/deactivate"

data = {
    "email": "john.hansarick@wallaroo.ai"
}

response = requests.post(endpoint, 
                         json=data, 
                         headers=headers, 
                         verify=True).json()
response
{}

Deactivate user via curl.

curl {wl.api_endpoint}/v1/api/users/deactivate \
    -H "Authorization: {wl.auth.auth_header()['Authorization']}" \
    -H "Content-Type: application/json" \
    --data '{{"email": "john.hansarick@wallaroo.ai"}}'
{}

Activate User

  • Endpoint: /v1/api/users/activate

A deactivated user can be reactivated to allow them access to their Wallaroo instance. Activated users count against the Wallaroo license count.

Activate User Parameters

FieldTypeDescription
emailString (Required)The email address user to activate.

Activate User Returns

{} on a successful request.

Activate User Examples

In this example, the user john.hansarick@wallaroo.ai will be activated.

Activate user via Requests.

# Retrieve the token 
headers = wl.auth.auth_header()
endpoint = f"{wl.api_endpoint}/v1/api/users/activate"

data = {
    "email": "john.hansarick@wallaroo.ai"
}

response = requests.post(endpoint, 
                         json=data, 
                         headers=headers, 
                         verify=True).json()
response
{}

Activate user via curl.

curl {wl.api_endpoint}/v1/api/users/activate \
    -H "Authorization: {wl.auth.auth_header()['Authorization']}" \
    -H "Content-Type: application/json" \
    --data '{{"email": "john.hansarick@wallaroo.ai"}}'
{}

Troubleshooting

When a new user logs in for the first time, they get an error when uploading a model or issues when they attempt to log in. How do I correct that?

When a new registered user attempts to upload a model, they may see the following error:

TransportQueryError: 
{'extensions': 
    {'path': 
        '$.selectionSet.insert_workspace_one.args.object[0]', 'code': 'not-supported'
    }, 
    'message': 
        'cannot proceed to insert array relations since insert to table "workspace" affects zero rows'

Or if they log into the Wallaroo Dashboard, they may see a Page not found error.

This is caused when a user has been registered without an appropriate email address. See the user guides here on inviting a user, or the Wallaroo Enterprise User Management on how to log into the Keycloak service and update users. Verify that the username and email address are both the same, and they are valid confirmed email addresses for the user.

2 - Wallaroo MLOps API Essentials Guide: Workspace Management

How to use the Wallaroo API for Workspace Management

Workspace Naming Requirements

Workspace names map onto Kubernetes objects, and must be DNS compliant. Workspace names must be ASCII alpha-numeric characters or dash (-) only. . and _ are not allowed.

Workspaces

Workspaces are used to segment groups of models and pipelines into separate environments. This allows different users to either manage or have access to each workspace, controlling the models and pipelines assigned to the workspace.

List User Workspaces

  • Endpoint: /v1/api/workspaces/list

List the workspaces for specified users.

List User Workspaces Parameters

FieldTypeDescription
user_idsList[Keycloak user ids] (Optional)An array of Keycloak user ids, typically in UUID format.

If an empty set {} is submitted as a parameter, then the workspaces for users are returned.

List User Workspaces Returns

Field TypeDescription
workspaces List[workspaces]A List of workspaces for the specified users.
 idIntegerThe numerical ID of the workspace.
 nameStringThe assigned name of the workspace.
 create_atStringThe DateTime the workspace was created.
 create_byStringThe Keycloak ID of the user who created the workspace.
 archivedBooleanWhether the workspace is archived or not.
 modelsList[Integer]The model ids uploaded to the workspace.
 pipelinesList[Integer]The pipeline ids built within the workspace.

List User Workspaces Examples

In these example, the workspaces for all users will be displayed.

List all workspaces via Requests.

# Retrieve the token 
headers = wl.auth.auth_header()

endpoint = f"{wl.api_endpoint}/v1/api/workspaces/list"

data = {
}

response = requests.post(endpoint, 
                         json=data, 
                         headers=headers, 
                         verify=True).json()
response
    {'workspaces': [{'id': 1,
       'name': 'john.hummel@wallaroo.ai - Default Workspace',
       'created_at': '2023-11-20T16:05:06.323911+00:00',
       'created_by': '12ea09d1-0f49-405e-bed1-27eb6d10fde4',
       'archived': False,
       'models': [],
       'pipelines': []},
      {'id': 5,
       'name': 'mobilenetworkspacetest',
       'created_at': '2023-11-20T16:05:48.271364+00:00',
       'created_by': '12ea09d1-0f49-405e-bed1-27eb6d10fde4',
       'archived': False,
       'models': [1, 2],
       'pipelines': [1]},
      {'id': 6,
       'name': 'edge-observability-assaysbaseline-examples',
       'created_at': '2023-11-20T16:09:31.950532+00:00',
       'created_by': '12ea09d1-0f49-405e-bed1-27eb6d10fde4',
       'archived': False,
       'models': [3],
       'pipelines': [4]},
      {'id': 7,
       'name': 'edge-observability-houseprice-demo',
       'created_at': '2023-11-20T17:36:07.131292+00:00',
       'created_by': '12ea09d1-0f49-405e-bed1-27eb6d10fde4',
       'archived': False,
       'models': [4, 5, 6, 7, 8, 9, 12],
       'pipelines': [7, 14, 16]},
      {'id': 8,
       'name': 'clip-demo',
       'created_at': '2023-11-20T18:57:53.667873+00:00',
       'created_by': '12ea09d1-0f49-405e-bed1-27eb6d10fde4',
       'archived': False,
       'models': [10, 11],
       'pipelines': [19]},
      {'id': 9,
       'name': 'onnx-tutorial',
       'created_at': '2023-11-22T16:24:47.786643+00:00',
       'created_by': '12ea09d1-0f49-405e-bed1-27eb6d10fde4',
       'archived': False,
       'models': [13],
       'pipelines': [22]}]}

List all workspaces via curl.

curl {wl.api_endpoint}/v1/api/workspaces/list \
    -H "Authorization: {wl.auth.auth_header()['Authorization']}" \
    -H "Content-Type: application/json" \
    --data '{{}}'
{
  "workspaces": [
    {
      "id": 1,
      "name": "john.hummel@wallaroo.ai - Default Workspace",
      "created_at": "2023-11-20T16:05:06.323911+00:00",
      "created_by": "12ea09d1-0f49-405e-bed1-27eb6d10fde4",
      "archived": false,
      "models": [],
      "pipelines": []
    },
    {
      "id": 5,
      "name": "mobilenetworkspacetest",
      "created_at": "2023-11-20T16:05:48.271364+00:00",
      "created_by": "12ea09d1-0f49-405e-bed1-27eb6d10fde4",
      "archived": false,
      "models": [1, 2],
      "pipelines": [1]
    },
    {
      "id": 6,
      "name": "edge-observability-assaysbaseline-examples",
      "created_at": "2023-11-20T16:09:31.950532+00:00",
      "created_by": "12ea09d1-0f49-405e-bed1-27eb6d10fde4",
      "archived": false,
      "models": [3],
      "pipelines": [4]
    },
    {
      "id": 7,
      "name": "edge-observability-houseprice-demo",
      "created_at": "2023-11-20T17:36:07.131292+00:00",
      "created_by": "12ea09d1-0f49-405e-bed1-27eb6d10fde4",
      "archived": false,
      "models": [4, 5, 6, 7, 8, 9, 12],
      "pipelines": [7, 14, 16]
    },
    {
      "id": 8,
      "name": "clip-demo",
      "created_at": "2023-11-20T18:57:53.667873+00:00",
      "created_by": "12ea09d1-0f49-405e-bed1-27eb6d10fde4",
      "archived": false,
      "models": [10, 11],
      "pipelines": [19]
    },
    {
      "id": 9,
      "name": "onnx-tutorial",
      "created_at": "2023-11-22T16:24:47.786643+00:00",
      "created_by": "12ea09d1-0f49-405e-bed1-27eb6d10fde4",
      "archived": false,
      "models": [13],
      "pipelines": [22]
    }
  ]
}

Create Workspace

  • Endpoint: /v1/api/workspaces/create

A new workspace will be created in the Wallaroo instance. Upon creating, the workspace owner will be assigned as the user making the MLOps API request.

Create Workspace Parameters

FieldTypeDescription
workspace_nameString (REQUIRED)The name of the new workspace with the following requirements:
  • Must be unique.
  • DNS compliant with only lowercase characters.

Create Workspace Returns

FieldTypeDescription
workspace_idIntegerThe ID of the new workspace.

Create Workspace Examples

In this example, workspaces named testapiworkspace-requests and testapiworkspace-curl will be created.

After the request is complete, the List Workspaces command will be issued to demonstrate the new workspace has been created.

Create workspace via Requests.

# Retrieve the token 
headers = wl.auth.auth_header()

endpoint = f"{wl.api_endpoint}/v1/api/workspaces/create"

data = {
  "workspace_name": "testapiworkspace-requests"
}

response = requests.post(endpoint, json=data, headers=headers, verify=True).json()
display(response)

# Stored for future examples
example_workspace_id = response['workspace_id']
{'workspace_id': 10}
## List workspaces

# Retrieve the token 
headers = wl.auth.auth_header()

endpoint = f"{wl.api_endpoint}/v1/api/workspaces/list"

data = {
}

response = requests.post(endpoint, 
                         json=data, 
                         headers=headers, 
                         verify=True).json()
response
{'workspaces': [{'id': 1,
   'name': 'john.hummel@wallaroo.ai - Default Workspace',
   'created_at': '2023-11-20T16:05:06.323911+00:00',
   'created_by': '12ea09d1-0f49-405e-bed1-27eb6d10fde4',
   'archived': False,
   'models': [],
   'pipelines': []},
  {'id': 5,
   'name': 'mobilenetworkspacetest',
   'created_at': '2023-11-20T16:05:48.271364+00:00',
   'created_by': '12ea09d1-0f49-405e-bed1-27eb6d10fde4',
   'archived': False,
   'models': [1, 2],
   'pipelines': [1]},
  {'id': 6,
   'name': 'edge-observability-assaysbaseline-examples',
   'created_at': '2023-11-20T16:09:31.950532+00:00',
   'created_by': '12ea09d1-0f49-405e-bed1-27eb6d10fde4',
   'archived': False,
   'models': [3],
   'pipelines': [4]},
  {'id': 7,
   'name': 'edge-observability-houseprice-demo',
   'created_at': '2023-11-20T17:36:07.131292+00:00',
   'created_by': '12ea09d1-0f49-405e-bed1-27eb6d10fde4',
   'archived': False,
   'models': [4, 5, 6, 7, 8, 9, 12],
   'pipelines': [7, 14, 16]},
  {'id': 8,
   'name': 'clip-demo',
   'created_at': '2023-11-20T18:57:53.667873+00:00',
   'created_by': '12ea09d1-0f49-405e-bed1-27eb6d10fde4',
   'archived': False,
   'models': [10, 11],
   'pipelines': [19]},
  {'id': 9,
   'name': 'onnx-tutorial',
   'created_at': '2023-11-22T16:24:47.786643+00:00',
   'created_by': '12ea09d1-0f49-405e-bed1-27eb6d10fde4',
   'archived': False,
   'models': [13],
   'pipelines': [22]},
  {'id': 10,
   'name': 'testapiworkspace-requests',
   'created_at': '2023-11-28T21:16:09.891951+00:00',
   'created_by': '12ea09d1-0f49-405e-bed1-27eb6d10fde4',
   'archived': False,
   'models': [],
   'pipelines': []}]}

Create workspace via curl.

curl {wl.api_endpoint}/v1/api/workspaces/create \
    -H "Authorization: {wl.auth.auth_header()['Authorization']}" \
    -H "Content-Type: application/json" \
    --data '{{"workspace_name": "testapiworkspace-curl"}}'
{"workspace_id":12}
curl {wl.api_endpoint}/v1/api/workspaces/list \
    -H "Authorization: {wl.auth.auth_header()['Authorization']}" \
    -H "Content-Type: application/json" \
    --data '{{}}'
{
  "workspaces": [
    {
      "id": 1,
      "name": "john.hummel@wallaroo.ai - Default Workspace",
      "created_at": "2023-11-20T16:05:06.323911+00:00",
      "created_by": "12ea09d1-0f49-405e-bed1-27eb6d10fde4",
      "archived": false,
      "models": [],
      "pipelines": []
    },
    {
      "id": 5,
      "name": "mobilenetworkspacetest",
      "created_at": "2023-11-20T16:05:48.271364+00:00",
      "created_by": "12ea09d1-0f49-405e-bed1-27eb6d10fde4",
      "archived": false,
      "models": [1, 2],
      "pipelines": [1]
    },
    {
      "id": 6,
      "name": "edge-observability-assaysbaseline-examples",
      "created_at": "2023-11-20T16:09:31.950532+00:00",
      "created_by": "12ea09d1-0f49-405e-bed1-27eb6d10fde4",
      "archived": false,
      "models": [3],
      "pipelines": [4]
    },
    {
      "id": 7,
      "name": "edge-observability-houseprice-demo",
      "created_at": "2023-11-20T17:36:07.131292+00:00",
      "created_by": "12ea09d1-0f49-405e-bed1-27eb6d10fde4",
      "archived": false,
      "models": [4, 5, 6, 7, 8, 9, 12],
      "pipelines": [7, 14, 16]
    },
    {
      "id": 8,
      "name": "clip-demo",
      "created_at": "2023-11-20T18:57:53.667873+00:00",
      "created_by": "12ea09d1-0f49-405e-bed1-27eb6d10fde4",
      "archived": false,
      "models": [10, 11],
      "pipelines": [19]
    },
    {
      "id": 9,
      "name": "onnx-tutorial",
      "created_at": "2023-11-22T16:24:47.786643+00:00",
      "created_by": "12ea09d1-0f49-405e-bed1-27eb6d10fde4",
      "archived": false,
      "models": [13],
      "pipelines": [22]
    },
    {
      "id": 10,
      "name": "testapiworkspace-requests",
      "created_at": "2023-11-28T21:16:09.891951+00:00",
      "created_by": "12ea09d1-0f49-405e-bed1-27eb6d10fde4",
      "archived": false,
      "models": [],
      "pipelines": []
    },
    {
      "id": 12,
      "name": "testapiworkspace-curl",
      "created_at": "2023-11-28T21:19:46.829351+00:00",
      "created_by": "12ea09d1-0f49-405e-bed1-27eb6d10fde4",
      "archived": false,
      "models": [],
      "pipelines": []
    }
  ]
}

Add User to Workspace

  • Endpoint: /v1/api/workspaces/add_user

Existing users of the Wallaroo instance can be added to an existing workspace.

Add User to Workspace Parameters

FieldTypeDescription
emailString (REQUIRED)The email address of the user to add to the workspace. This user must already exist in the Wallaroo instance.
workspace_idInteger (REQUIRED): The numerical id of the workspace.

Add User to Workspace Returns

Returns {} on a successful request.

Add User to Workspace Examples

The following example adds the user “john.hansarick@wallaroo.ai” to the workspace created in the previous step.

Add existing user to existing workspace via Requests.

# Retrieve the token 
headers = wl.auth.auth_header()

endpoint = f"{wl.api_endpoint}/v1/api/workspaces/add_user"

data = {
  "email": "john.hansarick@wallaroo.ai",
  "workspace_id": example_workspace_id
}

response = requests.post(endpoint, 
                         json=data, 
                         headers=headers, 
                         verify=True).json()
response
{}

Add existing user to existing workspace via curl.

curl {wl.api_endpoint}/v1/api/workspaces/add_user \
    -H "Authorization: {wl.auth.auth_header()['Authorization']}" \
    -H "Content-Type: application/json" \
    --data '{{"email": "john.hansarick@wallaroo.ai","workspace_id": {example_workspace_id}}}'
{}

List Users in a Workspace

  • Endpoint: /v1/api/workspaces/list_users

Lists the users who are either owners or collaborators of a workspace.

List Users in a Workspace Parameters

FieldTypeDescription
workspace_id*Integer (REQUIRED)The id of the workspace.

List Users in a Workspace Returns

Field TypeDescription
users List[users]The list of users and attributes in the workspace.
 user_idStringThe user’s Keycloak id.
 user_typeStringThe user’s workspace type of OWNER or COLLABORATOR.

List Users in a Workspace Examples

The following examples list all users part a workspace created in a previous request.

List users in a workspace via Requests.

# Retrieve the token 

headers = wl.auth.auth_header()

endpoint = f"{wl.api_endpoint}/v1/api/workspaces/list_users"

data = {
  "workspace_id": example_workspace_id
}

response = requests.post(endpoint, json=data, headers=headers, verify=True).json()
response
{
  "users": [
    { "user_id": "12ea09d1-0f49-405e-bed1-27eb6d10fde4", "user_type": "OWNER" },
    {
      "user_id": "57d61aed-3058-4327-9e65-a5d39a9718c0",
      "user_type": "COLLABORATOR"
    }
  ]
}

List users in a workspace via curl.

curl {wl.api_endpoint}/v1/api/workspaces/list_users \
    -H "Authorization: {wl.auth.auth_header()['Authorization']}" \
    -H "Content-Type: application/json" \
    --data '{{"workspace_id": {example_workspace_id}}}'
{
  "users": [
    { "user_id": "12ea09d1-0f49-405e-bed1-27eb6d10fde4", "user_type": "OWNER" },
    {
      "user_id": "57d61aed-3058-4327-9e65-a5d39a9718c0",
      "user_type": "COLLABORATOR"
    }
  ]
}

Remove User from a Workspace

Removes the user from the given workspace. In this request, either the user’s Keycloak ID is required OR the user’s email address is required.

Remove User from a Workspace Parameters

FieldTypeDescription
workspace_idInteger (Required)The id of the workspace.
user_idString (Optional)The Keycloak ID of the user. If email is not provided, then this parameter is REQUIRED.
emailString (Optional)The user’s email address. If user_id is not provided, then this parameter is REQUIRED.

Remove User from a Workspace Returns

FieldTypeDescription
affected_rowsIntegerThe number of workspaces effected by the change.

Remove User from a Workspace Examples

The following example will remove the user john.hansarick@wallaroo.ai from a workspace created the previous steps. Then the list of users for the workspace is retrieved to verify the change.

Remove existing user from an existing workspace via Requests.

# Retrieve the token 
headers = wl.auth.auth_header()

endpoint = f"{wl.api_endpoint}/v1/api/workspaces/remove_user"

data = {
  "email": "john.hansarick@wallaroo.ai",
  "workspace_id": example_workspace_id
}

response = requests.post(endpoint, json=data, headers=headers, verify=True).json()
response
{'affected_rows': 1}
# Retrieve the token 

headers = wl.auth.auth_header()

endpoint = f"{wl.api_endpoint}/v1/api/workspaces/list_users"

data = {
  "workspace_id": example_workspace_id
}

response = requests.post(endpoint, json=data, headers=headers, verify=True).json()
response
{
  "users": [
    { "user_id": "12ea09d1-0f49-405e-bed1-27eb6d10fde4", "user_type": "OWNER" }
  ]
}

Remove existing user from an existing workspace via curl.

curl {wl.api_endpoint}/v1/api/workspaces/remove_user \
    -H "Authorization: {wl.auth.auth_header()['Authorization']}" \
    -H "Content-Type: application/json" \
    --data '{{"email": "john.hansarick@wallaroo.ai","workspace_id": {example_workspace_id}}}'
{"affected_rows":0}
curl {wl.api_endpoint}/v1/api/workspaces/list_users \
    -H "Authorization: {wl.auth.auth_header()['Authorization']}" \
    -H "Content-Type: application/json" \
    --data '{{"workspace_id": {example_workspace_id}}}'
{
  "users": [
    { "user_id": "12ea09d1-0f49-405e-bed1-27eb6d10fde4", "user_type": "OWNER" }
  ]
}

3 - Wallaroo MLOps API Essentials Guide: Model Management

How to use the Wallaroo API for Model Management

Model Naming Requirements

Model names map onto Kubernetes objects, and must be DNS compliant. The strings for model names must lower case ASCII alpha-numeric characters or dash (-) only. . and _ are not allowed.

Models

The Wallaroo MLOps API allows users to upload different types of ML models and frameworks into Wallaroo.

Upload Model to Workspace

See Wallaroo MLOps API Essentials Guide: Model Upload and Registrations

List Models in Workspace

  • Endpoint: /v1/api/models/list

Returns a list of models added to a specific workspace.

List Models in Workspace Parameters

FieldTypeDescription
workspace_idInteger (REQUIRED)The workspace id to list.

List Models in Workspace Returns

Field TypeDescription
models List[models]List of models in the workspace.
 idIntegerThe numerical id of the model.
 **owner_idStringIdentifer of the model owner.
 created_atStringDateTime of the model’s creation.
 updated_atStringDateTime of the model’s last update.

List Models in Workspace Examples

Display the models for the workspace. This is assumed to be workspace_id of 10. Adjust the script for your own use.

List models in workspace via Requests.

# Retrieve the token 
headers = wl.auth.auth_header()

endpoint = f"{wl.api_endpoint}/v1/api/models/list"

data = {
  "workspace_id": workspace_id
}

response = requests.post(endpoint, 
                         json=data, 
                         headers=headers, 
                         verify=True).json()
display(response)
{
  "models": [
    {
      "id": 15,
      "name": "api-upload-pytorch-multi-io",
      "owner_id": "\"\"",
      "created_at": "2023-11-29T16:30:53.716526+00:00",
      "updated_at": "2023-11-29T18:20:39.610964+00:00"
    },
    {
      "id": 14,
      "name": "api-sample-model",
      "owner_id": "\"\"",
      "created_at": "2023-11-29T16:26:06.011817+00:00",
      "updated_at": "2023-11-29T16:26:06.011817+00:00"
    }
  ]
}

List models in workspace via curl.

curl {wl.api_endpoint}/v1/api/models/list \
    -H "Authorization: {wl.auth.auth_header()['Authorization']}" \
    -H "Content-Type: application/json" \
    --data '{{"workspace_id": {workspace_id}}}'
{
  "models": [
    {
      "id": 15,
      "name": "api-upload-pytorch-multi-io",
      "owner_id": "\"\"",
      "created_at": "2023-11-29T16:30:53.716526+00:00",
      "updated_at": "2023-11-29T18:20:39.610964+00:00"
    },
    {
      "id": 14,
      "name": "api-sample-model",
      "owner_id": "\"\"",
      "created_at": "2023-11-29T16:26:06.011817+00:00",
      "updated_at": "2023-11-29T16:26:06.011817+00:00"
    }
  ]
}

Get Model Details By ID

  • Endpoint: /v1/api/models/get_by_id

Returns the model details by the specific model id.

Get Model Details By ID Parameters

FieldTypeDescription
workspace_idInteger (REQUIRED)The workspace id to list.

Get Model Details By ID Returns

FieldTypeDescription
idIntegerNumerical id of the model.
owner_idStringId of the owner of the model.
workspace_idIntegerNumerical of the id the model is in.
nameStringName of the model.
updated_atStringDateTime of the model’s last update.
created_atStringDateTime of the model’s creation.
model_configStringDetails of the model’s configuration.

Model Config Options

Model version configurations are updated with the wallaroo.model_version.config and include the following parameters. Most are optional unless specified.

ParameterTypeDescription
runtimeString (Optional)The model runtime from wallaroo.framework. }
tensor_fields(List[string]) (Optional)A list of alternate input fields. For example, if the model accepts the input fields ['variable1', 'variable2'], tensor_fields allows those inputs to be overridden to ['square_feet', 'house_age'], or other values as required.
input_schemapyarrow.lib.SchemaThe input schema for the model in pyarrow.lib.Schema format.
output_schemapyarrow.lib.SchemaThe output schema for the model in pyarrow.lib.Schema format.
batch_config(List[string]) (Optional)Batch config is either None for multiple-input inferences, or single to accept an inference request with only one row of data.

Get Model Details By ID Examples

Retrieve the details for the model uploaded in the Upload Model to Workspace step. This will first list the models in the workspace with the id 10, then use that first model to display information. This assumes the workspace id and that there is at least one model uploaded to it.

Get Model Details By ID via Requests.

# Retrieve the token 
headers = wl.auth.auth_header()

endpoint = f"{wl.api_endpoint}/v1/api/models/list"

data = {
  "workspace_id": workspace_id
}

# first get the list of models in the workspace
response = requests.post(endpoint, 
                         json=data, 
                         headers=headers, 
                         verify=True).json()
example_model_id = response['models'][0]['id']
example_model_name = response['models'][0]['name']

# Get model details by id
# Retrieve the token 
headers = wl.auth.auth_header()

endpoint = f"{wl.api_endpoint}/v1/api/models/get_by_id"

data = {
  "id": example_model_id
}

response = requests.post(endpoint, json=data, headers=headers, verify=True).json()
display(response)
{
    'id': 15,
    'owner_id': '""',
    'workspace_id': 10,
    'name': 'api-upload-pytorch-multi-io',
    'updated_at': '2023-11-29T18:20:39.610964+00:00',
    'created_at': '2023-11-29T16:30:53.716526+00:00',
    'model_config': {
        'id': 25,
        'runtime': 'flight',
        'tensor_fields': None,
        'filter_threshold': None
    }
}

Get Model Details By ID via curl.

curl {wl.api_endpoint}/v1/api/models/get_by_id \
    -H "Authorization: {wl.auth.auth_header()['Authorization']}" \
    -H "Content-Type: application/json" \
    --data '{{"id": {example_model_id}}}'
{
  "id": 15,
  "owner_id": "\"\"",
  "workspace_id": 10,
  "name": "api-upload-pytorch-multi-io",
  "updated_at": "2023-11-29T18:20:39.610964+00:00",
  "created_at": "2023-11-29T16:30:53.716526+00:00",
  "model_config": {
    "id": 25,
    "runtime": "flight",
    "tensor_fields": null,
    "filter_threshold": null
  }
}

Get Model Versions

  • Endpoint: /v1/api/models/list_versions

Retrieves all versions of a model based on either the name of the model or the model_pk_id.

Get Model Versions Parameters

FieldTypeDescription
model_idString (REQUIRED)The model name.
models_pk_idInteger (REQUIRED)The model’s numerical id.

Get Model Versions Returns

Field TypeDescription
Unnamed List[models]A list of model versions for the requested model.
 shaStringThe sha hash of the model version.
 models_pk_idIntegerThe pk id of the model.
 model_versionStringThe UUID identifier of the model version.
 owner_idStringThe Keycloak user id of the model’s owner.
 model_idStringThe name of the model.
 idIntegerThe integer id of the model.
 file_nameStringThe filename used when uploading the model.
 image_pathStringThe image path of the model.

Retrieve the versions for a previously uploaded model. This assumes a workspace with id 10 has models already loaded into it.

Retrieve model versions via Requests.

# Retrieve the token 
headers = wl.auth.auth_header()

endpoint = f"{wl.api_endpoint}/v1/api/models/list"

data = {
  "workspace_id": workspace_id
}

# first get the list of models in the workspace
response = requests.post(endpoint, 
                         json=data, 
                         headers=headers, 
                         verify=True).json()
example_model_id = response['models'][0]['id']
example_model_name = response['models'][0]['name']

## List model versions

# Retrieve the token 
headers = wl.auth.auth_header()
endpoint = f"{wl.api_endpoint}/v1/api/models/list_versions"

data = {
  "model_id": example_model_name,
  "models_pk_id": example_model_id
}

response = requests.post(endpoint, json=data, headers=headers, verify=True).json()
display(response)
[{'sha': '792db9ee9f41aded3c1d4705f50ccdedd21cafb8b6232c03e4a849b6da1050a8',
  'models_pk_id': 15,
  'model_version': '271875ae-92ee-4137-b54f-c2ce1e88121c',
  'owner_id': '""',
  'model_id': 'api-upload-pytorch-multi-io',
  'id': 19,
  'file_name': 'model-auto-conversion_pytorch_multi_io_model.pt',
  'image_path': None,
  'status': 'error'},
 {'sha': 'bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507',
  'models_pk_id': 15,
  'model_version': 'ce648f2a-faee-4c8d-8a7b-e2789f3ab919',
  'owner_id': '""',
  'model_id': 'api-upload-pytorch-multi-io',
  'id': 17,
  'file_name': 'ccfraud.onnx',
  'image_path': None,
  'status': 'error'},
 {'sha': '792db9ee9f41aded3c1d4705f50ccdedd21cafb8b6232c03e4a849b6da1050a8',
  'models_pk_id': 15,
  'model_version': 'a6762893-be27-4142-ba09-4ce1b87b74a8',
  'owner_id': '""',
  'model_id': 'api-upload-pytorch-multi-io',
  'id': 15,
  'file_name': 'api-upload-pytorch-multi-io',
  'image_path': None,
  'status': 'error'},
 {'sha': 'bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507',
  'models_pk_id': 15,
  'model_version': '0f96fae1-8ebc-41d8-a11d-3eaa3bc26526',
  'owner_id': '""',
  'model_id': 'api-upload-pytorch-multi-io',
  'id': 18,
  'file_name': 'ccfraud.onnx',
  'image_path': None,
  'status': 'error'},
 {'sha': 'bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507',
  'models_pk_id': 15,
  'model_version': '9150b046-5df0-4c41-a60b-3016355f89d5',
  'owner_id': '""',
  'model_id': 'api-upload-pytorch-multi-io',
  'id': 16,
  'file_name': 'ccfraud.onnx',
  'image_path': 'proxy.replicated.com/proxy/wallaroo/ghcr.io/wallaroolabs/mlflow-deploy:v2023.4.0-4103',
  'status': 'ready'}]

Retrieve model versions via curl.

curl {wl.api_endpoint}/v1/api/models/list_versions \
    -H "Authorization: {wl.auth.auth_header()['Authorization']}" \
    -H "Content-Type: application/json" \
    -d '{json.dumps(data)}'
[
  {
    "sha": "792db9ee9f41aded3c1d4705f50ccdedd21cafb8b6232c03e4a849b6da1050a8",
    "models_pk_id": 15,
    "model_version": "271875ae-92ee-4137-b54f-c2ce1e88121c",
    "owner_id": "\"\"",
    "model_id": "api-upload-pytorch-multi-io",
    "id": 19,
    "file_name": "model-auto-conversion_pytorch_multi_io_model.pt",
    "image_path": null,
    "status": "error"
  },
  {
    "sha": "bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507",
    "models_pk_id": 15,
    "model_version": "ce648f2a-faee-4c8d-8a7b-e2789f3ab919",
    "owner_id": "\"\"",
    "model_id": "api-upload-pytorch-multi-io",
    "id": 17,
    "file_name": "ccfraud.onnx",
    "image_path": null,
    "status": "error"
  },
  {
    "sha": "792db9ee9f41aded3c1d4705f50ccdedd21cafb8b6232c03e4a849b6da1050a8",
    "models_pk_id": 15,
    "model_version": "a6762893-be27-4142-ba09-4ce1b87b74a8",
    "owner_id": "\"\"",
    "model_id": "api-upload-pytorch-multi-io",
    "id": 15,
    "file_name": "api-upload-pytorch-multi-io",
    "image_path": null,
    "status": "error"
  },
  {
    "sha": "bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507",
    "models_pk_id": 15,
    "model_version": "0f96fae1-8ebc-41d8-a11d-3eaa3bc26526",
    "owner_id": "\"\"",
    "model_id": "api-upload-pytorch-multi-io",
    "id": 18,
    "file_name": "ccfraud.onnx",
    "image_path": null,
    "status": "error"
  },
  {
    "sha": "bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507",
    "models_pk_id": 15,
    "model_version": "9150b046-5df0-4c41-a60b-3016355f89d5",
    "owner_id": "\"\"",
    "model_id": "api-upload-pytorch-multi-io",
    "id": 16,
    "file_name": "ccfraud.onnx",
    "image_path": "proxy.replicated.com/proxy/wallaroo/ghcr.io/wallaroolabs/mlflow-deploy:v2023.4.0-4103",
    "status": "ready"
  }
]

Get Model Configuration by Id

  • Endpoints: /v1/api/models/get_config_by_id

Returns the model’s configuration details.

Get Model Configuration by Id Parameters

FieldTypeDescription
model_idInteger (Required)The numerical value of the model’s id.

Get Model Configuration by Id Returns

FieldTypeDescription

Get Model Configuration by Id Examples

Submit the model id for the model uploaded in the Upload Model to Workspace step to retrieve configuration details.

Retrieve model configuration via Requests.

## Get model config by id

# Retrieve the token 
headers = wl.auth.auth_header()
endpoint = f"{wl.api_endpoint}/v1/api/models/get_config_by_id"

data = {
  "model_id": example_model_id
}

response = requests.post(endpoint, json=data, headers=headers, verify=True).json()
response
{'model_config': {'id': 25,
      'runtime': 'flight',
      'tensor_fields': None,
      'filter_threshold': None}}

Retrieve model configuration via curl.

curl {wl.api_endpoint}/v1/api/models/get_config_by_id \
    -H "Authorization: {wl.auth.auth_header()['Authorization']}" \
    -H "Content-Type: application/json" \
    -d '{json.dumps(data)}'
{
  "model_config": {
    "id": 25,
    "runtime": "flight",
    "tensor_fields": null,
    "filter_threshold": null
  }
}

Set Model Configuration by ID

  • Endpoint: /v1/api/models/insert_model_config

Model configurations are added with the endpoint /v1/api/models/insert_model_config endpoint. This sets the model configuration for the model version. Once uploaded, the model version configuration is set to the uploaded model config.

Set Model Configuration by ID Parameters

FieldTypeDescription
model_version_idInteger (REQUIRED)The model version id in numerical format.
tensor_fieldsList[String] (OPTIONAL)Overrides the model’s default input fields. Mainly used with ONNX models.
batch_configString (OPTIONAL)single: Only accepts one row of data at a time. None: Accepts multiple rows.
filter_thresholdInteger (OPTIONAL)The model’s filter threshold.
input_schemaString (OPTIONAL)The model’s input schema in PyArrow.Schema format. See Wallaroo MLOps API Essentials Guide: Model Upload and Registrations for details on supported model frameworks.
output_schemaString (OPTIONAL)The model’s output schema in PyArrow.Schema format. See Wallaroo MLOps API Essentials Guide: Model Upload and Registrations for details on supported model frameworks.
tensor_fieldsList[String] (OPTIONAL)The input field names for the model types. Primarily used to override ONNX model input fields.

Set Model Configuration by ID Returns

Returns the model config with the following fields.

FieldTypeDescription
idIntegerThe model configuration id.
model_version_idIntegerThe model version id.
runtimeStringThe Wallaroo runtime. See Wallaroo MLOps API Essentials Guide: Model Upload and Registrations for details on supported model frameworks.
tensor_fieldsList[String]The model input fields.
filter_thresholdIntegerThe model’s filter threshold.
input_schemaStringThe model’s input schema in PyArrow.Schema format. See Wallaroo MLOps API Essentials Guide: Model Upload and Registrations for details on supported model frameworks.
output_schemaStringThe model’s output schema in PyArrow.Schema format. See Wallaroo MLOps API Essentials Guide: Model Upload and Registrations for details on supported model frameworks.
tensor_fieldsList[String]The input field names for the model types. Primarily used to override ONNX model input fields.

Set Model Configuration by ID Example

The following example uses the Requests library to set a model’s configuration.

# Retrieve the token 
headers = wl.auth.auth_header()
api_request = f"{APIURL}/v1/api/models/insert_model_config"

data = {
    "model_version_id": 6,
    "batch_config": None,  # str
    "filter_threshold": 0.1,
    "input_schema": None,  # str
    "output_schema": None,  # str
    "tensor_fields": ["dense_1"]  # List[str]
}


response = requests.post(api_request, json=data, headers=headers, verify=True).json()
response

{'model_config': {'id': 20,
  'model_version_id': 6,
  'runtime': 'onnx',
  'filter_threshold': 0.1,
  'tensor_fields': ['dense_1'],
  'input_schema': None,
  'output_schema': None,
  'batch_config': None}}

Get Model Details

  • Endpoint: /v1/api/models/get

Get Model Details Parameters

Returns details regarding a single model, including versions.

FieldTypeDescription
model_idInteger (REQUIRED)The numerical value of the model’s id.

Get Model Details Returns

Field TypeDescription
id IntegerThe numerical value of the model’s id.
name StringThe name of the model.
owner_id StringThe model owner.
created_at StringDateTime of the model’s creation.
updated_at StringDateTime of the model’s last update.
models List[models]The list of model versions associated with this model.
 shaStringThe sha hash of the model version.
 models_pk_idIntegerThe model id.
 model_versionStringThe UUID identifier of the model version.
 owner_idStringThe model owner.
 model_idStringThe name of the model.
 idIntegerThe numerical identifier of the model version.
 file_nameStringThe file name used when uploading the model version
 image_pathString or NoneThe image path of the model verison.

Get Model Details Examples

Submit the model id for the model uploaded in the Upload Model to Workspace step to retrieve configuration details.

Get model details via Requests.

# Get model config by id
# Retrieve the token 
headers = wl.auth.auth_header()
endpoint = f"{wl.api_endpoint}/v1/api/models/get"

data = {
  "id": example_model_id
}

response = requests.post(endpoint, json=data, headers=headers, verify=True).json()
response
{'id': 15,
     'name': 'api-upload-pytorch-multi-io',
     'owner_id': '""',
     'created_at': '2023-11-29T16:30:53.716526+00:00',
     'updated_at': '2023-11-29T18:20:39.610964+00:00',
     'models': [{'sha': '792db9ee9f41aded3c1d4705f50ccdedd21cafb8b6232c03e4a849b6da1050a8',
       'models_pk_id': 15,
       'model_version': '271875ae-92ee-4137-b54f-c2ce1e88121c',
       'owner_id': '""',
       'model_id': 'api-upload-pytorch-multi-io',
       'id': 19,
       'file_name': 'model-auto-conversion_pytorch_multi_io_model.pt',
       'image_path': None},
      {'sha': 'bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507',
       'models_pk_id': 15,
       'model_version': 'ce648f2a-faee-4c8d-8a7b-e2789f3ab919',
       'owner_id': '""',
       'model_id': 'api-upload-pytorch-multi-io',
       'id': 17,
       'file_name': 'ccfraud.onnx',
       'image_path': None},
      {'sha': '792db9ee9f41aded3c1d4705f50ccdedd21cafb8b6232c03e4a849b6da1050a8',
       'models_pk_id': 15,
       'model_version': 'a6762893-be27-4142-ba09-4ce1b87b74a8',
       'owner_id': '""',
       'model_id': 'api-upload-pytorch-multi-io',
       'id': 15,
       'file_name': 'api-upload-pytorch-multi-io',
       'image_path': None},
      {'sha': 'bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507',
       'models_pk_id': 15,
       'model_version': '0f96fae1-8ebc-41d8-a11d-3eaa3bc26526',
       'owner_id': '""',
       'model_id': 'api-upload-pytorch-multi-io',
       'id': 18,
       'file_name': 'ccfraud.onnx',
       'image_path': None},
      {'sha': 'bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507',
       'models_pk_id': 15,
       'model_version': '9150b046-5df0-4c41-a60b-3016355f89d5',
       'owner_id': '""',
       'model_id': 'api-upload-pytorch-multi-io',
       'id': 16,
       'file_name': 'ccfraud.onnx',
       'image_path': 'proxy.replicated.com/proxy/wallaroo/ghcr.io/wallaroolabs/mlflow-deploy:v2023.4.0-4103'}]}

Get model details via curl.

curl {wl.api_endpoint}/v1/api/models/get \
    -H "Authorization: {wl.auth.auth_header()['Authorization']}" \
    -H "Content-Type: application/json" \
    -d '{{"id": {example_model_id}}}'
{
  "id": 15,
  "name": "api-upload-pytorch-multi-io",
  "owner_id": "\"\"",
  "created_at": "2023-11-29T16:30:53.716526+00:00",
  "updated_at": "2023-11-29T18:20:39.610964+00:00",
  "models": [
    {
      "sha": "792db9ee9f41aded3c1d4705f50ccdedd21cafb8b6232c03e4a849b6da1050a8",
      "models_pk_id": 15,
      "model_version": "271875ae-92ee-4137-b54f-c2ce1e88121c",
      "owner_id": "\"\"",
      "model_id": "api-upload-pytorch-multi-io",
      "id": 19,
      "file_name": "model-auto-conversion_pytorch_multi_io_model.pt",
      "image_path": null
    },
    {
      "sha": "bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507",
      "models_pk_id": 15,
      "model_version": "ce648f2a-faee-4c8d-8a7b-e2789f3ab919",
      "owner_id": "\"\"",
      "model_id": "api-upload-pytorch-multi-io",
      "id": 17,
      "file_name": "ccfraud.onnx",
      "image_path": null
    },
    {
      "sha": "792db9ee9f41aded3c1d4705f50ccdedd21cafb8b6232c03e4a849b6da1050a8",
      "models_pk_id": 15,
      "model_version": "a6762893-be27-4142-ba09-4ce1b87b74a8",
      "owner_id": "\"\"",
      "model_id": "api-upload-pytorch-multi-io",
      "id": 15,
      "file_name": "api-upload-pytorch-multi-io",
      "image_path": null
    },
    {
      "sha": "bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507",
      "models_pk_id": 15,
      "model_version": "0f96fae1-8ebc-41d8-a11d-3eaa3bc26526",
      "owner_id": "\"\"",
      "model_id": "api-upload-pytorch-multi-io",
      "id": 18,
      "file_name": "ccfraud.onnx",
      "image_path": null
    },
    {
      "sha": "bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507",
      "models_pk_id": 15,
      "model_version": "9150b046-5df0-4c41-a60b-3016355f89d5",
      "owner_id": "\"\"",
      "model_id": "api-upload-pytorch-multi-io",
      "id": 16,
      "file_name": "ccfraud.onnx",
      "image_path": "proxy.replicated.com/proxy/wallaroo/ghcr.io/wallaroolabs/mlflow-deploy:v2023.4.0-4103"
    }
  ]
}

4 - Wallaroo MLOps API Essentials Guide: Model Registry

How to use the Wallaroo API for Model Registry aka Artifact Registries

Wallaroo users can register their trained machine learning models from a model registry into their Wallaroo instance and perform inferences with it through a Wallaroo pipeline.

This guide details how to add ML Models from a model registry service into a Wallaroo instance.

Artifact Requirements

Models are uploaded to the Wallaroo instance as the specific artifact - the “file” or other data that represents the file itself. This must comply with the Wallaroo model requirements framework and version or it will not be deployed. Note that for models that fall outside of the supported model types, they can be registered to a Wallaroo workspace as MLFlow 1.30.0 containerized models.

Supported Models

The following frameworks are supported. Frameworks fall under either Native or Containerized runtimes in the Wallaroo engine. For more details, see the specific framework what runtime a specific model framework runs in.

Runtime DisplayModel Runtime SpacePipeline Configuration
tensorflowNativeNative Runtime Configuration Methods
onnxNativeNative Runtime Configuration Methods
pythonNativeNative Runtime Configuration Methods
mlflowContainerizedContainerized Runtime Deployment
flightContainerizedContainerized Runtime Deployment

Please note the following.

Wallaroo ONNX Requirements

Wallaroo natively supports Open Neural Network Exchange (ONNX) models into the Wallaroo engine.

ParameterDescription
Web Sitehttps://onnx.ai/
Supported LibrariesSee table below.
FrameworkFramework.ONNX aka onnx
RuntimeNative aka onnx

The following ONNX versions models are supported:

Wallaroo VersionONNX VersionONNX IR VersionONNX OPset VersionONNX ML Opset Version
2023.4.0 (October 2023)1.12.18173
2023.2.1 (July 2023)1.12.18173

For the most recent release of Wallaroo 2023.4.0, the following native runtimes are supported:

  • If converting another ML Model to ONNX (PyTorch, XGBoost, etc) using the onnxconverter-common library, the supported DEFAULT_OPSET_NUMBER is 17.

Using different versions or settings outside of these specifications may result in inference issues and other unexpected behavior.

ONNX models always run in the Wallaroo Native Runtime space.

Data Schemas

ONNX models deployed to Wallaroo have the following data requirements.

  • Equal rows constraint: The number of input rows and output rows must match.
  • All inputs are tensors: The inputs are tensor arrays with the same shape.
  • Data Type Consistency: Data types within each tensor are of the same type.

Equal Rows Constraint

Inference performed through ONNX models are assumed to be in batch format, where each input row corresponds to an output row. This is reflected in the in fields returned for an inference. In the following example, each input row for an inference is related directly to the inference output.

df = pd.read_json('./data/cc_data_1k.df.json')
display(df.head())

result = ccfraud_pipeline.infer(df.head())
display(result)

INPUT

 tensor
0[-1.0603297501, 2.3544967095000002, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192000001, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526000001, 1.9870535692, 0.7005485718000001, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756, -0.1466244739, -1.4463212439]
1[-1.0603297501, 2.3544967095000002, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192000001, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526000001, 1.9870535692, 0.7005485718000001, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756, -0.1466244739, -1.4463212439]
2[-1.0603297501, 2.3544967095000002, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192000001, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526000001, 1.9870535692, 0.7005485718000001, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756, -0.1466244739, -1.4463212439]
3[-1.0603297501, 2.3544967095000002, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192000001, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526000001, 1.9870535692, 0.7005485718000001, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756, -0.1466244739, -1.4463212439]
4[0.5817662108, 0.09788155100000001, 0.1546819424, 0.4754101949, -0.19788623060000002, -0.45043448540000003, 0.016654044700000002, -0.0256070551, 0.0920561602, -0.2783917153, 0.059329944100000004, -0.0196585416, -0.4225083157, -0.12175388770000001, 1.5473094894000001, 0.2391622864, 0.3553974881, -0.7685165301, -0.7000849355000001, -0.1190043285, -0.3450517133, -1.1065114108, 0.2523411195, 0.0209441826, 0.2199267436, 0.2540689265, -0.0450225094, 0.10867738980000001, 0.2547179311]

OUTPUT

 timein.tensorout.dense_1check_failures
02023-11-17 20:34:17.005[-1.0603297501, 2.3544967095, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526, 1.9870535692, 0.7005485718, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756, -0.1466244739, -1.4463212439][0.99300325]0
12023-11-17 20:34:17.005[-1.0603297501, 2.3544967095, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526, 1.9870535692, 0.7005485718, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756, -0.1466244739, -1.4463212439][0.99300325]0
22023-11-17 20:34:17.005[-1.0603297501, 2.3544967095, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526, 1.9870535692, 0.7005485718, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756, -0.1466244739, -1.4463212439][0.99300325]0
32023-11-17 20:34:17.005[-1.0603297501, 2.3544967095, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526, 1.9870535692, 0.7005485718, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756, -0.1466244739, -1.4463212439][0.99300325]0
42023-11-17 20:34:17.005[0.5817662108, 0.097881551, 0.1546819424, 0.4754101949, -0.1978862306, -0.4504344854, 0.0166540447, -0.0256070551, 0.0920561602, -0.2783917153, 0.0593299441, -0.0196585416, -0.4225083157, -0.1217538877, 1.5473094894, 0.2391622864, 0.3553974881, -0.7685165301, -0.7000849355, -0.1190043285, -0.3450517133, -1.1065114108, 0.2523411195, 0.0209441826, 0.2199267436, 0.2540689265, -0.0450225094, 0.1086773898, 0.2547179311][0.0010916889]0

All Inputs Are Tensors

All inputs into an ONNX model must be tensors. This requires that the shape of each element is the same. For example, the following is a proper input:

t [
    [2.35, 5.75],
    [3.72, 8.55],
    [5.55, 97.2]
]
Standard tensor array

Another example is a 2,2,3 tensor, where the shape of each element is (3,), and each element has 2 rows.

t = [
        [2.35, 5.75, 19.2],
        [3.72, 8.55, 10.5]
    ],
    [
        [5.55, 7.2, 15.7],
        [9.6, 8.2, 2.3]
    ]

In this example each element has a shape of (2,). Tensors with elements of different shapes, known as ragged tensors, are not supported. For example:

t = [
    [2.35, 5.75],
    [3.72, 8.55, 10.5],
    [5.55, 97.2]
])

**INVALID SHAPE**
Ragged tensor array - unsupported

For models that require ragged tensor or other shapes, see other data formatting options such as Bring Your Own Predict models.

Data Type Consistency

All inputs into an ONNX model must have the same internal data type. For example, the following is valid because all of the data types within each element are float32.

t = [
    [2.35, 5.75],
    [3.72, 8.55],
    [5.55, 97.2]
]

The following is invalid, as it mixes floats and strings in each element:

t = [
    [2.35, "Bob"],
    [3.72, "Nancy"],
    [5.55, "Wani"]
]

The following inputs are valid, as each data type is consistent within the elements.

df = pd.DataFrame({
    "t": [
        [2.35, 5.75, 19.2],
        [5.55, 7.2, 15.7],
    ],
    "s": [
        ["Bob", "Nancy", "Wani"],
        ["Jason", "Rita", "Phoebe"]
    ]
})
df
 ts
0[2.35, 5.75, 19.2][Bob, Nancy, Wani]
1[5.55, 7.2, 15.7][Jason, Rita, Phoebe]
ParameterDescription
Web Sitehttps://www.tensorflow.org/
Supported Librariestensorflow==2.9.3
FrameworkFramework.TENSORFLOW aka tensorflow
RuntimeNative aka tensorflow
Supported File TypesSavedModel format as .zip file

TensorFlow File Format

TensorFlow models are .zip file of the SavedModel format. For example, the Aloha sample TensorFlow model is stored in the directory alohacnnlstm:

├── saved_model.pb
└── variables
    ├── variables.data-00000-of-00002
    ├── variables.data-00001-of-00002
    └── variables.index

This is compressed into the .zip file alohacnnlstm.zip with the following command:

zip -r alohacnnlstm.zip alohacnnlstm/

ML models that meet the Tensorflow and SavedModel format will run as Wallaroo Native runtimes by default.

See the SavedModel guide for full details.

ParameterDescription
Web Sitehttps://www.python.org/
Supported Librariespython==3.8
FrameworkFramework.PYTHON aka python
RuntimeNative aka python

Python models uploaded to Wallaroo are executed as a native runtime.

Note that Python models - aka “Python steps” - are standalone python scripts that use the python libraries natively supported by the Wallaroo platform. These are used for either simple model deployment (such as ARIMA Statsmodels), or data formatting such as the postprocessing steps. A Wallaroo Python model will be composed of one Python script that matches the Wallaroo requirements.

This is contrasted with Arbitrary Python models, also known as Bring Your Own Predict (BYOP) allow for custom model deployments with supporting scripts and artifacts. These are used with pre-trained models (PyTorch, Tensorflow, etc) along with whatever supporting artifacts they require. Supporting artifacts can include other Python modules, model files, etc. These are zipped with all scripts, artifacts, and a requirements.txt file that indicates what other Python models need to be imported that are outside of the typical Wallaroo platform.

Python Models Requirements

Python models uploaded to Wallaroo are Python scripts that must include the wallaroo_json method as the entry point for the Wallaroo engine to use it as a Pipeline step.

This method receives the results of the previous Pipeline step, and its return value will be used in the next Pipeline step.

If the Python model is the first step in the pipeline, then it will be receiving the inference request data (for example: a preprocessing step). If it is the last step in the pipeline, then it will be the data returned from the inference request.

In the example below, the Python model is used as a post processing step for another ML model. The Python model expects to receive data from a ML Model who’s output is a DataFrame with the column dense_2. It then extracts the values of that column as a list, selects the first element, and returns a DataFrame with that element as the value of the column output.

def wallaroo_json(data: pd.DataFrame):
    print(data)
    return [{"output": [data["dense_2"].to_list()[0][0]]}]

In line with other Wallaroo inference results, the outputs of a Python step that returns a pandas DataFrame or Arrow Table will be listed in the out. metadata, with all inference outputs listed as out.{variable 1}, out.{variable 2}, etc. In the example above, this results the output field as the out.output field in the Wallaroo inference result.

 timein.tensorout.outputcheck_failures
02023-06-20 20:23:28.395[0.6878518042, 0.1760734021, -0.869514083, 0.3..[12.886651039123535]0
ParameterDescription
Web Sitehttps://huggingface.co/models
Supported Libraries
  • transformers==4.34.1
  • diffusers==0.14.0
  • accelerate==0.23.0
  • torchvision==0.14.1
  • torch==1.13.1
FrameworksThe following Hugging Face pipelines are supported by Wallaroo.
  • Framework.HUGGING_FACE_FEATURE_EXTRACTION aka hugging-face-feature-extraction
  • Framework.HUGGING_FACE_IMAGE_CLASSIFICATION aka hugging-face-image-classification
  • Framework.HUGGING_FACE_IMAGE_SEGMENTATION aka hugging-face-image-segmentation
  • Framework.HUGGING_FACE_IMAGE_TO_TEXT aka hugging-face-image-to-text
  • Framework.HUGGING_FACE_OBJECT_DETECTION aka hugging-face-object-detection
  • Framework.HUGGING_FACE_QUESTION_ANSWERING aka hugging-face-question-answering
  • Framework.HUGGING_FACE_STABLE_DIFFUSION_TEXT_2_IMG aka hugging-face-stable-diffusion-text-2-img
  • Framework.HUGGING_FACE_SUMMARIZATION aka hugging-face-summarization
  • Framework.HUGGING_FACE_TEXT_CLASSIFICATION aka hugging-face-text-classification
  • Framework.HUGGING_FACE_TRANSLATION aka hugging-face-translation
  • Framework.HUGGING_FACE_ZERO_SHOT_CLASSIFICATION aka hugging-face-zero-shot-classification
  • Framework.HUGGING_FACE_ZERO_SHOT_IMAGE_CLASSIFICATION aka hugging-face-zero-shot-image-classification
  • Framework.HUGGING_FACE_ZERO_SHOT_OBJECT_DETECTION aka hugging-face-zero-shot-object-detection
  • Framework.HUGGING_FACE_SENTIMENT_ANALYSIS aka hugging-face-sentiment-analysis
  • Framework.HUGGING_FACE_TEXT_GENERATION aka hugging-face-text-generation
  • Framework.HUGGING_FACE_AUTOMATIC_SPEECH_RECOGNITION aka hugging-face-automatic-speech-recognition
RuntimeContainerized flight

During the model upload process, the Wallaroo instance will attempt to convert the model to a Native Wallaroo Runtime. If unsuccessful based , it will create a Wallaroo Containerized Runtime for the model. See the model deployment section for details on how to configure pipeline resources based on the model’s runtime.

Hugging Face Schemas

Input and output schemas for each Hugging Face pipeline are defined below. Note that adding additional inputs not specified below will raise errors, except for the following:

  • Framework.HUGGING_FACE_IMAGE_TO_TEXT
  • Framework.HUGGING_FACE_TEXT_CLASSIFICATION
  • Framework.HUGGING_FACE_SUMMARIZATION
  • Framework.HUGGING_FACE_TRANSLATION

Additional inputs added to these Hugging Face pipelines will be added as key/pair value arguments to the model’s generate method. If the argument is not required, then the model will default to the values coded in the original Hugging Face model’s source code.

See the Hugging Face Pipeline documentation for more details on each pipeline and framework.

Wallaroo FrameworkReference
Framework.HUGGING_FACE_FEATURE_EXTRACTION

Schemas:

input_schema = pa.schema([
    pa.field('inputs', pa.string())
])
output_schema = pa.schema([
    pa.field('output', pa.list_(
        pa.list_(
            pa.float64(),
            list_size=128
        ),
    ))
])
Wallaroo FrameworkReference
Framework.HUGGING_FACE_IMAGE_CLASSIFICATION

Schemas:

input_schema = pa.schema([
    pa.field('inputs', pa.list_(
        pa.list_(
            pa.list_(
                pa.int64(),
                list_size=3
            ),
            list_size=100
        ),
        list_size=100
    )),
    pa.field('top_k', pa.int64()),
])

output_schema = pa.schema([
    pa.field('score', pa.list_(pa.float64(), list_size=2)),
    pa.field('label', pa.list_(pa.string(), list_size=2)),
])
Wallaroo FrameworkReference
Framework.HUGGING_FACE_IMAGE_SEGMENTATION

Schemas:

input_schema = pa.schema([
    pa.field('inputs', 
        pa.list_(
            pa.list_(
                pa.list_(
                    pa.int64(),
                    list_size=3
                ),
                list_size=100
            ),
        list_size=100
    )),
    pa.field('threshold', pa.float64()),
    pa.field('mask_threshold', pa.float64()),
    pa.field('overlap_mask_area_threshold', pa.float64()),
])

output_schema = pa.schema([
    pa.field('score', pa.list_(pa.float64())),
    pa.field('label', pa.list_(pa.string())),
    pa.field('mask', 
        pa.list_(
            pa.list_(
                pa.list_(
                    pa.int64(),
                    list_size=100
                ),
                list_size=100
            ),
    )),
])
Wallaroo FrameworkReference
Framework.HUGGING_FACE_IMAGE_TO_TEXT

Any parameter that is not part of the required inputs list will be forwarded to the model as a key/pair value to the underlying models generate method. If the additional input is not supported by the model, an error will be returned.

Schemas:

input_schema = pa.schema([
    pa.field('inputs', pa.list_( #required
        pa.list_(
            pa.list_(
                pa.int64(),
                list_size=3
            ),
            list_size=100
        ),
        list_size=100
    )),
    # pa.field('max_new_tokens', pa.int64()),  # optional
])

output_schema = pa.schema([
    pa.field('generated_text', pa.list_(pa.string())),
])
Wallaroo FrameworkReference
Framework.HUGGING_FACE_OBJECT_DETECTION

Schemas:

input_schema = pa.schema([
    pa.field('inputs', 
        pa.list_(
            pa.list_(
                pa.list_(
                    pa.int64(),
                    list_size=3
                ),
                list_size=100
            ),
        list_size=100
    )),
    pa.field('threshold', pa.float64()),
])

output_schema = pa.schema([
    pa.field('score', pa.list_(pa.float64())),
    pa.field('label', pa.list_(pa.string())),
    pa.field('box', 
        pa.list_( # dynamic output, i.e. dynamic number of boxes per input image, each sublist contains the 4 box coordinates 
            pa.list_(
                    pa.int64(),
                    list_size=4
                ),
            ),
    ),
])
Wallaroo FrameworkReference
Framework.HUGGING_FACE_QUESTION_ANSWERING

Schemas:

input_schema = pa.schema([
    pa.field('question', pa.string()),
    pa.field('context', pa.string()),
    pa.field('top_k', pa.int64()),
    pa.field('doc_stride', pa.int64()),
    pa.field('max_answer_len', pa.int64()),
    pa.field('max_seq_len', pa.int64()),
    pa.field('max_question_len', pa.int64()),
    pa.field('handle_impossible_answer', pa.bool_()),
    pa.field('align_to_words', pa.bool_()),
])

output_schema = pa.schema([
    pa.field('score', pa.float64()),
    pa.field('start', pa.int64()),
    pa.field('end', pa.int64()),
    pa.field('answer', pa.string()),
])
Wallaroo FrameworkReference
Framework.HUGGING_FACE_STABLE_DIFFUSION_TEXT_2_IMG

Schemas:

input_schema = pa.schema([
    pa.field('prompt', pa.string()),
    pa.field('height', pa.int64()),
    pa.field('width', pa.int64()),
    pa.field('num_inference_steps', pa.int64()), # optional
    pa.field('guidance_scale', pa.float64()), # optional
    pa.field('negative_prompt', pa.string()), # optional
    pa.field('num_images_per_prompt', pa.string()), # optional
    pa.field('eta', pa.float64()) # optional
])

output_schema = pa.schema([
    pa.field('images', pa.list_(
        pa.list_(
            pa.list_(
                pa.int64(),
                list_size=3
            ),
            list_size=128
        ),
        list_size=128
    )),
])
Wallaroo FrameworkReference
Framework.HUGGING_FACE_SUMMARIZATION

Any parameter that is not part of the required inputs list will be forwarded to the model as a key/pair value to the underlying models generate method. If the additional input is not supported by the model, an error will be returned.

Schemas:

input_schema = pa.schema([
    pa.field('inputs', pa.string()),
    pa.field('return_text', pa.bool_()),
    pa.field('return_tensors', pa.bool_()),
    pa.field('clean_up_tokenization_spaces', pa.bool_()),
    # pa.field('extra_field', pa.int64()), # every extra field you specify will be forwarded as a key/value pair
])

output_schema = pa.schema([
    pa.field('summary_text', pa.string()),
])
Wallaroo FrameworkReference
Framework.HUGGING_FACE_TEXT_CLASSIFICATION

Schemas

input_schema = pa.schema([
    pa.field('inputs', pa.string()), # required
    pa.field('top_k', pa.int64()), # optional
    pa.field('function_to_apply', pa.string()), # optional
])

output_schema = pa.schema([
    pa.field('label', pa.list_(pa.string(), list_size=2)), # list with a number of items same as top_k, list_size can be skipped but may lead in worse performance
    pa.field('score', pa.list_(pa.float64(), list_size=2)), # list with a number of items same as top_k, list_size can be skipped but may lead in worse performance
])
Wallaroo FrameworkReference
Framework.HUGGING_FACE_TRANSLATION

Any parameter that is not part of the required inputs list will be forwarded to the model as a key/pair value to the underlying models generate method. If the additional input is not supported by the model, an error will be returned.

Schemas:

input_schema = pa.schema([
    pa.field('inputs', pa.string()), # required
    pa.field('return_tensors', pa.bool_()), # optional
    pa.field('return_text', pa.bool_()), # optional
    pa.field('clean_up_tokenization_spaces', pa.bool_()), # optional
    pa.field('src_lang', pa.string()), # optional
    pa.field('tgt_lang', pa.string()), # optional
    # pa.field('extra_field', pa.int64()), # every extra field you specify will be forwarded as a key/value pair
])

output_schema = pa.schema([
    pa.field('translation_text', pa.string()),
])
Wallaroo FrameworkReference
Framework.HUGGING_FACE_ZERO_SHOT_CLASSIFICATION

Schemas:

input_schema = pa.schema([
    pa.field('inputs', pa.string()), # required
    pa.field('candidate_labels', pa.list_(pa.string(), list_size=2)), # required
    pa.field('hypothesis_template', pa.string()), # optional
    pa.field('multi_label', pa.bool_()), # optional
])

output_schema = pa.schema([
    pa.field('sequence', pa.string()),
    pa.field('scores', pa.list_(pa.float64(), list_size=2)), # same as number of candidate labels, list_size can be skipped by may result in slightly worse performance
    pa.field('labels', pa.list_(pa.string(), list_size=2)), # same as number of candidate labels, list_size can be skipped by may result in slightly worse performance
])
Wallaroo FrameworkReference
Framework.HUGGING_FACE_ZERO_SHOT_IMAGE_CLASSIFICATION

Schemas:

input_schema = pa.schema([
    pa.field('inputs', # required
        pa.list_(
            pa.list_(
                pa.list_(
                    pa.int64(),
                    list_size=3
                ),
                list_size=100
            ),
        list_size=100
    )),
    pa.field('candidate_labels', pa.list_(pa.string(), list_size=2)), # required
    pa.field('hypothesis_template', pa.string()), # optional
]) 

output_schema = pa.schema([
    pa.field('score', pa.list_(pa.float64(), list_size=2)), # same as number of candidate labels
    pa.field('label', pa.list_(pa.string(), list_size=2)), # same as number of candidate labels
])
Wallaroo FrameworkReference
Framework.HUGGING_FACE_ZERO_SHOT_OBJECT_DETECTION

Schemas:

input_schema = pa.schema([
    pa.field('images', 
        pa.list_(
            pa.list_(
                pa.list_(
                    pa.int64(),
                    list_size=3
                ),
                list_size=640
            ),
        list_size=480
    )),
    pa.field('candidate_labels', pa.list_(pa.string(), list_size=3)),
    pa.field('threshold', pa.float64()),
    # pa.field('top_k', pa.int64()), # we want the model to return exactly the number of predictions, we shouldn't specify this
])

output_schema = pa.schema([
    pa.field('score', pa.list_(pa.float64())), # variable output, depending on detected objects
    pa.field('label', pa.list_(pa.string())), # variable output, depending on detected objects
    pa.field('box', 
        pa.list_( # dynamic output, i.e. dynamic number of boxes per input image, each sublist contains the 4 box coordinates 
            pa.list_(
                    pa.int64(),
                    list_size=4
                ),
            ),
    ),
])
Wallaroo FrameworkReference
Framework.HUGGING_FACE_SENTIMENT_ANALYSISHugging Face Sentiment Analysis
Wallaroo FrameworkReference
Framework.HUGGING_FACE_TEXT_GENERATION

Any parameter that is not part of the required inputs list will be forwarded to the model as a key/pair value to the underlying models generate method. If the additional input is not supported by the model, an error will be returned.

input_schema = pa.schema([
    pa.field('inputs', pa.string()),
    pa.field('return_tensors', pa.bool_()), # optional
    pa.field('return_text', pa.bool_()), # optional
    pa.field('return_full_text', pa.bool_()), # optional
    pa.field('clean_up_tokenization_spaces', pa.bool_()), # optional
    pa.field('prefix', pa.string()), # optional
    pa.field('handle_long_generation', pa.string()), # optional
    # pa.field('extra_field', pa.int64()), # every extra field you specify will be forwarded as a key/value pair
])

output_schema = pa.schema([
    pa.field('generated_text', pa.list_(pa.string(), list_size=1))
])
Wallaroo FrameworkReference
Framework.HUGGING_FACE_AUTOMATIC_SPEECH_RECOGNITION

Sample input and output schema.

input_schema = pa.schema([
    pa.field('inputs', pa.list_(pa.float32())), # required: the audio stored in numpy arrays of shape (num_samples,) and data type `float32`
    pa.field('return_timestamps', pa.string()) # optional: return start & end times for each predicted chunk
]) 

output_schema = pa.schema([
    pa.field('text', pa.string()), # required: the output text corresponding to the audio input
    pa.field('chunks', pa.list_(pa.struct([('text', pa.string()), ('timestamp', pa.list_(pa.float32()))]))), # required (if `return_timestamps` is set), start & end times for each predicted chunk
])
ParameterDescription
Web Sitehttps://pytorch.org/
Supported Libraries
  • torch==1.13.1
  • torchvision==0.14.1
FrameworkFramework.PYTORCH aka pytorch
Supported File Typespt ot pth in TorchScript format
Runtimeonnx/flight
  • IMPORTANT NOTE: The PyTorch model must be in TorchScript format. scripting (i.e. torch.jit.script() is always recommended over tracing (i.e. torch.jit.trace()). From the PyTorch documentation: “Scripting preserves dynamic control flow and is valid for inputs of different sizes.” For more details, see TorchScript-based ONNX Exporter: Tracing vs Scripting.

During the model upload process, the Wallaroo instance will attempt to convert the model to a Native Wallaroo Runtime. If unsuccessful based , it will create a Wallaroo Containerized Runtime for the model. See the model deployment section for details on how to configure pipeline resources based on the model’s runtime.

  • IMPORTANT CONFIGURATION NOTE: For PyTorch input schemas, the floats must be pyarrow.float32() for the PyTorch model to be converted to the Native Wallaroo Runtime during the upload process.

Sci-kit Learn aka SKLearn.

ParameterDescription
Web Sitehttps://scikit-learn.org/stable/index.html
Supported Libraries
  • scikit-learn==1.3.0
FrameworkFramework.SKLEARN aka sklearn
Runtimeonnx / flight

During the model upload process, the Wallaroo instance will attempt to convert the model to a Native Wallaroo Runtime. If unsuccessful based , it will create a Wallaroo Containerized Runtime for the model. See the model deployment section for details on how to configure pipeline resources based on the model’s runtime.

SKLearn Schema Inputs

SKLearn schema follows a different format than other models. To prevent inputs from being out of order, the inputs should be submitted in a single row in the order the model is trained to accept, with all of the data types being the same. For example, the following DataFrame has 4 columns, each column a float.

 sepal length (cm)sepal width (cm)petal length (cm)petal width (cm)
05.13.51.40.2
14.93.01.40.2

For submission to an SKLearn model, the data input schema will be a single array with 4 float values.

input_schema = pa.schema([
    pa.field('inputs', pa.list_(pa.float64(), list_size=4))
])

When submitting as an inference, the DataFrame is converted to rows with the column data expressed as a single array. The data must be in the same order as the model expects, which is why the data is submitted as a single array rather than JSON labeled columns: this insures that the data is submitted in the exact order as the model is trained to accept.

Original DataFrame:

 sepal length (cm)sepal width (cm)petal length (cm)petal width (cm)
05.13.51.40.2
14.93.01.40.2

Converted DataFrame:

 inputs
0[5.1, 3.5, 1.4, 0.2]
1[4.9, 3.0, 1.4, 0.2]

SKLearn Schema Outputs

Outputs for SKLearn that are meant to be predictions or probabilities when output by the model are labeled in the output schema for the model when uploaded to Wallaroo. For example, a model that outputs either 1 or 0 as its output would have the output schema as follows:

output_schema = pa.schema([
    pa.field('predictions', pa.int32())
])

When used in Wallaroo, the inference result is contained in the out metadata as out.predictions.

pipeline.infer(dataframe)
 timein.inputsout.predictionscheck_failures
02023-07-05 15:11:29.776[5.1, 3.5, 1.4, 0.2]00
12023-07-05 15:11:29.776[4.9, 3.0, 1.4, 0.2]00
ParameterDescription
Web Sitehttps://www.tensorflow.org/api_docs/python/tf/keras/Model
Supported Libraries
  • tensorflow==2.9.3
  • keras==2.9.0
FrameworkFramework.KERAS aka keras
Supported File TypesSavedModel format as .zip file and HDF5 format
Runtimeonnx/flight

During the model upload process, the Wallaroo instance will attempt to convert the model to a Native Wallaroo Runtime. If unsuccessful based , it will create a Wallaroo Containerized Runtime for the model. See the model deployment section for details on how to configure pipeline resources based on the model’s runtime.

TensorFlow Keras SavedModel Format

TensorFlow Keras SavedModel models are .zip file of the SavedModel format. For example, the Aloha sample TensorFlow model is stored in the directory alohacnnlstm:

├── saved_model.pb
└── variables
    ├── variables.data-00000-of-00002
    ├── variables.data-00001-of-00002
    └── variables.index

This is compressed into the .zip file alohacnnlstm.zip with the following command:

zip -r alohacnnlstm.zip alohacnnlstm/

See the SavedModel guide for full details.

TensorFlow Keras H5 Format

Wallaroo supports the H5 for Tensorflow Keras models.

ParameterDescription
Web Sitehttps://xgboost.ai/
Supported Libraries
  • scikit-learn==1.3.0
  • xgboost==1.7.4
FrameworkFramework.XGBOOST aka xgboost
Supported File Typespickle (XGB files are not supported.)
Runtimeonnx / flight

During the model upload process, the Wallaroo instance will attempt to convert the model to a Native Wallaroo Runtime. If unsuccessful based , it will create a Wallaroo Containerized Runtime for the model. See the model deployment section for details on how to configure pipeline resources based on the model’s runtime.

XGBoost Schema Inputs

XGBoost schema follows a different format than other models. To prevent inputs from being out of order, the inputs should be submitted in a single row in the order the model is trained to accept, with all of the data types being the same. If a model is originally trained to accept inputs of different data types, it will need to be retrained to only accept one data type for each column - typically pa.float64() is a good choice.

For example, the following DataFrame has 4 columns, each column a float.

 sepal length (cm)sepal width (cm)petal length (cm)petal width (cm)
05.13.51.40.2
14.93.01.40.2

For submission to an XGBoost model, the data input schema will be a single array with 4 float values.

input_schema = pa.schema([
    pa.field('inputs', pa.list_(pa.float64(), list_size=4))
])

When submitting as an inference, the DataFrame is converted to rows with the column data expressed as a single array. The data must be in the same order as the model expects, which is why the data is submitted as a single array rather than JSON labeled columns: this insures that the data is submitted in the exact order as the model is trained to accept.

Original DataFrame:

 sepal length (cm)sepal width (cm)petal length (cm)petal width (cm)
05.13.51.40.2
14.93.01.40.2

Converted DataFrame:

 inputs
0[5.1, 3.5, 1.4, 0.2]
1[4.9, 3.0, 1.4, 0.2]

XGBoost Schema Outputs

Outputs for XGBoost are labeled based on the trained model outputs. For this example, the output is simply a single output listed as output. In the Wallaroo inference result, it is grouped with the metadata out as out.output.

output_schema = pa.schema([
    pa.field('output', pa.int32())
])
pipeline.infer(dataframe)
 timein.inputsout.outputcheck_failures
02023-07-05 15:11:29.776[5.1, 3.5, 1.4, 0.2]00
12023-07-05 15:11:29.776[4.9, 3.0, 1.4, 0.2]00
ParameterDescription
Web Sitehttps://www.python.org/
Supported Librariespython==3.8
FrameworkFramework.CUSTOM aka custom
RuntimeContainerized aka flight

Arbitrary Python models, also known as Bring Your Own Predict (BYOP) allow for custom model deployments with supporting scripts and artifacts. These are used with pre-trained models (PyTorch, Tensorflow, etc) along with whatever supporting artifacts they require. Supporting artifacts can include other Python modules, model files, etc. These are zipped with all scripts, artifacts, and a requirements.txt file that indicates what other Python models need to be imported that are outside of the typical Wallaroo platform.

Contrast this with Wallaroo Python models - aka “Python steps”. These are standalone python scripts that use the python libraries natively supported by the Wallaroo platform. These are used for either simple model deployment (such as ARIMA Statsmodels), or data formatting such as the postprocessing steps. A Wallaroo Python model will be composed of one Python script that matches the Wallaroo requirements.

Arbitrary Python File Requirements

Arbitrary Python (BYOP) models are uploaded to Wallaroo via a ZIP file with the following components:

ArtifactTypeDescription
Python scripts aka .py files with classes that extend mac.inference.Inference and mac.inference.creation.InferenceBuilderPython ScriptExtend the classes mac.inference.Inference and mac.inference.creation.InferenceBuilder. These are included with the Wallaroo SDK. Further details are in Arbitrary Python Script Requirements. Note that there is no specified naming requirements for the classes that extend mac.inference.Inference and mac.inference.creation.InferenceBuilder - any qualified class name is sufficient as long as these two classes are extended as defined below.
requirements.txtPython requirements fileThis sets the Python libraries used for the arbitrary python model. These libraries should be targeted for Python 3.8 compliance. These requirements and the versions of libraries should be exactly the same between creating the model and deploying it in Wallaroo. This insures that the script and methods will function exactly the same as during the model creation process.
Other artifactsFilesOther models, files, and other artifacts used in support of this model.

For example, the if the arbitrary python model will be known as vgg_clustering, the contents may be in the following structure, with vgg_clustering as the storage directory:

vgg_clustering\
    feature_extractor.h5
    kmeans.pkl
    custom_inference.py
    requirements.txt

Note the inclusion of the custom_inference.py file. This file name is not required - any Python script or scripts that extend the classes listed above are sufficient. This Python script could have been named vgg_custom_model.py or any other name as long as it includes the extension of the classes listed above.

The sample arbitrary python model file is created with the command zip -r vgg_clustering.zip vgg_clustering/.

Wallaroo Arbitrary Python uses the Wallaroo SDK mac module, included in the Wallaroo SDK 2023.2.1 and above. See the Wallaroo SDK Install Guides for instructions on installing the Wallaroo SDK.

Arbitrary Python Script Requirements

The entry point of the arbitrary python model is any python script that extends the following classes. These are included with the Wallaroo SDK. The required methods that must be overridden are specified in each section below.

  • mac.inference.Inference interface serves model inferences based on submitted input some input. Its purpose is to serve inferences for any supported arbitrary model framework (e.g. scikit, keras etc.).

    classDiagram
        class Inference {
            <<Abstract>>
            +model Optional[Any]
            +expected_model_types()* Set
            +predict(input_data: InferenceData)*  InferenceData
            -raise_error_if_model_is_not_assigned() None
            -raise_error_if_model_is_wrong_type() None
        }
  • mac.inference.creation.InferenceBuilder builds a concrete Inference, i.e. instantiates an Inference object, loads the appropriate model and assigns the model to to the Inference object.

    classDiagram
        class InferenceBuilder {
            +create(config InferenceConfig) * Inference
            -inference()* Any
        }

mac.inference.Inference

mac.inference.Inference Objects
ObjectTypeDescription
model (Required)[Any]One or more objects that match the expected_model_types. This can be a ML Model (for inference use), a string (for data conversion), etc. See Arbitrary Python Examples for examples.
mac.inference.Inference Methods
MethodReturnsDescription
expected_model_types (Required)SetReturns a Set of models expected for the inference as defined by the developer. Typically this is a set of one. Wallaroo checks the expected model types to verify that the model submitted through the InferenceBuilder method matches what this Inference class expects.
_predict (input_data: mac.types.InferenceData) (Required)mac.types.InferenceDataThe entry point for the Wallaroo inference with the following input and output parameters that are defined when the model is updated.
  • mac.types.InferenceData: The input InferenceData is a Dictionary of numpy arrays derived from the input_schema detailed when the model is uploaded, defined in PyArrow.Schema format.
  • mac.types.InferenceData: The output is a Dictionary of numpy arrays as defined by the output parameters defined in PyArrow.Schema format.
The InferenceDataValidationError exception is raised when the input data does not match mac.types.InferenceData.
raise_error_if_model_is_not_assignedN/AError when a model is not set to Inference.
raise_error_if_model_is_wrong_typeN/AError when the model does not match the expected_model_types.

The example, the expected_model_types can be defined for the KMeans model.

from sklearn.cluster import KMeans

class SampleClass(mac.inference.Inference):
    @property
    def expected_model_types(self) -> Set[Any]:
        return {KMeans}

mac.inference.creation.InferenceBuilder

InferenceBuilder builds a concrete Inference, i.e. instantiates an Inference object, loads the appropriate model and assigns the model to the Inference.

classDiagram
    class InferenceBuilder {
        +create(config InferenceConfig) * Inference
        -inference()* Any
    }

Each model that is included requires its own InferenceBuilder. InferenceBuilder loads one model, then submits it to the Inference class when created. The Inference class checks this class against its expected_model_types() Set.

mac.inference.creation.InferenceBuilder Methods
MethodReturnsDescription
create(config mac.config.inference.CustomInferenceConfig) (Required)The custom Inference instance.Creates an Inference subclass, then assigns a model and attributes. The CustomInferenceConfig is used to retrieve the config.model_path, which is a pathlib.Path object pointing to the folder where the model artifacts are saved. Every artifact loaded must be relative to config.model_path. This is set when the arbitrary python .zip file is uploaded and the environment for running it in Wallaroo is set. For example: loading the artifact vgg_clustering\feature_extractor.h5 would be set with config.model_path \ feature_extractor.h5. The model loaded must match an existing module. For our example, this is from sklearn.cluster import KMeans, and this must match the Inference expected_model_types.
inferencecustom Inference instance.Returns the instantiated custom Inference object created from the create method.

Arbitrary Python Runtime

Arbitrary Python always run in the containerized model runtime.

ParameterDescription
Web Sitehttps://mlflow.org
Supported Librariesmlflow==1.3.0
RuntimeContainerized aka mlflow

For models that do not fall under the supported model frameworks, organizations can use containerized MLFlow ML Models.

This guide details how to add ML Models from a model registry service into Wallaroo.

Wallaroo supports both public and private containerized model registries. See the Wallaroo Private Containerized Model Container Registry Guide for details on how to configure a Wallaroo instance with a private model registry.

Wallaroo users can register their trained MLFlow ML Models from a containerized model container registry into their Wallaroo instance and perform inferences with it through a Wallaroo pipeline.

As of this time, Wallaroo only supports MLFlow 1.30.0 containerized models. For information on how to containerize an MLFlow model, see the MLFlow Documentation.

Wallaroo supports both public and private containerized model registries. See the Wallaroo Private Containerized Model Container Registry Guide for details on how to configure a Wallaroo instance with a private model registry.

List Wallaroo Frameworks

Wallaroo frameworks are listed from the Wallaroo.Framework class. The following demonstrates listing all available supported frameworks.

from wallaroo.framework import Framework

[e.value for e in Framework]

    ['onnx',
    'tensorflow',
    'python',
    'keras',
    'sklearn',
    'pytorch',
    'xgboost',
    'hugging-face-feature-extraction',
    'hugging-face-image-classification',
    'hugging-face-image-segmentation',
    'hugging-face-image-to-text',
    'hugging-face-object-detection',
    'hugging-face-question-answering',
    'hugging-face-stable-diffusion-text-2-img',
    'hugging-face-summarization',
    'hugging-face-text-classification',
    'hugging-face-translation',
    'hugging-face-zero-shot-classification',
    'hugging-face-zero-shot-image-classification',
    'hugging-face-zero-shot-object-detection',
    'hugging-face-sentiment-analysis',
    'hugging-face-text-generation']

Registry Services Roles

Registry service use in Wallaroo typically falls under the following roles.

RoleRecommended ActionsDescription
DevOps EngineerCreate Model RegistryCreate the model (AKA artifact) registry service
 Retrieve Model Registry TokensGenerate the model registry service credentials.
MLOps EngineerConnect Model Registry to WallarooAdd the Registry Service URL and credentials into a Wallaroo instance for use by other users and scripts.
 Add Wallaroo Registry Service to WorkspaceAdd the registry service configuration to a Wallaroo workspace for use by workspace users.
 Get Registry DetailsRetrieve the connection details for a Wallaroo registry.
 Remove Wallaroo Registry from a WorkspaceAdd the registry service configuration to a Wallaroo workspace for use by workspace users.
Data ScientistList Models in RegistryList available models in a model registry.
 List Model Version ArtifactsRetrieve the artifacts (usually files) for a model stored in a model registry.
 Upload Model from RegistryUpload a model and artifacts stored in a model registry into a Wallaroo workspace.

Model Registry Operations

The following links to guides and information on setting up a model registry (also known as an artifact registry).

Create Model Registry

See Model serving with Azure Databricks for setting up a model registry service using Azure Databricks.

The following steps create an Access Token used to authenticate to an Azure Databricks Model Registry.

  1. Log into the Azure Databricks workspace.
  2. From the upper right corner access the User Settings.
  3. From the Access tokens, select Generate new token.
  4. Specify any token description and lifetime. Once complete, select Generate.
  5. Copy the token and store in a secure place. Once the Generate New Token module is closed, the token will not be retrievable.
Retrieve Azure Databricks User Token

The MLflow Model Registry provides a method of setting up a model registry service. Full details can be found at the MLflow Registry Quick Start Guide.

A generic MLFlow model registry requires no token.

Wallaroo Registry Operations

  • Connect Model Registry to Wallaroo: This details the link and connection information to a existing MLFlow registry service. Note that this does not create a MLFlow registry service, but adds the connection and credentials to Wallaroo to allow that MLFlow registry service to be used by other entities in the Wallaroo instance.
  • Add a Registry to a Workspace: Add the created Wallaroo Model Registry so make it available to other workspace members.
  • Remove a Registry from a Workspace: Remove the link between a Wallaroo Model Registry and a Wallaroo workspace.

Connect Model Registry to Wallaroo

MLFlow Registry connection information is added to a Wallaroo instance through the following endpoint.

  • REQUEST URL
    • v1/api/models/create_registry
  • PARAMETERS
    • workspace_id (Integer Required): The numerical ID of the workspace to create the registry in.
    • name (String Required): The name for the registry. Registry names are not unique.
    • url (String Required): The full URL of the registry service. For example: https://registry.wallaroo.ai
    • token (String Required): The authentication token used by the registry service.
  • RETURNS
    • id (String): The UUID of the registry.
    • workspace_id: The numerical ID of the workspace the registry was connected to.

Connect Model Registry to Wallaroo Example

The following registry will be added to the workspace with the id 1.

import requests
token = "abcdefg"

x = requests.post("https://{APIURL}/v1/api/models/create_registry", 
                    json={
                        "workspace_id": 1, 
                        "name": "sample registry", 
                        "url": "https://registry.wallaroo.ai", 
                        "token": token
                        }, 
                        headers=wl.auth.auth_header()
                    )

{'id': '98f9ca1d-c4e7-4d70-8df4-05c25a64be29', 'workspace_id': 1}

Add Wallaroo Registry Service to Workspace

Registries are assigned to a Wallaroo workspace with the following endpoint. This allows members of the workspace to access the registry connection. A registry can be associated with one or more workspaces.

  • REQUEST URL
    • v1/api/models//v1/api/models/attach_registry_to_workspace
  • PARAMETERS
    • workspace_id (Integer Required): The numerical ID of the workspace to create the registry in.
    • registry_id (String Required): The ID of the registry in UUID format.
  • RETURNS
    • id (String): The UUID of the registry.
    • workspace_id: The numerical ID of the workspace the registry was connected to.

Add Wallaroo Registry Service to Workspace Example

import requests
token = "abcdefg"

x = requests.post("https://{APIURL}/v1/api/models/attach_registry_to_workspace", 
                    json={
                      'id': '98f9ca1d-c4e7-4d70-8df4-05c25a64be29', 
                      'workspace_id': 1},
                        headers=wl.auth.auth_header()
                    )

{
  "registry_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
  "workspace_id": 0
}

Remove Wallaroo Registry from a Workspace

Registries are removed from a registry with the following endpoint. This does not remove the registry connection information from the Wallaroo instance, merely removes the association between the registry and that particular workspace.

  • REQUEST URL
    • v1/api/models//v1/api/models/remove_registry_from_workspace
  • PARAMETERS
    • workspace_id (Integer Required): The numerical ID of the workspace to remove the registry from.
    • registry_id (String Required): The ID of the registry in UUID format.
  • RETURNS
    • id (String): The UUID of the registry.
    • workspace_id: The numerical ID of the workspace the registry was connected to.

Remove Wallaroo Registry from a Workspace Example

import requests
token = "abcdefg"

x = requests.post("https://{APIURL}/v1/api/models/remove_registry_from_workspace", 
                    json={
                      'id': '98f9ca1d-c4e7-4d70-8df4-05c25a64be29', 
                      'workspace_id': 1
                      },
                        headers=wl.auth.auth_header()
                    )

{
  "registry_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
  "workspace_id": 0
}

Get Registry Details

  • REQUEST URL
    • v1/api/models/get_registry
  • PARAMETERS
    • id (String Required): The registry ID in UUID format.
  • RETURNS
    • id (String): The UUID of the registry.
    • name (String Required): The name for the registry. Registry names are not unique.
    • url (String Required): The full URL of the registry service. For example: https://registry.wallaroo.ai
    • token (String Required): The authentication token used by the registry service.
    • created_at (String Required): The creation date in DateTime format.
    • updated_at (String Required): The updated date in DateTime format.

Get Registry Details Example

The following example demonstrates retrieving the registry with id 98f9ca1d-c4e7-4d70-8df4-05c25a64be29.

import requests
x = requests.post("https://{APIURL}/v1/api/models/get_registry", 
                    json={
                        "registry_id": '98f9ca1d-c4e7-4d70-8df4-05c25a64be29'
                        }, 
                    headers=wl.auth.auth_header()
                )
{
    'id': '98f9ca1d-c4e7-4d70-8df4-05c25a64be29',
    'name': 'sample registry',
    'url': 'https://registry.wallaroo.ai',
    'token': 'dapi67c8c0b04606f730e78b7ae5e3221015-3',
    'created_at': '2023-06-23T15:37:38.38427+00:00',
    'updated_at': '2023-06-23T15:37:38.38427+00:00'
}

Wallaroo Registry Model Operations

List Registries in Workspace

A list of registries associates with a specific workspace are retrieved from the following endpoint.

  • REQUEST URL
    • POST /v1/api/models/list_registries
  • PARAMETERS
    • workspace-id (String Required): The numerical id of the workspace.
  • RETURNS
    • A list of registries with the following fields.
      • created_at (String): The UTC DateTime stamp of when the registry was created.
      • id (String): The id of the Wallaroo registry in UUID format.
      • name (String): The assigned name of the registry.
      • token (String): The authentication token to the model registry.
      • updated_at (String): The UTC DateTime stamp of when the registry was last updated.
      • url (String): The URL to the registry service.

List Registries in Workspace Example

import requests
x = requests.post("{APIURL}v1/api/models/list_registries", 
                    json={
                      "workspace_id": 1
                      }, 
                      headers=wl.auth.auth_header()
                      )
x.json()

[
  {
    "created_at": "2023-07-11T15:21:24.403Z",
    "id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
    "name": "sample registry",
    "token": "string",
    "updated_at": "2023-07-11T15:21:24.403Z",
    "url": "string"
  }
]

List Models in a Registry

A List of models available to the Wallaroo instance through the MLFlow Registry is performed with the following endpoint.

  • REQUEST URL
    • v1/api/models/list_registry_models
  • PARAMETERS
    • id (String Required): The registry ID in UUID format.
  • RETURNS
    • next_page_token (String): Used to retrieve the next List of models. If None, there are no additional models to list after this request.
    • registered_models (List): List of Models based on the mlflow.entities.model_registry.ModelVersion specification.
      • name (String): The name of the model.
      • user_id (String): The ID of the user.
      • latest_versions (List): The list of other versions of the model. If there are no other versions of the model, this will be None. Each version has the following fields.
        • name (String): The name of the artifact.
        • description (String): The description of the artifact.
        • version (Integer): The version number. Other versions may be removed from the registry, so it is possible for a version to be higher than 1 even if there are no other versions still stored in the registry.
        • status (String): The current status of the model.
        • run_id (String): The run id from the MLFlow service that created the model.
        • run_link (String): Link to the run from the MLFlow service that created the model.
        • source (String): The URL for the specific version artifact on this registry service. For example: 'dbfs:/databricks/mlflow-tracking/123456/abcdefg/artifacts/random_forest_model'.
        • current_stage (String): The current stage of the model version.
        • creation_timestamp (Integer): Creation timestamp in milliseconds since the Unix epoch.
        • last_updated_timestamp (Integer): Last updated timestamp in milliseconds since the Unix epoch.

List Models in a Registry Example

import requests
x = requests.post("{APIURL}v1/api/models/list_registry_models", 
                    json={
                      "registry_id": id
                      }, 
                      headers=wl.auth.auth_header()
                      )
x.json()

{
    'next_page_token': None,
    'registered_models': [
        {
            'name': 'testmodel', 
            'user_id': 'sample.usersj@wallaroo.ai', 
            'latest_versions': None, 
            'creation_timestamp': 1686940722329, 
            'last_updated_timestamp': 1686940722329
        },
        {
            'name': 'testmodel2', 
            'user_id': 'sample.user@wallaroo.ai', 
            'latest_versions': None, 
            'creation_timestamp': 1686940864528, 
            'last_updated_timestamp': 1686940864528
        },
        {
            'name': 'wine_quality', 
            'user_id': 'sample.user@wallaroo.ai', 
            'latest_versions': [
                {
                    'name': 'wine_quality', 
                    'description': None, 
                    'version': '1', 
                    'status': 'READY', 
                    'run_id': 'abcdefg', 
                    'run_link': None, 
                    'source': 'dbfs:/databricks/mlflow-tracking/abcdefg/abcdefg/artifacts/random_forest_model', 
                    'current_stage': 'Archived', 
                    'creation_timestamp': 1686942353367, 
                    'last_updated_timestamp': 1686942597509
                },
                {
                    'name': 'wine_quality', 
                    'description': None, 
                    'version': '2', 
                    'status': 'READY', 
                    'run_id': 'abcdefg', 
                    'run_link': None, 
                    'source': 'dbfs:/databricks/mlflow-tracking/abcdefg/abcdefg/artifacts/model', 
                    'current_stage': 'Production', 
                    'creation_timestamp': 1686942576120, 
                    'last_updated_timestamp': 1686942597646
                }
            ], 
            'creation_timestamp': 1686942353127, 
            'last_updated_timestamp': 1686942597646
        }
    ]
}

List Model Version Artifacts

The artifacts of a specific model version are retrieved through the following endpoint.

  • REQUEST URL
    • v1/api/models/list_registry_model_version_artifacts
  • PARAMETERS
    • registry_id (String Required): The registry ID in UUID format.
    • name (String Required): The name of the model to retrieve artifacts from.
    • version (String Required): The version of the model to retrieve artifacts from.
  • RETURNS
    • List of artifacts with the following fields.
      • file_size (Integer): The size of the file in bytes.
      • full_path (String): The full path to the artifact. For example: https://adb-5939996465837398.18.azuredatabricks.net/api/2.0/dbfs/read?path=/databricks/mlflow-registry/9f38797c1dbf4e7eb229c4011f0f1f18/models/testmodel2/model.pkl
      • is_dir (Boolean): Whether the artifact is a directory or file.
      • modification_time (Integer): Last modification timestamp in milliseconds since the Unix epoch
      • path (String): Relative path to the artifact within the registry. For example: /databricks/mlflow-registry/9f38797c1dbf4e7eb229c4011f0f1f18/models/testmodel2/model.pkl

List Model Version Artifacts Example

import requests
x = requests.post("{APIURL}v1/api/models//list_registry_model_version_artifacts", 
                    json={
                      "name": "wine_quality",
                      "registry_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
                      "version": "2"
                      }, 
                      headers=wl.auth.auth_header()
                      )
x.json()

[
  {
    "file_size": 156456,
    "full_path": "https://adb-5939996465837398.18.azuredatabricks.net/api/2.0/dbfs/read?path=/databricks/mlflow-registry/9f38797c1dbf4e7eb229c4011f0f1f18/models/testmodel2/model.pkl",
    "is_dir": false,
    "modification_time": 1686942597646,
    "path": "/databricks/mlflow-registry/9f38797c1dbf4e7eb229c4011f0f1f18/models/testmodel2/model.pkl"
  }
]

Upload Model from Registry

Models are uploaded from a model registry configured in Wallaroo through the following endpoint. The specific artifact that is the model to be deployed is the item to upload to the Wallaroo workspace. Models must comply with Wallaroo model framework and versions as defined in Artifact Requirements.

  • REQUEST URL
    • v1/api/models/upload_from_registry
  • PARAMETERS
    • registry_id (String Required): The registry ID in UUID format.
    • name (String Required): The name to assign the model in Wallaroo. Model names map onto Kubernetes objects, and must be DNS compliant. The strings for model names must be ASCII alpha-numeric characters or dash (-) only. . and _ are not allowed.
    • path (String Required): URL of the model to upload as specified in List Models in a Registry source field.
    • visibility (String Required): Whether the model is public or private.
    • workspace_id (Integer Required): The numerical id of the workspace to upload the model to.
    • conversion (String Required): The conversion parameters that include the following:
      • framework (String Required): The framework of the model being uploaded. See the list of supported models for more details.
      • python_version (String Required): The version of Python required for model.
      • requirements (String Required): Required libraries. Can be [] if the requirements are default Wallaroo JupyterHub libraries.
      • input_schema (String Optional): The input schema from the Apache Arrow pyarrow.lib.Schema format, encoded with base64.b64encode. Only required for non-native runtime models.
      • output_schema (String Optional): The output schema from the Apache Arrow pyarrow.lib.Schema format, encoded with base64.b64encode. Only required for non-native runtime models.
  • RETURNS
    • model_id (String): The numerical id of the model uploaded

Upload Model from Registry Example

import requests
x = requests.post("{APIURL}v1/api/models/upload_from_registry", json={
    "registry_id": id, 
    "name": "uploaded-model-name",
    "path": "<DBFS URL from list_artifacts() call here>",
    "visibility": "public",
    "workspace_id": 1,
    "conversion": {
        "framework": "sklearn",
        "requirements": [],
    },
    "input_schema": "<base64-encoded input schema here>",
    "output_schema": "<base64-encoded output schema here>"
}, headers=wl.auth.auth_header())
x.json()

{'model_id': 34}

5 - Wallaroo MLOps API Essentials Guide: Model Upload and Registrations

How to use the Wallaroo API to upload models of different frameworks.

Models are uploaded or registered to a Wallaroo workspace depending on the model framework and type.

Supported Models

The following frameworks are supported. Frameworks fall under either Native or Containerized runtimes in the Wallaroo engine. For more details, see the specific framework what runtime a specific model framework runs in.

Runtime DisplayModel Runtime SpacePipeline Configuration
tensorflowNativeNative Runtime Configuration Methods
onnxNativeNative Runtime Configuration Methods
pythonNativeNative Runtime Configuration Methods
mlflowContainerizedContainerized Runtime Deployment
flightContainerizedContainerized Runtime Deployment

Please note the following.

Wallaroo ONNX Requirements

Wallaroo natively supports Open Neural Network Exchange (ONNX) models into the Wallaroo engine.

ParameterDescription
Web Sitehttps://onnx.ai/
Supported LibrariesSee table below.
FrameworkFramework.ONNX aka onnx
RuntimeNative aka onnx

The following ONNX versions models are supported:

Wallaroo VersionONNX VersionONNX IR VersionONNX OPset VersionONNX ML Opset Version
2023.4.0 (October 2023)1.12.18173
2023.2.1 (July 2023)1.12.18173

For the most recent release of Wallaroo 2023.4.0, the following native runtimes are supported:

  • If converting another ML Model to ONNX (PyTorch, XGBoost, etc) using the onnxconverter-common library, the supported DEFAULT_OPSET_NUMBER is 17.

Using different versions or settings outside of these specifications may result in inference issues and other unexpected behavior.

ONNX models always run in the Wallaroo Native Runtime space.

Data Schemas

ONNX models deployed to Wallaroo have the following data requirements.

  • Equal rows constraint: The number of input rows and output rows must match.
  • All inputs are tensors: The inputs are tensor arrays with the same shape.
  • Data Type Consistency: Data types within each tensor are of the same type.

Equal Rows Constraint

Inference performed through ONNX models are assumed to be in batch format, where each input row corresponds to an output row. This is reflected in the in fields returned for an inference. In the following example, each input row for an inference is related directly to the inference output.

df = pd.read_json('./data/cc_data_1k.df.json')
display(df.head())

result = ccfraud_pipeline.infer(df.head())
display(result)

INPUT

 tensor
0[-1.0603297501, 2.3544967095000002, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192000001, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526000001, 1.9870535692, 0.7005485718000001, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756, -0.1466244739, -1.4463212439]
1[-1.0603297501, 2.3544967095000002, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192000001, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526000001, 1.9870535692, 0.7005485718000001, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756, -0.1466244739, -1.4463212439]
2[-1.0603297501, 2.3544967095000002, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192000001, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526000001, 1.9870535692, 0.7005485718000001, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756, -0.1466244739, -1.4463212439]
3[-1.0603297501, 2.3544967095000002, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192000001, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526000001, 1.9870535692, 0.7005485718000001, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756, -0.1466244739, -1.4463212439]
4[0.5817662108, 0.09788155100000001, 0.1546819424, 0.4754101949, -0.19788623060000002, -0.45043448540000003, 0.016654044700000002, -0.0256070551, 0.0920561602, -0.2783917153, 0.059329944100000004, -0.0196585416, -0.4225083157, -0.12175388770000001, 1.5473094894000001, 0.2391622864, 0.3553974881, -0.7685165301, -0.7000849355000001, -0.1190043285, -0.3450517133, -1.1065114108, 0.2523411195, 0.0209441826, 0.2199267436, 0.2540689265, -0.0450225094, 0.10867738980000001, 0.2547179311]

OUTPUT

 timein.tensorout.dense_1check_failures
02023-11-17 20:34:17.005[-1.0603297501, 2.3544967095, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526, 1.9870535692, 0.7005485718, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756, -0.1466244739, -1.4463212439][0.99300325]0
12023-11-17 20:34:17.005[-1.0603297501, 2.3544967095, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526, 1.9870535692, 0.7005485718, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756, -0.1466244739, -1.4463212439][0.99300325]0
22023-11-17 20:34:17.005[-1.0603297501, 2.3544967095, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526, 1.9870535692, 0.7005485718, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756, -0.1466244739, -1.4463212439][0.99300325]0
32023-11-17 20:34:17.005[-1.0603297501, 2.3544967095, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526, 1.9870535692, 0.7005485718, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756, -0.1466244739, -1.4463212439][0.99300325]0
42023-11-17 20:34:17.005[0.5817662108, 0.097881551, 0.1546819424, 0.4754101949, -0.1978862306, -0.4504344854, 0.0166540447, -0.0256070551, 0.0920561602, -0.2783917153, 0.0593299441, -0.0196585416, -0.4225083157, -0.1217538877, 1.5473094894, 0.2391622864, 0.3553974881, -0.7685165301, -0.7000849355, -0.1190043285, -0.3450517133, -1.1065114108, 0.2523411195, 0.0209441826, 0.2199267436, 0.2540689265, -0.0450225094, 0.1086773898, 0.2547179311][0.0010916889]0

All Inputs Are Tensors

All inputs into an ONNX model must be tensors. This requires that the shape of each element is the same. For example, the following is a proper input:

t [
    [2.35, 5.75],
    [3.72, 8.55],
    [5.55, 97.2]
]
Standard tensor array

Another example is a 2,2,3 tensor, where the shape of each element is (3,), and each element has 2 rows.

t = [
        [2.35, 5.75, 19.2],
        [3.72, 8.55, 10.5]
    ],
    [
        [5.55, 7.2, 15.7],
        [9.6, 8.2, 2.3]
    ]

In this example each element has a shape of (2,). Tensors with elements of different shapes, known as ragged tensors, are not supported. For example:

t = [
    [2.35, 5.75],
    [3.72, 8.55, 10.5],
    [5.55, 97.2]
])

**INVALID SHAPE**
Ragged tensor array - unsupported

For models that require ragged tensor or other shapes, see other data formatting options such as Bring Your Own Predict models.

Data Type Consistency

All inputs into an ONNX model must have the same internal data type. For example, the following is valid because all of the data types within each element are float32.

t = [
    [2.35, 5.75],
    [3.72, 8.55],
    [5.55, 97.2]
]

The following is invalid, as it mixes floats and strings in each element:

t = [
    [2.35, "Bob"],
    [3.72, "Nancy"],
    [5.55, "Wani"]
]

The following inputs are valid, as each data type is consistent within the elements.

df = pd.DataFrame({
    "t": [
        [2.35, 5.75, 19.2],
        [5.55, 7.2, 15.7],
    ],
    "s": [
        ["Bob", "Nancy", "Wani"],
        ["Jason", "Rita", "Phoebe"]
    ]
})
df
 ts
0[2.35, 5.75, 19.2][Bob, Nancy, Wani]
1[5.55, 7.2, 15.7][Jason, Rita, Phoebe]
ParameterDescription
Web Sitehttps://www.tensorflow.org/
Supported Librariestensorflow==2.9.3
FrameworkFramework.TENSORFLOW aka tensorflow
RuntimeNative aka tensorflow
Supported File TypesSavedModel format as .zip file

TensorFlow File Format

TensorFlow models are .zip file of the SavedModel format. For example, the Aloha sample TensorFlow model is stored in the directory alohacnnlstm:

├── saved_model.pb
└── variables
    ├── variables.data-00000-of-00002
    ├── variables.data-00001-of-00002
    └── variables.index

This is compressed into the .zip file alohacnnlstm.zip with the following command:

zip -r alohacnnlstm.zip alohacnnlstm/

ML models that meet the Tensorflow and SavedModel format will run as Wallaroo Native runtimes by default.

See the SavedModel guide for full details.

ParameterDescription
Web Sitehttps://www.python.org/
Supported Librariespython==3.8
FrameworkFramework.PYTHON aka python
RuntimeNative aka python

Python models uploaded to Wallaroo are executed as a native runtime.

Note that Python models - aka “Python steps” - are standalone python scripts that use the python libraries natively supported by the Wallaroo platform. These are used for either simple model deployment (such as ARIMA Statsmodels), or data formatting such as the postprocessing steps. A Wallaroo Python model will be composed of one Python script that matches the Wallaroo requirements.

This is contrasted with Arbitrary Python models, also known as Bring Your Own Predict (BYOP) allow for custom model deployments with supporting scripts and artifacts. These are used with pre-trained models (PyTorch, Tensorflow, etc) along with whatever supporting artifacts they require. Supporting artifacts can include other Python modules, model files, etc. These are zipped with all scripts, artifacts, and a requirements.txt file that indicates what other Python models need to be imported that are outside of the typical Wallaroo platform.

Python Models Requirements

Python models uploaded to Wallaroo are Python scripts that must include the wallaroo_json method as the entry point for the Wallaroo engine to use it as a Pipeline step.

This method receives the results of the previous Pipeline step, and its return value will be used in the next Pipeline step.

If the Python model is the first step in the pipeline, then it will be receiving the inference request data (for example: a preprocessing step). If it is the last step in the pipeline, then it will be the data returned from the inference request.

In the example below, the Python model is used as a post processing step for another ML model. The Python model expects to receive data from a ML Model who’s output is a DataFrame with the column dense_2. It then extracts the values of that column as a list, selects the first element, and returns a DataFrame with that element as the value of the column output.

def wallaroo_json(data: pd.DataFrame):
    print(data)
    return [{"output": [data["dense_2"].to_list()[0][0]]}]

In line with other Wallaroo inference results, the outputs of a Python step that returns a pandas DataFrame or Arrow Table will be listed in the out. metadata, with all inference outputs listed as out.{variable 1}, out.{variable 2}, etc. In the example above, this results the output field as the out.output field in the Wallaroo inference result.

 timein.tensorout.outputcheck_failures
02023-06-20 20:23:28.395[0.6878518042, 0.1760734021, -0.869514083, 0.3..[12.886651039123535]0
ParameterDescription
Web Sitehttps://huggingface.co/models
Supported Libraries
  • transformers==4.34.1
  • diffusers==0.14.0
  • accelerate==0.23.0
  • torchvision==0.14.1
  • torch==1.13.1
FrameworksThe following Hugging Face pipelines are supported by Wallaroo.
  • Framework.HUGGING_FACE_FEATURE_EXTRACTION aka hugging-face-feature-extraction
  • Framework.HUGGING_FACE_IMAGE_CLASSIFICATION aka hugging-face-image-classification
  • Framework.HUGGING_FACE_IMAGE_SEGMENTATION aka hugging-face-image-segmentation
  • Framework.HUGGING_FACE_IMAGE_TO_TEXT aka hugging-face-image-to-text
  • Framework.HUGGING_FACE_OBJECT_DETECTION aka hugging-face-object-detection
  • Framework.HUGGING_FACE_QUESTION_ANSWERING aka hugging-face-question-answering
  • Framework.HUGGING_FACE_STABLE_DIFFUSION_TEXT_2_IMG aka hugging-face-stable-diffusion-text-2-img
  • Framework.HUGGING_FACE_SUMMARIZATION aka hugging-face-summarization
  • Framework.HUGGING_FACE_TEXT_CLASSIFICATION aka hugging-face-text-classification
  • Framework.HUGGING_FACE_TRANSLATION aka hugging-face-translation
  • Framework.HUGGING_FACE_ZERO_SHOT_CLASSIFICATION aka hugging-face-zero-shot-classification
  • Framework.HUGGING_FACE_ZERO_SHOT_IMAGE_CLASSIFICATION aka hugging-face-zero-shot-image-classification
  • Framework.HUGGING_FACE_ZERO_SHOT_OBJECT_DETECTION aka hugging-face-zero-shot-object-detection
  • Framework.HUGGING_FACE_SENTIMENT_ANALYSIS aka hugging-face-sentiment-analysis
  • Framework.HUGGING_FACE_TEXT_GENERATION aka hugging-face-text-generation
  • Framework.HUGGING_FACE_AUTOMATIC_SPEECH_RECOGNITION aka hugging-face-automatic-speech-recognition
RuntimeContainerized flight

During the model upload process, the Wallaroo instance will attempt to convert the model to a Native Wallaroo Runtime. If unsuccessful based , it will create a Wallaroo Containerized Runtime for the model. See the model deployment section for details on how to configure pipeline resources based on the model’s runtime.

Hugging Face Schemas

Input and output schemas for each Hugging Face pipeline are defined below. Note that adding additional inputs not specified below will raise errors, except for the following:

  • Framework.HUGGING_FACE_IMAGE_TO_TEXT
  • Framework.HUGGING_FACE_TEXT_CLASSIFICATION
  • Framework.HUGGING_FACE_SUMMARIZATION
  • Framework.HUGGING_FACE_TRANSLATION

Additional inputs added to these Hugging Face pipelines will be added as key/pair value arguments to the model’s generate method. If the argument is not required, then the model will default to the values coded in the original Hugging Face model’s source code.

See the Hugging Face Pipeline documentation for more details on each pipeline and framework.

Wallaroo FrameworkReference
Framework.HUGGING_FACE_FEATURE_EXTRACTION

Schemas:

input_schema = pa.schema([
    pa.field('inputs', pa.string())
])
output_schema = pa.schema([
    pa.field('output', pa.list_(
        pa.list_(
            pa.float64(),
            list_size=128
        ),
    ))
])
Wallaroo FrameworkReference
Framework.HUGGING_FACE_IMAGE_CLASSIFICATION

Schemas:

input_schema = pa.schema([
    pa.field('inputs', pa.list_(
        pa.list_(
            pa.list_(
                pa.int64(),
                list_size=3
            ),
            list_size=100
        ),
        list_size=100
    )),
    pa.field('top_k', pa.int64()),
])

output_schema = pa.schema([
    pa.field('score', pa.list_(pa.float64(), list_size=2)),
    pa.field('label', pa.list_(pa.string(), list_size=2)),
])
Wallaroo FrameworkReference
Framework.HUGGING_FACE_IMAGE_SEGMENTATION

Schemas:

input_schema = pa.schema([
    pa.field('inputs', 
        pa.list_(
            pa.list_(
                pa.list_(
                    pa.int64(),
                    list_size=3
                ),
                list_size=100
            ),
        list_size=100
    )),
    pa.field('threshold', pa.float64()),
    pa.field('mask_threshold', pa.float64()),
    pa.field('overlap_mask_area_threshold', pa.float64()),
])

output_schema = pa.schema([
    pa.field('score', pa.list_(pa.float64())),
    pa.field('label', pa.list_(pa.string())),
    pa.field('mask', 
        pa.list_(
            pa.list_(
                pa.list_(
                    pa.int64(),
                    list_size=100
                ),
                list_size=100
            ),
    )),
])
Wallaroo FrameworkReference
Framework.HUGGING_FACE_IMAGE_TO_TEXT

Any parameter that is not part of the required inputs list will be forwarded to the model as a key/pair value to the underlying models generate method. If the additional input is not supported by the model, an error will be returned.

Schemas:

input_schema = pa.schema([
    pa.field('inputs', pa.list_( #required
        pa.list_(
            pa.list_(
                pa.int64(),
                list_size=3
            ),
            list_size=100
        ),
        list_size=100
    )),
    # pa.field('max_new_tokens', pa.int64()),  # optional
])

output_schema = pa.schema([
    pa.field('generated_text', pa.list_(pa.string())),
])
Wallaroo FrameworkReference
Framework.HUGGING_FACE_OBJECT_DETECTION

Schemas:

input_schema = pa.schema([
    pa.field('inputs', 
        pa.list_(
            pa.list_(
                pa.list_(
                    pa.int64(),
                    list_size=3
                ),
                list_size=100
            ),
        list_size=100
    )),
    pa.field('threshold', pa.float64()),
])

output_schema = pa.schema([
    pa.field('score', pa.list_(pa.float64())),
    pa.field('label', pa.list_(pa.string())),
    pa.field('box', 
        pa.list_( # dynamic output, i.e. dynamic number of boxes per input image, each sublist contains the 4 box coordinates 
            pa.list_(
                    pa.int64(),
                    list_size=4
                ),
            ),
    ),
])
Wallaroo FrameworkReference
Framework.HUGGING_FACE_QUESTION_ANSWERING

Schemas:

input_schema = pa.schema([
    pa.field('question', pa.string()),
    pa.field('context', pa.string()),
    pa.field('top_k', pa.int64()),
    pa.field('doc_stride', pa.int64()),
    pa.field('max_answer_len', pa.int64()),
    pa.field('max_seq_len', pa.int64()),
    pa.field('max_question_len', pa.int64()),
    pa.field('handle_impossible_answer', pa.bool_()),
    pa.field('align_to_words', pa.bool_()),
])

output_schema = pa.schema([
    pa.field('score', pa.float64()),
    pa.field('start', pa.int64()),
    pa.field('end', pa.int64()),
    pa.field('answer', pa.string()),
])
Wallaroo FrameworkReference
Framework.HUGGING_FACE_STABLE_DIFFUSION_TEXT_2_IMG

Schemas:

input_schema = pa.schema([
    pa.field('prompt', pa.string()),
    pa.field('height', pa.int64()),
    pa.field('width', pa.int64()),
    pa.field('num_inference_steps', pa.int64()), # optional
    pa.field('guidance_scale', pa.float64()), # optional
    pa.field('negative_prompt', pa.string()), # optional
    pa.field('num_images_per_prompt', pa.string()), # optional
    pa.field('eta', pa.float64()) # optional
])

output_schema = pa.schema([
    pa.field('images', pa.list_(
        pa.list_(
            pa.list_(
                pa.int64(),
                list_size=3
            ),
            list_size=128
        ),
        list_size=128
    )),
])
Wallaroo FrameworkReference
Framework.HUGGING_FACE_SUMMARIZATION

Any parameter that is not part of the required inputs list will be forwarded to the model as a key/pair value to the underlying models generate method. If the additional input is not supported by the model, an error will be returned.

Schemas:

input_schema = pa.schema([
    pa.field('inputs', pa.string()),
    pa.field('return_text', pa.bool_()),
    pa.field('return_tensors', pa.bool_()),
    pa.field('clean_up_tokenization_spaces', pa.bool_()),
    # pa.field('extra_field', pa.int64()), # every extra field you specify will be forwarded as a key/value pair
])

output_schema = pa.schema([
    pa.field('summary_text', pa.string()),
])
Wallaroo FrameworkReference
Framework.HUGGING_FACE_TEXT_CLASSIFICATION

Schemas

input_schema = pa.schema([
    pa.field('inputs', pa.string()), # required
    pa.field('top_k', pa.int64()), # optional
    pa.field('function_to_apply', pa.string()), # optional
])

output_schema = pa.schema([
    pa.field('label', pa.list_(pa.string(), list_size=2)), # list with a number of items same as top_k, list_size can be skipped but may lead in worse performance
    pa.field('score', pa.list_(pa.float64(), list_size=2)), # list with a number of items same as top_k, list_size can be skipped but may lead in worse performance
])
Wallaroo FrameworkReference
Framework.HUGGING_FACE_TRANSLATION

Any parameter that is not part of the required inputs list will be forwarded to the model as a key/pair value to the underlying models generate method. If the additional input is not supported by the model, an error will be returned.

Schemas:

input_schema = pa.schema([
    pa.field('inputs', pa.string()), # required
    pa.field('return_tensors', pa.bool_()), # optional
    pa.field('return_text', pa.bool_()), # optional
    pa.field('clean_up_tokenization_spaces', pa.bool_()), # optional
    pa.field('src_lang', pa.string()), # optional
    pa.field('tgt_lang', pa.string()), # optional
    # pa.field('extra_field', pa.int64()), # every extra field you specify will be forwarded as a key/value pair
])

output_schema = pa.schema([
    pa.field('translation_text', pa.string()),
])
Wallaroo FrameworkReference
Framework.HUGGING_FACE_ZERO_SHOT_CLASSIFICATION

Schemas:

input_schema = pa.schema([
    pa.field('inputs', pa.string()), # required
    pa.field('candidate_labels', pa.list_(pa.string(), list_size=2)), # required
    pa.field('hypothesis_template', pa.string()), # optional
    pa.field('multi_label', pa.bool_()), # optional
])

output_schema = pa.schema([
    pa.field('sequence', pa.string()),
    pa.field('scores', pa.list_(pa.float64(), list_size=2)), # same as number of candidate labels, list_size can be skipped by may result in slightly worse performance
    pa.field('labels', pa.list_(pa.string(), list_size=2)), # same as number of candidate labels, list_size can be skipped by may result in slightly worse performance
])
Wallaroo FrameworkReference
Framework.HUGGING_FACE_ZERO_SHOT_IMAGE_CLASSIFICATION

Schemas:

input_schema = pa.schema([
    pa.field('inputs', # required
        pa.list_(
            pa.list_(
                pa.list_(
                    pa.int64(),
                    list_size=3
                ),
                list_size=100
            ),
        list_size=100
    )),
    pa.field('candidate_labels', pa.list_(pa.string(), list_size=2)), # required
    pa.field('hypothesis_template', pa.string()), # optional
]) 

output_schema = pa.schema([
    pa.field('score', pa.list_(pa.float64(), list_size=2)), # same as number of candidate labels
    pa.field('label', pa.list_(pa.string(), list_size=2)), # same as number of candidate labels
])
Wallaroo FrameworkReference
Framework.HUGGING_FACE_ZERO_SHOT_OBJECT_DETECTION

Schemas:

input_schema = pa.schema([
    pa.field('images', 
        pa.list_(
            pa.list_(
                pa.list_(
                    pa.int64(),
                    list_size=3
                ),
                list_size=640
            ),
        list_size=480
    )),
    pa.field('candidate_labels', pa.list_(pa.string(), list_size=3)),
    pa.field('threshold', pa.float64()),
    # pa.field('top_k', pa.int64()), # we want the model to return exactly the number of predictions, we shouldn't specify this
])

output_schema = pa.schema([
    pa.field('score', pa.list_(pa.float64())), # variable output, depending on detected objects
    pa.field('label', pa.list_(pa.string())), # variable output, depending on detected objects
    pa.field('box', 
        pa.list_( # dynamic output, i.e. dynamic number of boxes per input image, each sublist contains the 4 box coordinates 
            pa.list_(
                    pa.int64(),
                    list_size=4
                ),
            ),
    ),
])
Wallaroo FrameworkReference
Framework.HUGGING_FACE_SENTIMENT_ANALYSISHugging Face Sentiment Analysis
Wallaroo FrameworkReference
Framework.HUGGING_FACE_TEXT_GENERATION

Any parameter that is not part of the required inputs list will be forwarded to the model as a key/pair value to the underlying models generate method. If the additional input is not supported by the model, an error will be returned.

input_schema = pa.schema([
    pa.field('inputs', pa.string()),
    pa.field('return_tensors', pa.bool_()), # optional
    pa.field('return_text', pa.bool_()), # optional
    pa.field('return_full_text', pa.bool_()), # optional
    pa.field('clean_up_tokenization_spaces', pa.bool_()), # optional
    pa.field('prefix', pa.string()), # optional
    pa.field('handle_long_generation', pa.string()), # optional
    # pa.field('extra_field', pa.int64()), # every extra field you specify will be forwarded as a key/value pair
])

output_schema = pa.schema([
    pa.field('generated_text', pa.list_(pa.string(), list_size=1))
])
Wallaroo FrameworkReference
Framework.HUGGING_FACE_AUTOMATIC_SPEECH_RECOGNITION

Sample input and output schema.

input_schema = pa.schema([
    pa.field('inputs', pa.list_(pa.float32())), # required: the audio stored in numpy arrays of shape (num_samples,) and data type `float32`
    pa.field('return_timestamps', pa.string()) # optional: return start & end times for each predicted chunk
]) 

output_schema = pa.schema([
    pa.field('text', pa.string()), # required: the output text corresponding to the audio input
    pa.field('chunks', pa.list_(pa.struct([('text', pa.string()), ('timestamp', pa.list_(pa.float32()))]))), # required (if `return_timestamps` is set), start & end times for each predicted chunk
])
ParameterDescription
Web Sitehttps://pytorch.org/
Supported Libraries
  • torch==1.13.1
  • torchvision==0.14.1
FrameworkFramework.PYTORCH aka pytorch
Supported File Typespt ot pth in TorchScript format
Runtimeonnx/flight
  • IMPORTANT NOTE: The PyTorch model must be in TorchScript format. scripting (i.e. torch.jit.script() is always recommended over tracing (i.e. torch.jit.trace()). From the PyTorch documentation: “Scripting preserves dynamic control flow and is valid for inputs of different sizes.” For more details, see TorchScript-based ONNX Exporter: Tracing vs Scripting.

During the model upload process, the Wallaroo instance will attempt to convert the model to a Native Wallaroo Runtime. If unsuccessful based , it will create a Wallaroo Containerized Runtime for the model. See the model deployment section for details on how to configure pipeline resources based on the model’s runtime.

  • IMPORTANT CONFIGURATION NOTE: For PyTorch input schemas, the floats must be pyarrow.float32() for the PyTorch model to be converted to the Native Wallaroo Runtime during the upload process.

Sci-kit Learn aka SKLearn.

ParameterDescription
Web Sitehttps://scikit-learn.org/stable/index.html
Supported Libraries
  • scikit-learn==1.3.0
FrameworkFramework.SKLEARN aka sklearn
Runtimeonnx / flight

During the model upload process, the Wallaroo instance will attempt to convert the model to a Native Wallaroo Runtime. If unsuccessful based , it will create a Wallaroo Containerized Runtime for the model. See the model deployment section for details on how to configure pipeline resources based on the model’s runtime.

SKLearn Schema Inputs

SKLearn schema follows a different format than other models. To prevent inputs from being out of order, the inputs should be submitted in a single row in the order the model is trained to accept, with all of the data types being the same. For example, the following DataFrame has 4 columns, each column a float.

 sepal length (cm)sepal width (cm)petal length (cm)petal width (cm)
05.13.51.40.2
14.93.01.40.2

For submission to an SKLearn model, the data input schema will be a single array with 4 float values.

input_schema = pa.schema([
    pa.field('inputs', pa.list_(pa.float64(), list_size=4))
])

When submitting as an inference, the DataFrame is converted to rows with the column data expressed as a single array. The data must be in the same order as the model expects, which is why the data is submitted as a single array rather than JSON labeled columns: this insures that the data is submitted in the exact order as the model is trained to accept.

Original DataFrame:

 sepal length (cm)sepal width (cm)petal length (cm)petal width (cm)
05.13.51.40.2
14.93.01.40.2

Converted DataFrame:

 inputs
0[5.1, 3.5, 1.4, 0.2]
1[4.9, 3.0, 1.4, 0.2]

SKLearn Schema Outputs

Outputs for SKLearn that are meant to be predictions or probabilities when output by the model are labeled in the output schema for the model when uploaded to Wallaroo. For example, a model that outputs either 1 or 0 as its output would have the output schema as follows:

output_schema = pa.schema([
    pa.field('predictions', pa.int32())
])

When used in Wallaroo, the inference result is contained in the out metadata as out.predictions.

pipeline.infer(dataframe)
 timein.inputsout.predictionscheck_failures
02023-07-05 15:11:29.776[5.1, 3.5, 1.4, 0.2]00
12023-07-05 15:11:29.776[4.9, 3.0, 1.4, 0.2]00
ParameterDescription
Web Sitehttps://www.tensorflow.org/api_docs/python/tf/keras/Model
Supported Libraries
  • tensorflow==2.9.3
  • keras==2.9.0
FrameworkFramework.KERAS aka keras
Supported File TypesSavedModel format as .zip file and HDF5 format
Runtimeonnx/flight

During the model upload process, the Wallaroo instance will attempt to convert the model to a Native Wallaroo Runtime. If unsuccessful based , it will create a Wallaroo Containerized Runtime for the model. See the model deployment section for details on how to configure pipeline resources based on the model’s runtime.

TensorFlow Keras SavedModel Format

TensorFlow Keras SavedModel models are .zip file of the SavedModel format. For example, the Aloha sample TensorFlow model is stored in the directory alohacnnlstm:

├── saved_model.pb
└── variables
    ├── variables.data-00000-of-00002
    ├── variables.data-00001-of-00002
    └── variables.index

This is compressed into the .zip file alohacnnlstm.zip with the following command:

zip -r alohacnnlstm.zip alohacnnlstm/

See the SavedModel guide for full details.

TensorFlow Keras H5 Format

Wallaroo supports the H5 for Tensorflow Keras models.

ParameterDescription
Web Sitehttps://xgboost.ai/
Supported Libraries
  • scikit-learn==1.3.0
  • xgboost==1.7.4
FrameworkFramework.XGBOOST aka xgboost
Supported File Typespickle (XGB files are not supported.)
Runtimeonnx / flight

During the model upload process, the Wallaroo instance will attempt to convert the model to a Native Wallaroo Runtime. If unsuccessful based , it will create a Wallaroo Containerized Runtime for the model. See the model deployment section for details on how to configure pipeline resources based on the model’s runtime.

XGBoost Schema Inputs

XGBoost schema follows a different format than other models. To prevent inputs from being out of order, the inputs should be submitted in a single row in the order the model is trained to accept, with all of the data types being the same. If a model is originally trained to accept inputs of different data types, it will need to be retrained to only accept one data type for each column - typically pa.float64() is a good choice.

For example, the following DataFrame has 4 columns, each column a float.

 sepal length (cm)sepal width (cm)petal length (cm)petal width (cm)
05.13.51.40.2
14.93.01.40.2

For submission to an XGBoost model, the data input schema will be a single array with 4 float values.

input_schema = pa.schema([
    pa.field('inputs', pa.list_(pa.float64(), list_size=4))
])

When submitting as an inference, the DataFrame is converted to rows with the column data expressed as a single array. The data must be in the same order as the model expects, which is why the data is submitted as a single array rather than JSON labeled columns: this insures that the data is submitted in the exact order as the model is trained to accept.

Original DataFrame:

 sepal length (cm)sepal width (cm)petal length (cm)petal width (cm)
05.13.51.40.2
14.93.01.40.2

Converted DataFrame:

 inputs
0[5.1, 3.5, 1.4, 0.2]
1[4.9, 3.0, 1.4, 0.2]

XGBoost Schema Outputs

Outputs for XGBoost are labeled based on the trained model outputs. For this example, the output is simply a single output listed as output. In the Wallaroo inference result, it is grouped with the metadata out as out.output.

output_schema = pa.schema([
    pa.field('output', pa.int32())
])
pipeline.infer(dataframe)
 timein.inputsout.outputcheck_failures
02023-07-05 15:11:29.776[5.1, 3.5, 1.4, 0.2]00
12023-07-05 15:11:29.776[4.9, 3.0, 1.4, 0.2]00
ParameterDescription
Web Sitehttps://www.python.org/
Supported Librariespython==3.8
FrameworkFramework.CUSTOM aka custom
RuntimeContainerized aka flight

Arbitrary Python models, also known as Bring Your Own Predict (BYOP) allow for custom model deployments with supporting scripts and artifacts. These are used with pre-trained models (PyTorch, Tensorflow, etc) along with whatever supporting artifacts they require. Supporting artifacts can include other Python modules, model files, etc. These are zipped with all scripts, artifacts, and a requirements.txt file that indicates what other Python models need to be imported that are outside of the typical Wallaroo platform.

Contrast this with Wallaroo Python models - aka “Python steps”. These are standalone python scripts that use the python libraries natively supported by the Wallaroo platform. These are used for either simple model deployment (such as ARIMA Statsmodels), or data formatting such as the postprocessing steps. A Wallaroo Python model will be composed of one Python script that matches the Wallaroo requirements.

Arbitrary Python File Requirements

Arbitrary Python (BYOP) models are uploaded to Wallaroo via a ZIP file with the following components:

ArtifactTypeDescription
Python scripts aka .py files with classes that extend mac.inference.Inference and mac.inference.creation.InferenceBuilderPython ScriptExtend the classes mac.inference.Inference and mac.inference.creation.InferenceBuilder. These are included with the Wallaroo SDK. Further details are in Arbitrary Python Script Requirements. Note that there is no specified naming requirements for the classes that extend mac.inference.Inference and mac.inference.creation.InferenceBuilder - any qualified class name is sufficient as long as these two classes are extended as defined below.
requirements.txtPython requirements fileThis sets the Python libraries used for the arbitrary python model. These libraries should be targeted for Python 3.8 compliance. These requirements and the versions of libraries should be exactly the same between creating the model and deploying it in Wallaroo. This insures that the script and methods will function exactly the same as during the model creation process.
Other artifactsFilesOther models, files, and other artifacts used in support of this model.

For example, the if the arbitrary python model will be known as vgg_clustering, the contents may be in the following structure, with vgg_clustering as the storage directory:

vgg_clustering\
    feature_extractor.h5
    kmeans.pkl
    custom_inference.py
    requirements.txt

Note the inclusion of the custom_inference.py file. This file name is not required - any Python script or scripts that extend the classes listed above are sufficient. This Python script could have been named vgg_custom_model.py or any other name as long as it includes the extension of the classes listed above.

The sample arbitrary python model file is created with the command zip -r vgg_clustering.zip vgg_clustering/.

Wallaroo Arbitrary Python uses the Wallaroo SDK mac module, included in the Wallaroo SDK 2023.2.1 and above. See the Wallaroo SDK Install Guides for instructions on installing the Wallaroo SDK.

Arbitrary Python Script Requirements

The entry point of the arbitrary python model is any python script that extends the following classes. These are included with the Wallaroo SDK. The required methods that must be overridden are specified in each section below.

  • mac.inference.Inference interface serves model inferences based on submitted input some input. Its purpose is to serve inferences for any supported arbitrary model framework (e.g. scikit, keras etc.).

    classDiagram
        class Inference {
            <<Abstract>>
            +model Optional[Any]
            +expected_model_types()* Set
            +predict(input_data: InferenceData)*  InferenceData
            -raise_error_if_model_is_not_assigned() None
            -raise_error_if_model_is_wrong_type() None
        }
  • mac.inference.creation.InferenceBuilder builds a concrete Inference, i.e. instantiates an Inference object, loads the appropriate model and assigns the model to to the Inference object.

    classDiagram
        class InferenceBuilder {
            +create(config InferenceConfig) * Inference
            -inference()* Any
        }

mac.inference.Inference

mac.inference.Inference Objects
ObjectTypeDescription
model (Required)[Any]One or more objects that match the expected_model_types. This can be a ML Model (for inference use), a string (for data conversion), etc. See Arbitrary Python Examples for examples.
mac.inference.Inference Methods
MethodReturnsDescription
expected_model_types (Required)SetReturns a Set of models expected for the inference as defined by the developer. Typically this is a set of one. Wallaroo checks the expected model types to verify that the model submitted through the InferenceBuilder method matches what this Inference class expects.
_predict (input_data: mac.types.InferenceData) (Required)mac.types.InferenceDataThe entry point for the Wallaroo inference with the following input and output parameters that are defined when the model is updated.
  • mac.types.InferenceData: The input InferenceData is a Dictionary of numpy arrays derived from the input_schema detailed when the model is uploaded, defined in PyArrow.Schema format.
  • mac.types.InferenceData: The output is a Dictionary of numpy arrays as defined by the output parameters defined in PyArrow.Schema format.
The InferenceDataValidationError exception is raised when the input data does not match mac.types.InferenceData.
raise_error_if_model_is_not_assignedN/AError when a model is not set to Inference.
raise_error_if_model_is_wrong_typeN/AError when the model does not match the expected_model_types.

The example, the expected_model_types can be defined for the KMeans model.

from sklearn.cluster import KMeans

class SampleClass(mac.inference.Inference):
    @property
    def expected_model_types(self) -> Set[Any]:
        return {KMeans}

mac.inference.creation.InferenceBuilder

InferenceBuilder builds a concrete Inference, i.e. instantiates an Inference object, loads the appropriate model and assigns the model to the Inference.

classDiagram
    class InferenceBuilder {
        +create(config InferenceConfig) * Inference
        -inference()* Any
    }

Each model that is included requires its own InferenceBuilder. InferenceBuilder loads one model, then submits it to the Inference class when created. The Inference class checks this class against its expected_model_types() Set.

mac.inference.creation.InferenceBuilder Methods
MethodReturnsDescription
create(config mac.config.inference.CustomInferenceConfig) (Required)The custom Inference instance.Creates an Inference subclass, then assigns a model and attributes. The CustomInferenceConfig is used to retrieve the config.model_path, which is a pathlib.Path object pointing to the folder where the model artifacts are saved. Every artifact loaded must be relative to config.model_path. This is set when the arbitrary python .zip file is uploaded and the environment for running it in Wallaroo is set. For example: loading the artifact vgg_clustering\feature_extractor.h5 would be set with config.model_path \ feature_extractor.h5. The model loaded must match an existing module. For our example, this is from sklearn.cluster import KMeans, and this must match the Inference expected_model_types.
inferencecustom Inference instance.Returns the instantiated custom Inference object created from the create method.

Arbitrary Python Runtime

Arbitrary Python always run in the containerized model runtime.

ParameterDescription
Web Sitehttps://mlflow.org
Supported Librariesmlflow==1.3.0
RuntimeContainerized aka mlflow

For models that do not fall under the supported model frameworks, organizations can use containerized MLFlow ML Models.

This guide details how to add ML Models from a model registry service into Wallaroo.

Wallaroo supports both public and private containerized model registries. See the Wallaroo Private Containerized Model Container Registry Guide for details on how to configure a Wallaroo instance with a private model registry.

Wallaroo users can register their trained MLFlow ML Models from a containerized model container registry into their Wallaroo instance and perform inferences with it through a Wallaroo pipeline.

As of this time, Wallaroo only supports MLFlow 1.30.0 containerized models. For information on how to containerize an MLFlow model, see the MLFlow Documentation.

Wallaroo supports both public and private containerized model registries. See the Wallaroo Private Containerized Model Container Registry Guide for details on how to configure a Wallaroo instance with a private model registry.

List Wallaroo Frameworks

Wallaroo frameworks are listed from the Wallaroo.Framework class. The following demonstrates listing all available supported frameworks.

from wallaroo.framework import Framework

[e.value for e in Framework]

    ['onnx',
    'tensorflow',
    'python',
    'keras',
    'sklearn',
    'pytorch',
    'xgboost',
    'hugging-face-feature-extraction',
    'hugging-face-image-classification',
    'hugging-face-image-segmentation',
    'hugging-face-image-to-text',
    'hugging-face-object-detection',
    'hugging-face-question-answering',
    'hugging-face-stable-diffusion-text-2-img',
    'hugging-face-summarization',
    'hugging-face-text-classification',
    'hugging-face-translation',
    'hugging-face-zero-shot-classification',
    'hugging-face-zero-shot-image-classification',
    'hugging-face-zero-shot-object-detection',
    'hugging-face-sentiment-analysis',
    'hugging-face-text-generation']

Upload Model to Workspace

  • Endpoint: /v1/api/models/upload_and_convert
  • Content-Type: multipart/form-data

Models uploaded through this method that are not Wallaroo Native Runtimes (ONNX, Tensorflow, and Python script) are containerized within the Wallaroo instance then run by the Wallaroo engine. See Wallaroo MLOps API Essentials Guide: Pipeline Management for details on pipeline configurations and deployments.

Upload Model to Workspace Parameters

Field TypeDescription
name String (Required)The model name.
visibility String (Required)Either public or private.
workspace_id String (Required)The numerical ID of the workspace to upload the model to.
conversion String (Required)The conversion parameters that include the following:
 frameworkString (Required)The framework of the model being uploaded. See the list of supported models for more details.
 python_versionString (Required)The version of Python required for model.
 requirementsString (Required)Required libraries. Can be [] if the requirements are default Wallaroo JupyterHub libraries.
 input_schemaString (Optional)The input schema from the Apache Arrow pyarrow.lib.Schema format, encoded with base64.b64encode. Only required for Containerized Wallaroo Runtime models.
 output_schemaString (Optional)The output schema from the Apache Arrow pyarrow.lib.Schema format, encoded with base64.b64encode. Only required for non-native runtime models.

Files are uploaded in the multipart/form-data format with two parts:

  • metadata: Contains the parameters listed above as application/json.
  • file: The binary file (ONNX, .zip, etc) as Content-Type application/octet-stream.

Target Architecture for ARM Deployment

Models uploaded to Wallaroo default to a target architecture deployment of x86. Pipeline deployments default to x86.

Models with the target architecture ARM require the pipeline deployment configuration also be ARM. For example:

model_data = {
    "name": model_name,
    "visibility": "public",
    "workspace_id": workspaceId,
    "conversion": {
        "arch": "arm"
        "framework": framework,
        "python_version": "3.8",
        "requirements": []
    }
}

pipeline_data = {
    "deploy_id": exampleModelDeployId,
    "pipeline_version_pk_id": exampleModelPipelineVariantId,
    "engine_config": {
        "arch": "arm"
    }
    "models": [
        {
            "name":exampleModelName,
            "version":exampleModelVersion,
            "sha":exampleModelSha
        }
    ],
    "pipeline_id": exampleModelPipelineId
}

Upload Model to Workspace Returns

Field TypeDescription
insert_models{‘returning’: [models]} List[models]The uploaded models details.
 idIntegerThe model’s numerical id.

Upload Model to Workspace Examples

The following example shows uploading an ONNX model to a Wallaroo instance. Note that the input_schema and output_schema encoded details are not required.

This example assumes the workspace id of 10. Modify this code block based on your Wallaroo Ops instance.

Upload model via Requests library.

# Retrieve the token 
headers = wl.auth.auth_header()

endpoint = f"{wl.api_endpoint}/v1/api/models/upload_and_convert"

workspace_id = 10

framework='onnx'

model_name = f"api-sample-model"

metadata = {
    "name": model_name,
    "visibility": "public",
    "workspace_id": workspace_id,
    "conversion": {
        "framework": framework,
        "python_version": "3.8",
        "requirements": []
    }
}

files = {
    "metadata": (None, json.dumps(data), "application/json"),
    'file': (model_name, open('./models/ccfraud.onnx', 'rb'), "application/octet-stream")
    }

response = requests.post(endpoint, files=files, headers=headers).json()

display(f"Uploaded Model Name: {model_name}.")
display(f"Sample model file: ./models/ccfraud.onnx")
display(response)
'Uploaded Model Name: api-sample-model.'

'Sample model file: ./models/ccfraud.onnx'

{'insert_models': {'returning': [{'models': [{'id': 14}]}]}}

Upload ONNX model via curl.

metadata = {
    "name": model_name,
    "visibility": "public",
    "workspace_id": workspace_id,
    "conversion": {
        "framework": framework,
        "python_version": "3.8",
        "requirements": []
    }
}

# save metadata to a file
with open("data/onnx_file_upload.json", "w") as outfile:
    json.dump(metadata, outfile)
curl {wl.api_endpoint}/v1/api/models/upload_and_convert \
    -H "Authorization: {wl.auth.auth_header()['Authorization']}" \
    --form 'metadata=@./data/onnx_file_upload.json;type=application/json' \
    --form 'file=@./models/ccfraud.onnx;type=application/octet-stream'
{"insert_models":{"returning":[{"models":[{"id":18}]}]}}

The following example shows uploading a Pytorch model to a Wallaroo instance. Note that the input_schema and output_schema encoded details are required.

Upload Pytorch via Requests.

input_schema = pa.schema([
    pa.field('input_1', pa.list_(pa.float32(), list_size=10)),
    pa.field('input_2', pa.list_(pa.float32(), list_size=5))
])
output_schema = pa.schema([
    pa.field('output_1', pa.list_(pa.float32(), list_size=3)),
    pa.field('output_2', pa.list_(pa.float32(), list_size=2))
])

encoded_input_schema = base64.b64encode(
                bytes(input_schema.serialize())
            ).decode("utf8")

encoded_output_schema = base64.b64encode(
                bytes(output_schema.serialize())
            ).decode("utf8")

framework = 'pytorch'

model_name = 'api-upload-pytorch-multi-io'

metadata = {
    "name": model_name,
    "visibility": "private",
    "workspace_id": workspace_id,
    "conversion": {
        "framework": framework,
        "python_version": "3.8",
        "requirements": []
    },
    "input_schema": encoded_input_schema,
    "output_schema": encoded_output_schema,
}

headers = wl.auth.auth_header()

files = {
    'metadata': (None, json.dumps(metadata), "application/json"),
    'file': (model_name, open('./models/model-auto-conversion_pytorch_multi_io_model.pt','rb'),'application/octet-stream')
}

response = requests.post(endpoint, files=files, headers=headers).json()

display(f"Uploaded Model Name: {model_name}.")
display(f"Sample model file: ./models/model-auto-conversion_pytorch_multi_io_model.pt")
display(response)
'Uploaded Model Name: api-upload-pytorch-multi-io.'

'Sample model file: ./models/model-auto-conversion_pytorch_multi_io_model.pt'

{'insert_models': {'returning': [{'models': [{'id': 15}]}]}}

Upload Pytorch via curl.

# save metadata to a file
with open("./data/pytorch_file_upload.json", "w") as outfile:
    json.dump(metadata, outfile)

curl {wl.api_endpoint}/v1/api/models/upload_and_convert \
    -H "Authorization: {wl.auth.auth_header()['Authorization']}" \
    --form 'metadata=@./data/pytorch_file_upload.json;type=application/json' \
    --form 'file=@./models/model-auto-conversion_pytorch_multi_io_model.pt;type=application/octet-stream'
{"insert_models":{"returning":[{"models":[{"id":19}]}]}}

6 - Wallaroo MLOps API Essentials Guide: Pipeline Management

How to use the Wallaroo API for Pipeline Management

Pipeline Naming Requirements

Pipeline names map onto Kubernetes objects, and must be DNS compliant. Pipeline names must be ASCII alpha-numeric characters or dash (-) only. . and _ are not allowed.

Pipeline Management

Pipelines are managed through the Wallaroo API or the Wallaroo SDK. Pipelines are the vehicle used for deploying, serving, and monitoring ML models. For more information, see the Wallaroo Glossary.

Create Pipeline in a Workspace

  • Endpoint: /v1/api/pipelines/create

Creates a new pipeline in the specified workspace.

Create Pipeline in a Workspace Parameters

Field TypeDescription
pipeline_id String (Required)Name of the new pipeline.
workspace_id Integer (Required)Numerical id of the workspace for the new pipeline.
definition String (Required)Pipeline definitions, can be {} for none. This is where the pipeline steps are set.
 stepsList[steps]The pipeline steps to add to the pipeline.
Model Inference Pipeline Step

Pipeline steps from models follow the following schema.

{
  "ModelInference": {
    "models": [
      {
        "name": "{name of model: String}",
        "version": {model version: Integer},
        "sha": "{model sha: String}"
      }
    ]
  }
}

Create Pipeline in a Workspace Returns

FieldTypeDescription
pipeline_pk_idIntegerThe pipeline id.
pipeline_variant_pk_idIntegerThe pipeline version id.
pipeline_variant_versionStringThe pipeline version UUID identifier.

Create Pipeline in a Workspace Examples

Two pipelines are created in the workspace id 10. This assumes that the workspace is created and has models uploaded to it.

One pipeline is an empty pipeline without any models.

For the other pipeline, sample models are uploaded then added pipeline.

The pipeline id, variant id, and variant version of each pipeline will be stored for later examples.

Create empty pipeline via Requests.

# Create pipeline in a workspace
# Retrieve the token 
headers = wl.auth.auth_header()

endpoint = f"{wl.api_endpoint}/v1/api/pipelines/create"

example_workspace_id = 10

data = {
  "pipeline_id": "api-empty-pipeline",
  "workspace_id": example_workspace_id,
  "definition": {}
}

response = requests.post(endpoint, 
                         json=data, 
                         headers=headers, 
                         verify=True).json()

empty_pipeline_id = response['pipeline_pk_id']
empty_pipeline_variant_id=response['pipeline_variant_pk_id']
example_pipeline_variant_version=['pipeline_variant_version']
display(json.dumps(response))
    '{"pipeline_pk_id": 25, "pipeline_variant_pk_id": 26, "pipeline_variant_version": "c29a277a-10b9-48f9-a738-aafb296df8c2"}'

Create empty pipeline via curl.

curl {wl.api_endpoint}/v1/api/pipelines/create \
    -H "Authorization: {wl.auth.auth_header()['Authorization']}" \
    -H "Content-Type: application/json" \
    --data '{{"pipeline_id": "api-empty-pipeline", "workspace_id": {example_workspace_id},"definition": {{}} }}'
{"pipeline_pk_id":25,"pipeline_variant_pk_id":27,"pipeline_variant_version":"f6241f32-85a8-4ad8-9e71-da2763717811"}

Create pipeline with model steps via Requests.

# Create pipeline in a workspace with models

# First upload a model
# Retrieve the token 
headers = wl.auth.auth_header()

endpoint = f"{wl.api_endpoint}/v1/api/models/upload_and_convert"

workspace_id = 10

framework='onnx'

example_model_name = f"api-sample-model"

metadata = {
    "name": example_model_name,
    "visibility": "public",
    "workspace_id": workspace_id,
    "conversion": {
        "framework": framework,
        "python_version": "3.8",
        "requirements": []
    }
}

files = {
    "metadata": (None, json.dumps(metadata), "application/json"),
    'file': (example_model_name, open('./models/ccfraud.onnx', 'rb'), "application/octet-stream")
    }

response = requests.post(endpoint, files=files, headers=headers).json()

example_model_id = response['insert_models']['returning'][0]['models'][0]['id']

# Second, get the model version

# Retrieve the token 
headers = wl.auth.auth_header()
endpoint = f"{wl.api_endpoint}/v1/api/models/list_versions"

data = {
  "model_id": example_model_name,
  "models_pk_id": example_model_id
}

response = requests.post(endpoint, json=data, headers=headers, verify=True).json()
example_model_sha = response[-1]['sha']
example_model_version = response[-1]['model_version']

# Now create the pipeline with the new model
# Retrieve the token 
headers = wl.auth.auth_header()

endpoint = f"{wl.api_endpoint}/v1/api/pipelines/create"

data = {
  "pipeline_id": "api-pipeline-with-models",
  "workspace_id": example_workspace_id,
  "definition": {
      'steps': [
          {
            'ModelInference': 
              {
                  'models': 
                    [
                        {
                            'name': example_model_name, 
                            'version': example_model_version, 
                            'sha': example_model_sha
                        }
                    ]
              }
          }
        ]
      }
    }

response = requests.post(endpoint, 
                         json=data, 
                         headers=headers, 
                         verify=True).json()
display(json.dumps(response))

# saved for later steps

model_pipeline_id = response['pipeline_pk_id']
model_pipeline_variant_id=response['pipeline_variant_pk_id']
model_pipeline_variant_version=['pipeline_variant_version']
'{"pipeline_pk_id": 28, "pipeline_variant_pk_id": 29, "pipeline_variant_version": "5d0326fa-6753-4252-bb56-3b2106a8c671"}'

Create pipeline with model steps via curl.

curl {wl.api_endpoint}/v1/api/pipelines/create \
    -H "Authorization: {wl.auth.auth_header()['Authorization']}" \
    -H "Content-Type: application/json" \
    --data '{{ \
    "pipeline_id": "api-pipeline-with-models", \
    "workspace_id": {example_workspace_id}, \
    "definition": {{ \
        "steps": [ \
            {{ \
              "ModelInference": \
                {{ \
                    "models": \
                      [ \
                          {{ \
                              "name": "{example_model_name}", \
                              "version": "{example_model_version}", \
                              "sha": "{example_model_sha}" \
                          }} \
                      ] \
                }} \
            }} \
          ] \
        }} \
      }}'
{
  "pipeline_pk_id": 28,
  "pipeline_variant_pk_id": 30,
  "pipeline_variant_version": "82148e63-3950-4ec4-99f7-b9a22212bfdf"
}

GPU Support

Wallaroo 2023.2.1 and above supports Kubernetes nodepools with Nvidia Cuda GPUs.

See the Create GPU Nodepools for Kubernetes Clusters guide for instructions on adding GPU enabled nodepools to a Kubernetes cluster.

Deploy a Pipeline

  • Endpoint: /v1/api/pipelines/deploy

Deploy a an existing pipeline. Note that for any pipeline that has model steps, they must be included either in model_configs, model_ids or models.

Deploy a Pipeline Parameters

Field TypeDescription
deploy_id String (REQUIRED)The name for the pipeline deployment. This must match the name of the pipeline being deployed.
engine_config String (OPTIONAL)Additional configuration options for the pipeline.
pipeline_version_pk_id Integer REQUIRED)Pipeline version id.
model_configs List[Integer] (OPTIONAL)Ids of model configs to apply.
model_ids List[Integer] (OPTIONAL)Ids of models to apply to the pipeline. If passed in, model_configs will be created automatically.
models List[models] (OPTIONAL)If the model ids are not available as a pipeline step, the models’ data can be passed to it through this method. The options below are only required if models are provided as a parameter.
 nameString (REQUIRED)Name of the uploaded model that is in the same workspace as the pipeline.
 versionString (REQUIRED)Version of the model to use.
 shaString (REQUIRED)SHA value of the model.
pipeline_id *Integer (REQUIRED)Numerical value of the pipeline to deploy.

Deploy a Pipeline Returns

FieldTypeDescription
idIntegerThe deployment id.

Deploy a Pipeline Returns

The pipeline with models created in the step Create Pipeline in a Workspace will be deployed and their deployment information saved for later examples.

Deploy pipeline via Requests.

# Deploy a pipeline with models

# Retrieve the token 
headers = wl.auth.auth_header()
endpoint = f"{wl.api_endpoint}/v1/api/pipelines/deploy"

# verify this matches the pipeline with model created earlier
pipeline_with_models_id = "api-pipeline-with-models"

data = {
    "deploy_id": pipeline_with_models_id,
    "pipeline_version_pk_id": model_pipeline_variant_id,
    "models": [
        {
            "name": example_model_name,
            "version":example_model_version,
            "sha":example_model_sha
        }
    ],
    "pipeline_id": model_pipeline_id
}

response = requests.post(endpoint, json=data, headers=headers, verify=True).json()
display(response)
model_deployment_id=response['id']
{'id': 14}

Deploy pipeline via curl.

curl {wl.api_endpoint}/v1/api/pipelines/deploy \
    -H "Authorization: {wl.auth.auth_header()['Authorization']}" \
    -H "Content-Type: application/json" \
    --data '{{ \
        "deploy_id": "{pipeline_with_models_id}", \
        "pipeline_version_pk_id": {model_pipeline_variant_id}, \
        "models": [ \
            {{ \
                "name": "{example_model_name}", \
                "version": "{example_model_version}", \
                "sha": "{example_model_sha}" \
            }} \
        ], \
        "pipeline_id": {model_pipeline_id} \
    }}'
{"id":14}

Get Deployment Status

  • Endpoint: /v1/api/status/get_deployment

Get Deployment Status Parameters

FieldTypeDescription
idString (Required)The deployment in the format {deployment_name}-{deployment-id}.

Get Deployment Status Returns

FieldTypeDescription
statusStringStatus of the pipeline deployment. Values are: Running: the Deployment successfully started. Starting: The Deployment is still loading. Error: There is an error with the deployment.
detailsList[details]The list of deployment details.
enginesList[engines]A list of engines deployed in the pipeline.
engine_lbsList[engine_lbs]A list of engine load balancers.
sidekicksList[sidekicks]A list of deployment sidekicks. These are used for Containerized Deployment Runtimes.

Get Deployment Status Examples

The deployed pipeline with model details from the previous step is displayed.

Get deployment status via Requests.

# Retrieve the token 
headers = wl.auth.auth_header()

# Get model pipeline deployment

endpoint = f"{wl.api_endpoint}/v1/api/status/get_deployment"

data = {
  "name": f"{pipeline_with_models_id}-{model_deployment_id}"
}

response = requests.post(endpoint, json=data, headers=headers, verify=True).json()
response
    {'status': 'Running',
     'details': [],
     'engines': [{'ip': '10.244.3.145',
       'name': 'engine-797d8958d9-fsszh',
       'status': 'Running',
       'reason': None,
       'details': [],
       'pipeline_statuses': {'pipelines': [{'id': 'api-pipeline-with-models',
          'status': 'Running'}]},
       'model_statuses': {'models': [{'name': 'api-sample-model',
          'version': 'bdfc8c60-b5bc-4c0e-aa87-157cd52895b6',
          'sha': 'bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507',
          'status': 'Running'}]}}],
     'engine_lbs': [{'ip': '10.244.4.157',
       'name': 'engine-lb-584f54c899-qfjgp',
       'status': 'Running',
       'reason': None,
       'details': []}],
     'sidekicks': []}

Get deployment status via curl.

curl {wl.api_endpoint}/v1/api/status/get_deployment \
    -H "Authorization: {wl.auth.auth_header()['Authorization']}" \
    -H "Content-Type: application/json" \
    --data '{{ \
        "name": "{pipeline_with_models_id}-{model_deployment_id}" \
}}'
{
  "status": "Running",
  "details": [],
  "engines": [
    {
      "ip": "10.244.3.145",
      "name": "engine-797d8958d9-fsszh",
      "status": "Running",
      "reason": null,
      "details": [],
      "pipeline_statuses": {
        "pipelines": [{ "id": "api-pipeline-with-models", "status": "Running" }]
      },
      "model_statuses": {
        "models": [
          {
            "name": "api-sample-model",
            "version": "bdfc8c60-b5bc-4c0e-aa87-157cd52895b6",
            "sha": "bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507",
            "status": "Running"
          }
        ]
      }
    }
  ],
  "engine_lbs": [
    {
      "ip": "10.244.4.157",
      "name": "engine-lb-584f54c899-qfjgp",
      "status": "Running",
      "reason": null,
      "details": []
    }
  ],
  "sidekicks": []
}

Target Architecture for ARM Deployment

Models uploaded to Wallaroo default to a target architecture deployment of x86. Pipeline deployments default to x86.

Models with the target architecture ARM require the pipeline deployment configuration also be ARM. For example:

model_data = {
    "name": model_name,
    "visibility": "public",
    "workspace_id": workspaceId,
    "conversion": {
        "arch": "arm"
        "framework": framework,
        "python_version": "3.8",
        "requirements": []
    }
}

pipeline_data = {
    "deploy_id": exampleModelDeployId,
    "pipeline_version_pk_id": exampleModelPipelineVariantId,
    "engine_config": {
        "arch": "arm"
    }
    "models": [
        {
            "name":exampleModelName,
            "version":exampleModelVersion,
            "sha":exampleModelSha
        }
    ],
    "pipeline_id": exampleModelPipelineId
}

Get External Inference URL

  • Endpoint: /v1/api/admin/get_pipeline_external_url

Retrieves the external inference URL for a specific pipeline in a workspace.

Get External Inference URL Parameters

FieldTypeDescription
workspace_idInteger (*REQUIRED)The workspace integer id.
pipeline_nameString (REQUIRED)The name of the deployment.

Get External Inference URL Returns

FieldTypeDescription
urlStringThe pipeline’s external inference URL.

Get External Inference URL Examples

In this example, the pipeline’s external inference URL from the previous example is retrieved.

Get external inference url via Requests.

## Retrieve the pipeline's External Inference URL

# Retrieve the token 
headers = wl.auth.auth_header()

endpoint = f"{wl.api_endpoint}/v1/api/admin/get_pipeline_external_url"

data = {
    "workspace_id": example_workspace_id,
    "pipeline_name": pipeline_with_models_id
}

response = requests.post(endpoint, json=data, headers=headers, verify=True).json()
print(response)
deployurl = response['url']
{'url': 'https://doc-test.api.wallarooexample.ai/v1/api/pipelines/infer/api-pipeline-with-models-14/api-pipeline-with-models'}

Get external inference url via Requests.

curl {wl.api_endpoint}/v1/api/admin/get_pipeline_external_url \
    -H "Authorization: {wl.auth.auth_header()['Authorization']}" \
    -H "Content-Type: application/json" \
    --data '{{ \
        "workspace_id": {example_workspace_id}, \
        "pipeline_name": "{pipeline_with_models_id}" \
}}'
{"url":"https://doc-test.api.wallarooexample.ai/v1/api/pipelines/infer/api-pipeline-with-models-14/api-pipeline-with-models"}

Perform Inference Through External URL

The inference can now be performed through the External Inference URL. This URL will accept the same inference data file that is used with the Wallaroo SDK, or with an Internal Inference URL as used in the Internal Pipeline Inference URL Tutorial.

Deployed pipelines have their own Inference URL that accepts HTTP POST submissions.

For connections that are external to the Kubernetes cluster hosting the Wallaroo instance, model endpoints must be enabled.

HTTP Headers

The following headers are required for connecting the the Pipeline Deployment URL:

  • Authorization: This requires the JWT token in the format 'Bearer ' + token. For example:

    Authorization: Bearer abcdefg==
    
  • Content-Type:

  • For DataFrame formatted JSON:

    Content-Type:application/json; format=pandas-records
    
  • For Arrow binary files, the Content-Type is application/vnd.apache.arrow.file.

    Content-Type:application/vnd.apache.arrow.file
    
  • IMPORTANT NOTE: Verify that the pipeline deployed has status Running before attempting an inference.

Perform inference via external url via Requests.

# Retrieve the token
headers = wl.auth.auth_header()

# set Content-Type type
headers['Content-Type']='application/json; format=pandas-records'

## Inference through external URL using dataframe

# retrieve the json data to submit
data = [
    {
        "dense_input":[
            1.0678324729,
            0.2177810266,
            -1.7115145262,
            0.682285721,
            1.0138553067,
            -0.4335000013,
            0.7395859437,
            -0.2882839595,
            -0.447262688,
            0.5146124988,
            0.3791316964,
            0.5190619748,
            -0.4904593222,
            1.1656456469,
            -0.9776307444,
            -0.6322198963,
            -0.6891477694,
            0.1783317857,
            0.1397992467,
            -0.3554220649,
            0.4394217877,
            1.4588397512,
            -0.3886829615,
            0.4353492889,
            1.7420053483,
            -0.4434654615,
            -0.1515747891,
            -0.2668451725,
            -1.4549617756
        ]
    }
]

# submit the request via POST, import as pandas DataFrame
response = pd.DataFrame.from_records(
    requests.post(
        deployurl, 
        json=data, 
        headers=headers)
        .json()
    )

display(response)
timeinoutcheck_failuresmetadata
01701376879347{'dense_input': [1.0678324729, 0.2177810266, -...{'dense_1': [0.0014974177]}[]{'last_model': '{"model_name":"api-sample-mode...

Perform inference via external url via curl.

curl {deployurl} \
    -H "Content-Type: application/json; format=pandas-records" \
    -H "Authorization: {wl.auth.auth_header()['Authorization']}" \
    --data '[ \
    {{ \
        "dense_input":[ \
            1.0678324729, \
            0.2177810266, \
            -1.7115145262, \
            0.682285721, \
            1.0138553067, \
            -0.4335000013, \
            0.7395859437, \
            -0.2882839595, \
            -0.447262688, \
            0.5146124988, \
            0.3791316964, \
            0.5190619748, \
            -0.4904593222, \
            1.1656456469, \
            -0.9776307444, \
            -0.6322198963, \
            -0.6891477694, \
            0.1783317857, \
            0.1397992467, \
            -0.3554220649, \
            0.4394217877, \
            1.4588397512, \
            -0.3886829615, \
            0.4353492889, \
            1.7420053483, \
            -0.4434654615, \
            -0.1515747891, \
            -0.2668451725, \
            -1.4549617756 \
        ] \
    }} \
]'
[
  {
    "time": 1701377058894,
    "in": {
      "dense_input": [
        1.0678324729, 0.2177810266, -1.7115145262, 0.682285721, 1.0138553067,
        -0.4335000013, 0.7395859437, -0.2882839595, -0.447262688, 0.5146124988,
        0.3791316964, 0.5190619748, -0.4904593222, 1.1656456469, -0.9776307444,
        -0.6322198963, -0.6891477694, 0.1783317857, 0.1397992467, -0.3554220649,
        0.4394217877, 1.4588397512, -0.3886829615, 0.4353492889, 1.7420053483,
        -0.4434654615, -0.1515747891, -0.2668451725, -1.4549617756
      ]
    },
    "out": { "dense_1": [0.0014974177] },
    "check_failures": [],
    "metadata": {
      "last_model": "{\"model_name\":\"api-sample-model\",\"model_sha\":\"bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507\"}",
      "pipeline_version": "",
      "elapsed": [53101, 318104],
      "dropped": [],
      "partition": "engine-797d8958d9-fsszh"
    }
  }
]

Undeploy a Pipeline

  • Endpoint: /v1/api/pipelines/undeploy

Undeploys a deployed pipeline.

Undeploy a Pipeline Parameters

FieldTypeDescription
pipeline_idInteger (*REQUIRED)The numerical id of the pipeline.
deployment_idInteger (*REQUIRED)The numerical id of the deployment.

Undeploy a Pipeline Returns

Nothing if the call is successful.

Undeploy a Pipeline Examples

The pipeline with models deployed is undeployed.

Undeploy the pipeline via Requests.

# Undeploy pipeline with models
# Retrieve the token 
headers = wl.auth.auth_header()
endpoint = f"{wl.api_endpoint}/v1/api/pipelines/undeploy"

data = {
    "pipeline_id": model_pipeline_id,
    "deployment_id":model_deployment_id
}

response = requests.post(endpoint, json=data, headers=headers, verify=True).json()
display(response)

Undeploy the pipeline via curl.

curl {wl.api_endpoint}/v1/api/pipelines/undeploy \
    -H "Authorization: {wl.auth.auth_header()['Authorization']}" \
    -H "Content-Type: application/json" \
    --data '{{ \
        "pipeline_id": {model_pipeline_id}, \
        "deployment_id": {model_deployment_id} \
}}'
null

Copy a Pipeline

Copies an existing pipeline into a new one in the same workspace. A new engine configuration can be set for the copied pipeline.

Copy a Pipeline Parameters

Copy a Pipeline Returns

  • Parameters
    • name - (REQUIRED string): The name of the new pipeline.
    • workspace_id - (REQUIRED int): The numerical id of the workspace to copy the source pipeline from.
    • source_pipeline - (REQUIRED int): The numerical id of the pipeline to copy from.
    • deploy - (OPTIONAL string): Name of the deployment.
    • engine_config - (OPTIONAL string): Engine configuration options.
    • pipeline_version - (OPTIONAL string): Optional version of the copied pipeline to create.

Copy a Pipeline Examples

The pipeline with models created in the step Create Pipeline in a Workspace will be copied into a new one.

Copy a pipeline via Requests.
## Copy a pipeline

# Retrieve the token 
headers = wl.auth.auth_header()

endpoint = f"{wl.api_endpoint}/v1/api/pipelines/copy"

data = {
  "name": "api-copied-pipeline-requests",
  "workspace_id": example_workspace_id,
  "source_pipeline": model_pipeline_id
}

response = requests.post(endpoint, json=data, headers=headers, verify=True).json()
response
    {'pipeline_pk_id': 9,
     'pipeline_variant_pk_id': 9,
     'pipeline_version': None,
     'deployment': None}

Copy a pipeline via curl.

curl {wl.api_endpoint}/v1/api/pipelines/copy \
    -H "Authorization: {wl.auth.auth_header()['Authorization']}" \
    -H "Content-Type: application/json" \
    --data '{{ \
        "name": "api-copied-pipeline-curl", \
        "workspace_id": {example_workspace_id}, \
        "source_pipeline": {model_pipeline_id} \
}}'
{
  "pipeline_pk_id": 32,
  "pipeline_variant_pk_id": 32,
  "pipeline_version": null,
  "deployment": null
}

7 - Wallaroo MLOps API Essentials Guide: Pipeline Edge Publishing Management

How to use the Wallaroo API for Pipeline Edge Publishing Management

Wallaroo pipelines can be published to a Edge Open Container Initiative (OCI) Registry Service, known here as the Edge Registry Service, as a container images. This allows the Wallaroo pipelines to be deployed in other environments, such as Docker or Kubernetes with all of the pipeline model. When deployed, these pipelines can perform inferences from the ML models exactly as if they were deployed as part of a Wallaroo instance.

When a pipeline is updated with new model steps or deployment configurations, the updated pipeline is republished to the Edge Registry as a new repo and version. This allows DevOps engineers to update an Wallaroo pipeline in any container supporting environment with the new versions of the pipeline.

Pipeline Publishing Flow

A typical ML Model and Pipeline deployment to Wallaroo Ops and to remote locations as a Wallaroo Inference server is as follows:

  • Components:
    • Wallaroo Ops: The Wallaroo Ops provides the backbone services for ML Model deployment. This is where ML models are uploaded, pipelines created and deployed for inferencing, pipelines published to OCI compliant registries, and other functions.
    • Wallaroo Inference Server: A remote deployment of a published Wallaroo pipeline with the Wallaroo Inference Engine outside the Wallaroo Ops instance. When the edge name is added to a Wallaroo publish, the Wallaroo Inference Server’s inference logs are submitted to the Wallaroo Ops instance. These inference logs are stored as part of the Wallaroo pipeline the remote deployment is published from.
  1. DevOps:
    1. Add Edge Publishing and Edge Observability to the Wallaroo Ops center. See Edge Deployment Registry Guide for details on updating the Wallaroo instance with Edge Publishing and Edge Observability.
  2. Data Scientists:
    1. Develop and train models.
    2. Test their deployments in Wallaroo Ops Center as Pipelines with:
      1. Pipeline Steps: The models part of the inference flow.
      2. Pipeline Deployment Configurations: CPUs, RAM, GPU, and Architecture settings to run the pipeline.
    3. Publish the Pipeline from the Wallaroo Ops to an OCI Registry: Store a image version of the Pipeline with models and pipeline configuration into the OCI Registry set by the DevOps engineers as the Wallaroo Edge Registry Service.
  3. DevOps:
    1. Retrieve the new or updated Wallaroo published pipeline from the Wallaroo Edge Registry Service.
    2. (Optional): Add an edge to the Wallaroo publish. This provides the EDGE_BUNDLE with the credentials for the Wallaroo Inference Server to transmit its inference result logs back to the Wallaroo Ops instance. These inference logs are added to the originating Wallaroo pipeline, labeled with the metadata.partition being the name of the edge deployed Wallaroo Inference server. For more details, see Wallaroo SDK Essentials Guide: Pipeline Edge Publication: Edge Observability
    3. Deploy the Pipeline as a Wallaroo Inference Server as a Docker or Kubernetes container, updating the resource allocations as part of the Helm chart, Docker Compose file, etc.

Enable Wallaroo Edge Registry

Set Edge Registry Service

Wallaroo Pipeline Publishes aka Wallaroo Servers are automatically routed to the Edge Open Container Initiative (OCI) Registry Service registered in the Wallaroo instance. This is enabled through either the Wallaroo Administrative Dashboard through kots, or by enabling it through a helm chart setting. From here on out, we will refer to it as the Edge Registry Service.

Set Edge Registry Service through Kots

To set the Edge Registry Settings through the Wallaroo Administrative Dashboard:

  1. Launch the Wallaroo Administrative Dashboard using the following command, replacing the --namespace parameter with the Kubernetes namespace for the Wallaroo instance:

    kubectl kots admin-console --namespace wallaroo
    
  2. Open a browser at the URL detailed in the step above and authenticate using the console password set as described in the as detailed in the Wallaroo Install Guides.

  3. From the top menu, select Config then scroll to Edge Deployment.

  4. Enable Provide OCI registry credentials for pipelines.

  5. Enter the following:

    Edge deployment registry service details
    1. Registry URL: The address of the registry service. For example: us-west1-docker.pkg.dev.
    2. email: The email address of the user account used to authenticate to the service.
    3. username: The account used to authenticate to the registry service.
    4. password: The password or token used to authenticate to the registry service.
  6. Save the updated configuration, then deploy it. Once complete, the edge registry settings will be available.

Set Edge Registry Service through Helm

The helm settings for adding the Edge Server configuration details are set through the ociRegistry element, with the following settings.

  • ociRegistry: Sets the Edge Server registry information.
    • enabled: true enables the Edge Server registry information, false disables it.
    • registry: The registry url. For example: reg.big.corp:3579.
    • repository: The repository within the registry. This may include the cloud account, or the full path where the Wallaroo published pipelines should be kept. For example: account123/wallaroo/pipelines.
    • email: Optional field to track the email address of the registry credential.
    • username: The username to the registry. This may vary based on the provider. For example, GCP Artifact Registry with service accounts uses the username _json_key_base64 with the password as a base64 processed token of the credential information.
    • password: The password or token for the registry service.

Set Edge Observability Service

Edge Observability allows published Wallaroo Servers to community with the Wallaroo Ops center to update their associated Wallaroo Pipeline with inference results, visible in the Pipeline logs.

This process will create a new Kubernetes service edge-lb. Based on the configuration options below, the service will require an additional IP address separate from the Wallaroo service api-lb. The edge-lb will require a DNS hostname.

Set Edge Observability Service through Kots

To enable Edge Observability using the Wallaroo Administrative Dashboard for kots installed instances of Wallaroo Ops:

  1. Launch the Wallaroo Administrative Dashboard using the following command, replacing the --namespace parameter with the Kubernetes namespace for the Wallaroo instance:

    kubectl kots admin-console --namespace wallaroo
    
  2. Open a browser at the URL detailed in the step above and authenticate using the console password set as described in the as detailed in the Wallaroo Install Guides.

  3. Access Config and scroll to Edge Deployment and enable Enable pipelines deployed on the edge to send data back to the OpsCenter.

  4. Set the following:

    Edge deployment registry service details
    1. Specify the OpsCenter hostname or IP address, as reachable from edge sites.: Set the DNS address in the format https://service.{suffix domain}. For example, if the domain suffix is wallaroo.example.com and the Wallaroo Edge Observabilty Service is set to the hostname edge, then the URL to access the edge service is:

      edge.wallaroo.example.com
      
    2. Edge ingress mode: Set one of the following.

      1. None - Services are cluster local and kubernetes port forwarding must be used for access.
      2. Internal - Private network users can connect directly and do not need to port forward anything.
      3. External - Internet facing users can connect directly to interactive Wallaroo services. Exercise caution.
  5. Save the updated configuration, then deploy it. Once complete, the edge observability service is available.

  6. Set the DNS Hostname as described in the steps Set Edge Observability Service DNS Hostname.

Set Edge Observability Service through Helm

To enable the Edge Observability Service for Wallaroo Ops Helm based installation, include the following variables for the helm settings. For these instructions they are stored in local-values.yaml:

edgelb:
    serviceType: LoadBalancer
    enabled: true
    opscenterHost: mitch4.edge.wallaroocommunity.ninja

pipelines:
    enabled: true

Update the Wallaroo Helm installation with the same version as the Wallaroo ops and the channel. For example, if updating Wallaroo Enterprise server, use the following:

helm upgrade wallaroo oci://registry.replicated.com/wallaroo/ee/wallaroo --version 2023.4.0-4092 --values local-values.yaml

This process will take 5-15 minutes depending on other configuration options. Once complete, set the DNS address as described in Set Edge Observability Service DNS Hostname.

Set Edge Observability Service DNS Hostname

Once enabled, the Wallaroo Edge Observability Service requires a DNS address. The following instructions are specified for Edge ingress mode:External.

  1. Obtain the external IP address of the the Wallaroo Edge Observability Service with the following command, replacing the -n wallaroo namespace option with the one the Wallaroo Ops instance is installed into.

    EDGE_LOADBALANCER=$(kubectl get svc edge-lb -n wallaroo -o jsonpath='{.status.loadBalancer.ingress[0].ip}') && echo $EDGE_LOADBALANCER
    
  2. Set the DNS address to the hostname set in the step Set Edge Observability Service through Kots if using kots to install, or Set Edge Observability Service through Helm if using helm.

Registry Setup Guides

The following are short guides for setting up the credentials for different registry services. Refer to the registry documentation for full details.

The following process is used with a GitHub Container Registry to create the authentication tokens for use with a Wallaroo instance’s Private Model Registry configuration.

See the GitHub Working with the Container registry for full details.

The following process is used register a GitHub Container Registry with Wallaroo.

  1. Create a new token as per the instructions from the Creating a personal access token (classic) guide. Note that a classic token is recommended for this process. Store this token in a secure location as it will not be able to be retrieved later from GitHub. Verify the following permissions are set:

    1. Select the write:packages scope to download and upload container images and read and write their metadata.

    2. Select the read:packages scope to download container images and read their metadata (selected when write:packages is selected by default).

    3. Select the delete:packages scope to delete container images.

  2. Store the token in a secure location.

This can be tested with docker by logging into the specified registry. For example:

docker login -u {Github Username} --password {Your Token} ghcr.io/{Your Github Username or Organization}

The following process is an example of setting up an Artifact Registry Service with Google Cloud Platform (GCP) that is used to store containerized model images and retrieve them for use with Wallaroo.

Uploading and downloading containerized models to a Google Cloud Platform Registry follows these general steps.

  • Create the GCP registry.

  • Create a Service Account that will manage the registry service requests.

  • Assign appropriate Artifact Registry role to the Service Account

  • Retrieve the Service Account credentials.

  • Using either a specific user, or the Service Account credentials, upload the containerized model to the registry service.

  • Add the service account credentials to the Wallaroo instance’s containerized model private registry configuration.

  • Prerequisites

The commands below use the Google gcloud command line tool, and expect that a Google Cloud Platform account is created and the gcloud application is associated with the GCP Project for the organization.

For full details on the process and other methods, see the Google GCP documentation.

  • Create the Registry

The following is based on the Create a repository using the Google Cloud CLI.

The following information is needed up front:

  • $REPOSITORY_NAME: What to call the registry.
  • $LOCATION: Where the repository will be located. GCP locations are derived through the gcloud artifacts locations list command.
  • $DESCRIPTION: Any details to be displayed. Sensitive data should not be included.

The follow example script will create a GCP registry with the minimum requirements.

REPOSITORY_NAME="YOUR NAME"
LOCATION="us-west1"
DESCRIPTION="My amazing registry."

gcloud artifacts repositories create REPOSITORY \
    --repository-format=docker \
    --location=LOCATION \
    --description="$DESCRIPTION" \
    --async
  • Create a GCP Registry Service Account

The GCP Registry Service Account is used to manage the GCP registry service. The steps are details from the Google Create a service account guide.

The gcloud process for these steps are:

  1. Connect the gcloud application to the organization’s project.

    $PROJECT_ID="YOUR PROJECT ID"
    gcloud config set project $PROJECT_ID
    
  2. Create the service account with the following:

    1. The name of the service account.
    2. A description of its purpose.
    3. The name to show when displayed.
    SA_NAME="YOUR SERVICE ACCOUNT NAME"
    DESCRIPTION="Wallaroo container registry SA"
    DISPLAY_NAME="Wallaroo the Roo"
    
    gcloud iam service-accounts create $SA_NAME \
    --description=$DESCRIPTION \
    --display-name=$DISPLAY_NAME
    
  • Assign Artifact Registry Role

Assign one or more of the following accounts to the new registry role based on the following criteria, as detailed in the Google GCP Repository Roles and Permissions Guide.

  • For pkg.dev domains.
RoleDescription
Artifact Registry Reader (roles/artifactregistry.reader)View and get artifacts, view repository metadata.
Artifact Registry Writer (roles/artifactregistry.writer)Read and write artifacts.
Artifact Registry Repository Administrator (roles/artifactregistry.repoAdmin)Read, write, and delete artifacts.
Artifact Registry Administrator (roles/artifactregistry.admin)Create and manage repositories and artifacts.
  • For gcr.io repositories.
RoleDescription
Artifact Registry Create-on-push Writer (roles/artifactregistry.createOnPushWriter)Read and write artifacts. Create gcr.io repositories.
Artifact Registry Create-on-push Repository Administrator (roles/artifactregistry.createOnPushRepoAdmin)Read, write, and delete artifacts. Create gcr.io repositories.

For this example, we will add the Artifact Registry Create-on-push Writer to the created Service Account from the previous step.

  1. Add the role to the service account, specifying the member as the new service account, and the role as the selected role. For this example, a pkg.dev is assumed for the Artifact Registry type.

    
    # for pkg.dev
    ROLE="roles/artifactregistry.writer"
    
    # for gcr.io 
    #ROLE="roles/artifactregistry.createOnPushWriter
    
    gcloud projects add-iam-policy-binding \
        $PROJECT_ID \
        --member="serviceAccount:$SA_NAME@$PROJECT_ID.iam.gserviceaccount.com" \
        --role=$ROLE
    
  • Authenticate to Repository

To push and pull image from the new registry, we’ll use our new service account and authenticate through the local Docker application. See the GCP Push and pull images for details on using Docker and other methods to add artifacts to the GCP artifact registry.

  • Set up Service Account Key

To set up the Service Account key, we’ll use the Google Console IAM & ADMIN dashboard based on the Set up authentication for Docker, using the JSON key approach.

  1. From GCP console, search for IAM & Admin.

  2. Select Service Accounts.

    Service account page
  3. Select the service account to generate keys for.

  4. Select the Email address listed and store this for later steps with the key generated through this process.

    Service account email
  5. Select Keys, then Add Key, then Create new key.

    Create service account key
  6. Select JSON, then Create.

  7. Store the key in a safe location.

  • Convert SA Key to Base64

The key file downloaded in Set up Service Account Key needs to be converted to base64 with the following command, replacing the locations of KEY_FILE and KEYFILEBASE64:

KEY_FILE = ~/.gcp-sa-registry-keyfile.json
KEYFILEBASE64 = ~/.gcp-sa-registry-keyfile-b64.json
base64 -i $KEY_FILE -o $KEYFILEBASE64

This base64 key is then used as the authentication token, with the username _json_key_base64.

This can be tested with docker by logging into the specified registry. For example:

token=$(cat $KEYFILEBASE64)
cat $tok | docker login -u _json_key_base64 --password-stdin https://{GCP artifact registry region}.pkg.dev

Pipeline Publish Endpoints

Publish a Pipeline

Pipelines are published as images to the edge registry set in the Enable Wallaroo Edge Registry through the following endpoint.

  • Endpoint:
    • /v1/api/pipelines/publish
  • Parameters
    • pipeline_id (Integer Required): The numerical id of the pipeline to publish to the edge registry.
    • pipeline_version_id (Integer Required): The numerical id of the pipeline’s version to publish to the edge registry.
    • model_config_ids (List Optional): The list of model config ids to include.
  • Returns
    • id (Integer): Numerical Wallaroo id of the published pipeline.
    • pipeline_version_id (Integer): Numerical Wallaroo id of the pipeline version published.
    • status: The status of the pipeline publish. Values include:
      • PendingPublish: The pipeline publication is about to be uploaded or is in the process of being uploaded.
      • Published: The pipeline is published and ready for use.
    • created_at (String): When the published pipeline was created.
    • updated_at (String): When the published pipeline was updated.
    • created_by (String): The email address of the Wallaroo user that created the pipeline publish.
    • pipeline_url (String): The URL of the published pipeline in the edge registry. May be null until the status is Published)
    • engine_url (String): The URL of the published pipeline engine in the edge registry. May be null until the status is Published.
    • helm: The Helm chart information including the following fields:
      • reference (String): The Helm reference.
      • chart (String): The Helm chart URL.
      • version (String): The Helm chart version.
    • engine_config (*wallaroo.deployment_config.DeploymentConfig) | The pipeline configuration included with the published pipeline.

Publish a Pipeline Example

The following example shows how to publish a pipeline to the edge registry service associated with the Wallaroo instance.

# get the pipeline version
pipeline_version_id = pipeline.versions()[-1].id()

pipeline_id = pipeline.id()

headers = {
    "Authorization": wl.auth._bearer_token_str(),
    "Content-Type": "application/json",
}

url = f"{wl.api_endpoint}/v1/api/pipelines/publish"
response = requests.post(
    url,
    headers=headers,
    json={"pipeline_id": 1, 
          "pipeline_version_id": pipeline_version, 
          "model_config_ids": []
          }
    )
display(response.json())

{'id': 10,
 'pipeline_version_id': 30,
 'pipeline_version_name': 'ab7ab48f-9aab-4c3d-9f66-097477bd34bf',
 'status': 'PendingPublish',
 'created_at': '2023-08-28T14:54:54.14098+00:00',
 'updated_at': '2023-08-28T14:54:54.14098+00:00',
 'created_by': 'db364f8c-b866-4865-96b7-0b65662cb384',
 'pipeline_url': None,
 'engine_url': None,
 'helm': None,
 'engine_config': {'engine': {'resources': {'limits': {'cpu': 4.0,
     'memory': '3Gi'},
    'requests': {'cpu': 4.0, 'memory': '3Gi'}}},
  'enginelb': {'resources': {'limits': {'cpu': 1.0, 'memory': '512Mi'},
    'requests': {'cpu': 0.2, 'memory': '512Mi'}}},
  'engineAux': {}}}

Get Publish Status

The status of a created Wallaroo pipeline publish is available through the following endpoint.

  • Endpoint:
    • /v1/api/pipelines/get_publish_status
  • Parameters
    • publish_id (Integer Required): The numerical id of the pipeline publish to retrieve.
  • Returns
    • id (Integer): Numerical Wallaroo id of the published pipeline.
    • pipeline_version_id (Integer): Numerical Wallaroo id of the pipeline version published.
    • status: The status of the pipeline publish. Values include:
      • PendingPublish: The pipeline publication is about to be uploaded or is in the process of being uploaded.
      • Published: The pipeline is published and ready for use.
    • created_at (String): When the published pipeline was created.
    • updated_at (String): When the published pipeline was updated.
    • created_by (String): The email address of the Wallaroo user that created the pipeline publish.
    • pipeline_url (String): The URL of the published pipeline in the edge registry. May be null until the status is Published)
    • engine_url (String): The URL of the published pipeline engine in the edge registry. May be null until the status is Published.
    • helm: The Helm chart information including the following fields:
      • reference (String): The Helm reference.
      • chart (String): The Helm chart URL.
      • version (String): The Helm chart version.
    • engine_config (*wallaroo.deployment_config.DeploymentConfig) | The pipeline configuration included with the published pipeline.

Get Publish Status Example

The following example shows retrieving the status of a recently created pipeline publish.

# Publish Status

headers = {
    "Authorization": wl.auth._bearer_token_str(),
    "Content-Type": "application/json",
}
url = f"{wl.api_endpoint}/v1/api/pipelines/get_publish_status"
response = requests.post(
    url,
    headers=headers,
    json={
        "id": publish_id
        },
)
display(response.json())

{'id': 10,
 'pipeline_version_id': 30,
 'pipeline_version_name': 'ab7ab48f-9aab-4c3d-9f66-097477bd34bf',
 'status': 'Published',
 'created_at': '2023-08-28T14:54:54.14098+00:00',
 'updated_at': '2023-08-28T14:54:54.14098+00:00',
 'created_by': 'db364f8c-b866-4865-96b7-0b65662cb384',
 'pipeline_url': 'ghcr.io/wallaroolabs/doc-samples/pipelines/edge-pipeline:ab7ab48f-9aab-4c3d-9f66-097477bd34bf',
 'engine_url': 'ghcr.io/wallaroolabs/doc-samples/engine:v2023.3.0-main-3731',
 'helm': {'reference': 'ghcr.io/wallaroolabs/doc-samples/charts@sha256:dbb46f807dbd75840cf1433daee51f4cc272791da5836366bfb8b2b863abf63b',
  'chart': 'ghcr.io/wallaroolabs/doc-samples/charts/edge-pipeline',
  'version': '0.0.1-ab7ab48f-9aab-4c3d-9f66-097477bd34bf'},
 'engine_config': {'engine': {'resources': {'limits': {'cpu': 4.0,
     'memory': '3Gi'},
    'requests': {'cpu': 4.0, 'memory': '3Gi'}}},
  'enginelb': {'resources': {'limits': {'cpu': 1.0, 'memory': '512Mi'},
    'requests': {'cpu': 0.2, 'memory': '512Mi'}}},
  'engineAux': {}}}

List Publishes for a Pipeline

A list of publishes created for a specific pipeline is retrieved through the following endpoint.

  • Endpoint
    • /v1/api/pipelines/list_publishes_for_pipeline
  • Parameters
    • publish_id (Integer Required): The numerical id of the pipeline to retrieve all publishes for.
  • Returns a List of pipeline publishes with the following fields:
    • id (Integer): Numerical Wallaroo id of the published pipeline.
    • pipeline_version_id (Integer): Numerical Wallaroo id of the pipeline version published.
    • status: The status of the pipeline publish. Values include:
      • PendingPublish: The pipeline publication is about to be uploaded or is in the process of being uploaded.
      • Published: The pipeline is published and ready for use.
    • created_at (String): When the published pipeline was created.
    • updated_at (String): When the published pipeline was updated.
    • created_by (String): The email address of the Wallaroo user that created the pipeline publish.
    • pipeline_url (String): The URL of the published pipeline in the edge registry. May be null until the status is Published)
    • engine_url (String): The URL of the published pipeline engine in the edge registry. May be null until the status is Published.
    • helm: The Helm chart information including the following fields:
      • reference (String): The Helm reference.
      • chart (String): The Helm chart URL.
      • version (String): The Helm chart version.
    • engine_config (*wallaroo.deployment_config.DeploymentConfig) | The pipeline configuration included with the published pipeline.

List Publishes for a Pipeline Example

The following retrieves a list of all publishes for a specific pipeline.

headers = {
    "Authorization": wl.auth._bearer_token_str(),
    "Content-Type": "application/json",
}

url = f"{wl.api_endpoint}/v1/api/pipelines/list_publishes_for_pipeline"

response = requests.post(
    url,
    headers=headers,
    json={"pipeline_id": pipeline_id},
)
display(response.json())

{'pipelines': [{'id': 10,
   'pipeline_version_id': 30,
   'pipeline_version_name': 'ab7ab48f-9aab-4c3d-9f66-097477bd34bf',
   'status': 'Published',
   'created_at': '2023-08-28T14:54:54.14098+00:00',
   'updated_at': '2023-08-28T14:54:54.14098+00:00',
   'created_by': 'db364f8c-b866-4865-96b7-0b65662cb384',
   'pipeline_url': 'ghcr.io/wallaroolabs/doc-samples/pipelines/edge-pipeline:ab7ab48f-9aab-4c3d-9f66-097477bd34bf',
   'engine_url': 'ghcr.io/wallaroolabs/doc-samples/engine:v2023.3.0-main-3731',
   'helm': {'reference': 'ghcr.io/wallaroolabs/doc-samples/charts@sha256:dbb46f807dbd75840cf1433daee51f4cc272791da5836366bfb8b2b863abf63b',
    'chart': 'ghcr.io/wallaroolabs/doc-samples/charts/edge-pipeline',
    'version': '0.0.1-ab7ab48f-9aab-4c3d-9f66-097477bd34bf'},
   'engine_config': {'engine': {'resources': {'limits': {'cpu': 4.0,
       'memory': '3Gi'},
      'requests': {'cpu': 4.0, 'memory': '3Gi'}}},
    'enginelb': {'resources': {'limits': {'cpu': 1.0, 'memory': '512Mi'},
      'requests': {'cpu': 0.2, 'memory': '512Mi'}}},
    'engineAux': {}}}]}

Edge Observability

Edge Observability allows edge deployments of Wallaroo Server to transmit inference results back to the Wallaroo Ops center and become part of the pipeline’s logs. This is valuable for data scientists and MLOps engineers to retrieve edge deployment logs for use in model observability, drift, and other use cases.

Before starting, the Edge Observability Service must be enabled in the Wallaroo Ops center. See the Edge Deployment Registry Guide for details on enabling the Wallaroo Edge Deployment service.

Wallaroo Server edge observability is enabled when a new edge location is added to the pipeline publish. Each location has its own EDGE_BUNDLE settings, a Base64 encoded set of instructions informing the edge deployed Wallaroo Server on how to communicate with Wallaroo Ops center.

Add Publish Edge

Edges are added to an existing pipeline publish with the following endpoint.

  • Endpoint
    • /v1/api/pipelines/add_edge_to_publish
  • Parameters
    • name (String Required): The name of the edge. This must be a unique value across all edges in the Wallaroo instance.
    • pipeline_publish_id (Integer Required): The numerical identifier of the pipeline publish to add this edge to.
    • tags (List[String] Optional): A list of optional tags.
  • Returns
    • id (Integer): The integer ID of the pipeline publish.
    • created_at: (String): The DateTime of the pipeline publish.
    • docker_run_variables (String) The Docker variables in base64 encoded format that include the following: The BUNDLE_VERSION, EDGE_NAME, JOIN_TOKEN_, OPSCENTER_HOST, PIPELINE_URL, and WORKSPACE_ID.
    • engine_config (String): The Wallaroo wallaroo.deployment_config.DeploymentConfig for the pipeline.
    • pipeline_version_id (Integer): The integer identifier of the pipeline version published.
    • status (String): The status of the publish. Published is a successful publish.
    • updated_at (DateTime): The DateTime when the pipeline publish was updated.
    • user_images (List[String]): User images used in the pipeline publish.
    • created_by (String): The UUID of the Wallaroo user that created the pipeline publish.
    • engine_url (String): The URL for the published pipeline’s Wallaroo engine in the OCI registry.
    • error (String): Any errors logged.
    • helm (String): The helm chart, helm reference and helm version.
    • pipeline_url (String): The URL for the published pipeline’s container in the OCI registry.
    • pipeline_version_name (String): The UUID identifier of the pipeline version published.
    • additional_properties (String): Any additional properties.

Add Publish Edge Example

headers = {
    "Authorization": wl.auth._bearer_token_str(),
    "Content-Type": "application/json",
}

edge_name = f"ccfraud-observability-api{suffix_random}"

url = f"{wl.api_endpoint}/v1/api/pipelines/add_edge_to_publish"

response = requests.post(
    url,
    headers=headers,
    json={
        "pipeline_publish_id": publish_id,
        "name": edge_name,
        "tags": []
    }
)
display(response.json())
{'id': 9,
 'pipeline_version_id': 17,
 'pipeline_version_name': 'cd70ed84-2db7-4fc5-a766-482686ca3fcc',
 'status': 'Published',
 'created_at': '2023-10-30T19:04:16.758117+00:00',
 'updated_at': '2023-10-30T19:04:16.758117+00:00',
 'created_by': '1b26fc10-d0f9-4f92-a1a2-ab342b3d069e',
 'pipeline_url': 'ghcr.io/wallaroolabs/doc-samples/pipelines/edge-pipeline:cd70ed84-2db7-4fc5-a766-482686ca3fcc',
 'engine_url': 'ghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/standalone-mini:v2023.4.0-main-4079',
 'helm': {'reference': 'ghcr.io/wallaroolabs/doc-samples/charts@sha256:6f557685fb6e7509dcee4e09af1ece94df1dcc9886ef19a9adc6c015e9a7279b',
  'chart': 'ghcr.io/wallaroolabs/doc-samples/charts/edge-pipeline',
  'version': '0.0.1-cd70ed84-2db7-4fc5-a766-482686ca3fcc',
  'values': {'edgeBundle': 'ZXhwb3J0IEJVTkRMRV9WRVJTSU9OPTEKZXhwb3J0IEVER0VfTkFNRT1jY2ZyYXVkLW9ic2VydmFiaWxpdHktYXBpZnpkdgpleHBvcnQgSk9JTl9UT0tFTj1kYTM2Njk3Zi1iYWIyLTQyYTktYmNmNy03MTk5YWU0YWVkZjkKZXhwb3J0IE9QU0NFTlRFUl9IT1NUPWRvYy10ZXN0LmVkZ2Uud2FsbGFyb29jb21tdW5pdHkubmluamEKZXhwb3J0IFBJUEVMSU5FX1VSTD1naGNyLmlvL3dhbGxhcm9vbGFicy9kb2Mtc2FtcGxlcy9waXBlbGluZXMvZWRnZS1waXBlbGluZTpjZDcwZWQ4NC0yZGI3LTRmYzUtYTc2Ni00ODI2ODZjYTNmY2MKZXhwb3J0IFdPUktTUEFDRV9JRD05'}},
 'engine_config': {'engine': {'resources': {'limits': {'cpu': 4.0,
     'memory': '3Gi'},
    'requests': {'cpu': 4.0, 'memory': '3Gi'}}},
  'enginelb': {'resources': {'limits': {'cpu': 1.0, 'memory': '512Mi'},
    'requests': {'cpu': 0.2, 'memory': '512Mi'}}},
  'engineAux': {}},
 'error': None,
 'user_images': [],
 'docker_run_variables': {'EDGE_BUNDLE': 'ZXhwb3J0IEJVTkRMRV9WRVJTSU9OPTEKZXhwb3J0IEVER0VfTkFNRT1jY2ZyYXVkLW9ic2VydmFiaWxpdHktYXBpZnpkdgpleHBvcnQgSk9JTl9UT0tFTj1kYTM2Njk3Zi1iYWIyLTQyYTktYmNmNy03MTk5YWU0YWVkZjkKZXhwb3J0IE9QU0NFTlRFUl9IT1NUPWRvYy10ZXN0LmVkZ2Uud2FsbGFyb29jb21tdW5pdHkubmluamEKZXhwb3J0IFBJUEVMSU5FX1VSTD1naGNyLmlvL3dhbGxhcm9vbGFicy9kb2Mtc2FtcGxlcy9waXBlbGluZXMvZWRnZS1waXBlbGluZTpjZDcwZWQ4NC0yZGI3LTRmYzUtYTc2Ni00ODI2ODZjYTNmY2MKZXhwb3J0IFdPUktTUEFDRV9JRD05'}}

Remove Publish Edge

Edges are removed from an existing pipeline publish with the following endpoint.

  • Endpoint
    • /v1/api/pipelines/remove_edge_from_publish
  • Parameters
    • name (String Required): The name of the edge. This must be a unique value across all edges in the Wallaroo instance.
  • Returns
    • null

Remove Publish Edge Example

The following example will add an edge to a pipeline publish, then remove the same edge.

# adding another edge to remove

headers = {
    "Authorization": wl.auth._bearer_token_str(),
    "Content-Type": "application/json",
}

edge_name2 = f"ccfraud-observability-api2{suffix_random}"

url = f"{wl.api_endpoint}/v1/api/pipelines/add_edge_to_publish"

response = requests.post(
    url,
    headers=headers,
    json={
        "pipeline_publish_id": publish_id,
        "name": edge_name2,
        "tags": []
    }
)
display(response.json())
{'id': 10,
 'pipeline_version_id': 17,
 'pipeline_version_name': 'cd70ed84-2db7-4fc5-a766-482686ca3fcc',
 'status': 'Published',
 'created_at': '2023-10-30T19:50:32.164154+00:00',
 'updated_at': '2023-10-30T19:50:32.164154+00:00',
 'created_by': '1b26fc10-d0f9-4f92-a1a2-ab342b3d069e',
 'pipeline_url': 'ghcr.io/wallaroolabs/doc-samples/pipelines/edge-pipeline:cd70ed84-2db7-4fc5-a766-482686ca3fcc',
 'engine_url': 'ghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/standalone-mini:v2023.4.0-main-4079',
 'helm': {'reference': 'ghcr.io/wallaroolabs/doc-samples/charts@sha256:cfd0e77347ec46b6d3dec2c2cc83733c353c5b583428b5c6f4fca11277a2c43e',
  'chart': 'ghcr.io/wallaroolabs/doc-samples/charts/edge-pipeline',
  'version': '0.0.1-cd70ed84-2db7-4fc5-a766-482686ca3fcc',
  'values': {'edgeBundle': 'ZXhwb3J0IEJVTkRMRV9WRVJTSU9OPTEKZXhwb3J0IEVER0VfTkFNRT1jY2ZyYXVkLW9ic2VydmFiaWxpdHktYXBpMmtwd2wKZXhwb3J0IEpPSU5fVE9LRU49NWVmZmIyYzYtZTNmNi00MmY1LWFhYjAtMWE5YTNiN2YwMTIyCmV4cG9ydCBPUFNDRU5URVJfSE9TVD1kb2MtdGVzdC5lZGdlLndhbGxhcm9vY29tbXVuaXR5Lm5pbmphCmV4cG9ydCBQSVBFTElORV9VUkw9Z2hjci5pby93YWxsYXJvb2xhYnMvZG9jLXNhbXBsZXMvcGlwZWxpbmVzL2VkZ2UtcGlwZWxpbmU6Y2Q3MGVkODQtMmRiNy00ZmM1LWE3NjYtNDgyNjg2Y2EzZmNjCmV4cG9ydCBXT1JLU1BBQ0VfSUQ9OQ=='}},
 'engine_config': {'engine': {'resources': {'limits': {'cpu': 4.0,
     'memory': '3Gi'},
    'requests': {'cpu': 4.0, 'memory': '3Gi'}}},
  'enginelb': {'resources': {'limits': {'cpu': 1.0, 'memory': '512Mi'},
    'requests': {'cpu': 0.2, 'memory': '512Mi'}}},
  'engineAux': {}},
 'error': None,
 'user_images': [],
 'docker_run_variables': {'EDGE_BUNDLE': 'ZXhwb3J0IEJVTkRMRV9WRVJTSU9OPTEKZXhwb3J0IEVER0VfTkFNRT1jY2ZyYXVkLW9ic2VydmFiaWxpdHktYXBpMmtwd2wKZXhwb3J0IEpPSU5fVE9LRU49NWVmZmIyYzYtZTNmNi00MmY1LWFhYjAtMWE5YTNiN2YwMTIyCmV4cG9ydCBPUFNDRU5URVJfSE9TVD1kb2MtdGVzdC5lZGdlLndhbGxhcm9vY29tbXVuaXR5Lm5pbmphCmV4cG9ydCBQSVBFTElORV9VUkw9Z2hjci5pby93YWxsYXJvb2xhYnMvZG9jLXNhbXBsZXMvcGlwZWxpbmVzL2VkZ2UtcGlwZWxpbmU6Y2Q3MGVkODQtMmRiNy00ZmM1LWE3NjYtNDgyNjg2Y2EzZmNjCmV4cG9ydCBXT1JLU1BBQ0VfSUQ9OQ=='}}
# remove the edge

headers = {
    "Authorization": wl.auth._bearer_token_str(),
    "Content-Type": "application/json",
}

edge_name2 = f"ccfraud-observability-api2{suffix_random}"

url = f"{wl.api_endpoint}/v1/api/pipelines/remove_edge_from_publish"

response = requests.post(
    url,
    headers=headers,
    json={
        "name": edge_name2,
    }
)
display(response.json())
null

Edge Bundle Token TTL

When an edge is added to a pipeline publish, the field docker_run_variables contains a JSON value for edge devices to connect to the Wallaroo Ops instance. The settings are stored in the key EDGE_BUNDLE as a base64 encoded value that include the following:

  • BUNDLE_VERSION: The current version of the bundled Wallaroo pipeline.
  • EDGE_NAME: The edge name as defined when created and added to the pipeline publish.
  • JOIN_TOKEN_: The one time authentication token for authenticating to the Wallaroo Ops instance.
  • OPSCENTER_HOST: The hostname of the Wallaroo Ops edge service. See Edge Deployment Registry Guide for full details on enabling pipeline publishing and edge observability to Wallaroo.
  • PIPELINE_URL
  • WORKSPACE_ID.

For example:

{'edgeBundle': 'ZXhwb3J0IEJVTkRMRV9WRVJTSU9OPTEKZXhwb3J0IEVER0VfTkFNRT14Z2ItY2NmcmF1ZC1lZGdlLXRlc3QKZXhwb3J0IEpPSU5fVE9LRU49MzE0OGFkYTUtMjg1YS00ZmNhLWIzYjgtYjUwYTQ4ZDc1MTFiCmV4cG9ydCBPUFNDRU5URVJfSE9TVD1kb2MtdGVzdC5lZGdlLndhbGxhcm9vY29tbXVuaXR5Lm5pbmphCmV4cG9ydCBQSVBFTElORV9VUkw9Z2hjci5pby93YWxsYXJvb2xhYnMvZG9jLXNhbXBsZXMvcGlwZWxpbmVzL2VkZ2UtcGlwZWxpbmU6ZjM4OGMxMDktOGQ1Ny00ZWQyLTk4MDYtYWExM2Y4NTQ1NzZiCmV4cG9ydCBXT1JLU1BBQ0VfSUQ9NQ=='}
base64 -D
ZXhwb3J0IEJVTkRMRV9WRVJTSU9OPTEKZXhwb3J0IEVER0VfTkFNRT14Z2ItY2NmcmF1ZC1lZGdlLXRlc3QKZXhwb3J0IEpPSU5fVE9LRU49MzE0OGFkYTUtMjg1YS00ZmNhLWIzYjgtYjUwYTQ4ZDc1MTFiCmV4cG9ydCBPUFNDRU5URVJfSE9TVD1kb2MtdGVzdC5lZGdlLndhbGxhcm9vY29tbXVuaXR5Lm5pbmphCmV4cG9ydCBQSVBFTElORV9VUkw9Z2hjci5pby93YWxsYXJvb2xhYnMvZG9jLXNhbXBsZXMvcGlwZWxpbmVzL2VkZ2UtcGlwZWxpbmU6ZjM4OGMxMDktOGQ1Ny00ZWQyLTk4MDYtYWExM2Y4NTQ1NzZiCmV4cG9ydCBXT1JLU1BBQ0VfSUQ9NQ==^D
export BUNDLE_VERSION=1
export EDGE_NAME=xgb-ccfraud-edge-test
export JOIN_TOKEN=3148ada5-285a-4fca-b3b8-b50a48d7511b
export OPSCENTER_HOST=doc-test.edge.wallaroocommunity.ninja
export PIPELINE_URL=ghcr.io/wallaroolabs/doc-samples/pipelines/edge-pipeline:f388c109-8d57-4ed2-9806-aa13f854576b
export WORKSPACE_ID=5

The JOIN_TOKEN is a one time access token. Once used, a JOIN_TOKEN expires. The authentication session data is stored in persistent volumes. Persistent volumes must be specified for docker and docker compose based deployments of Wallaroo pipelines; helm based deployments automatically provide persistent volumes to store authentication credentials.

The JOIN_TOKEN has the following time to live (TTL) parameters.

  • Once created, the JOIN_TOKEN is valid for 24 hours. After it expires the edge will not be allowed to contact the OpsCenter the first time and a new edge bundle will have to be created.
  • After an Edge joins to Wallaroo Ops for the first time with persistent storage, the edge must contact the Wallaroo Ops instance at least once every 7 days.
    • If this period is exceeded, the authentication credentials will expire and a new edge bundle must be created with a new and valid JOIN_TOKEN.

Wallaroo edges require unique names. To create a new edge bundle with the same name:

  • Use the Remove Edge to remove the edge by name.
  • Use Add Edge to add the edge with the same name. A new EDGE_BUNDLE is generated with a new JOIN_TOKEN.

DevOps - Pipeline Edge Deployment

Once a pipeline is deployed to the Edge Registry service, it can be deployed in environments such as Docker, Kubernetes, or similar container running services by a DevOps engineer.

Docker Deployment

First, the DevOps engineer must authenticate to the same OCI Registry service used for the Wallaroo Edge Deployment registry.

For more details, check with the documentation on your artifact service. The following are provided for the three major cloud services:

For the deployment, the engine URL is specified with the following environmental variables:

  • DEBUG (true|false): Whether to include debug output.
  • OCI_REGISTRY: The URL of the registry service.
  • CONFIG_CPUS: The number of CPUs to use.
  • OCI_USERNAME: The edge registry username.
  • OCI_PASSWORD: The edge registry password or token.
  • PIPELINE_URL: The published pipeline URL.
  • EDGE_BUNDLE (Optional): The base64 encoded edge token and other values to connect to the Wallaroo Ops instance. This is used for edge management and transmitting inference results for observability. IMPORTANT NOTE: The token for EDGE_BUNDLE is valid for one deployment. For subsequent deployments, generate a new edge location with its own EDGE_BUNDLE.

Docker Deployment Example

Using our sample environment, here’s sample deployment using Docker with a computer vision ML model, the same used in the Wallaroo Use Case Tutorials Computer Vision: Retail tutorials.

  1. Login through docker to confirm access to the registry service. First, docker login. For example, logging into the artifact registry with the token stored in the variable tok:

    cat $tok | docker login -u _json_key_base64 --password-stdin https://sample-registry.com
    
  2. Then deploy the Wallaroo published pipeline with an edge added to the pipeline publish through docker run.

    IMPORTANT NOTE: Edge deployments with Edge Observability enabled with the EDGE_BUNDLE option include an authentication token that only authenticates once. To store the token long term, include the persistent volume flag -v {path to storage} setting.

    Deployment with EDGE_BUNDLE for observability.

    docker run -p 8080:8080 \
    -v ./data:/persist \
    -e DEBUG=true \
    -e OCI_REGISTRY=$REGISTRYURL \
    -e EDGE_BUNDLE=ZXhwb3J0IEJVTkRMRV9WRVJTSU9OPTEKZXhwb3J0IEVER0VfTkFNRT1lZGdlLWNjZnJhdWQtb2JzZXJ2YWJpbGl0eXlhaWcKZXhwb3J0IEpPSU5fVE9LRU49MjZmYzFjYjgtMjUxMi00YmU3LTk0ZGUtNjQ2NGI1MGQ2MzhiCmV4cG9ydCBPUFNDRU5URVJfSE9TVD1kb2MtdGVzdC5lZGdlLndhbGxhcm9vY29tbXVuaXR5Lm5pbmphCmV4cG9ydCBQSVBFTElORV9VUkw9Z2hjci5pby93YWxsYXJvb2xhYnMvZG9jLXNhbXBsZXMvcGlwZWxpbmVzL2VkZ2Utb2JzZXJ2YWJpbGl0eS1waXBlbGluZTozYjQ5ZmJhOC05NGQ4LTRmY2EtYWVjYy1jNzUyNTdmZDE2YzYKZXhwb3J0IFdPUktTUEFDRV9JRD03 \
    -e CONFIG_CPUS=1 \
    -e OCI_USERNAME=$REGISTRYUSERNAME \
    -e OCI_PASSWORD=$REGISTRYPASSWORD \
    -e PIPELINE_URL=ghcr.io/wallaroolabs/doc-samples/pipelines/edge-observability-pipeline:3b49fba8-94d8-4fca-aecc-c75257fd16c6 \
    ghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/standalone-mini:v2023.4.0-main-4079
    

    Connection to the Wallaroo Ops instance from edge deployment with EDGE_BUNDLE is verified with the long entry Node attestation was successful.

    Deployment without observability.

    docker run -p 8080:8080 \
    -e DEBUG=true \
    -e OCI_REGISTRY=$REGISTRYURL \
    -e CONFIG_CPUS=1 \
    -e OCI_USERNAME=$REGISTRYUSERNAME \
    -e OCI_PASSWORD=$REGISTRYPASSWORD \
    -e PIPELINE_URL=ghcr.io/wallaroolabs/doc-samples/pipelines/edge-observability-pipeline:3b49fba8-94d8-4fca-aecc-c75257fd16c6 \
    ghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/standalo
    

Docker Compose Deployment

For users who prefer to use docker compose, the following sample compose.yaml file is used to launch the Wallaroo Edge pipeline. This is the same used in the Wallaroo Use Case Tutorials Computer Vision: Retail tutorials. The volumes tag is used to preserve the login session from the one-time token generated as part of the EDGE_BUNDLE.

EDGE_BUNDLE is only required when adding an edge to a Wallaroo publish for observability. The following is deployed without observability.

services:
  engine:
    image: {Your Engine URL}
    ports:
      - 8080:8080
    environment:
      PIPELINE_URL: {Your Pipeline URL}
      OCI_REGISTRY: {Your Edge Registry URL}
      OCI_USERNAME:  {Your Registry Username}
      OCI_PASSWORD: {Your Token or Password}
      CONFIG_CPUS: 4

The procedure is:

  1. Login through docker to confirm access to the registry service. First, docker login. For example, logging into the artifact registry with the token stored in the variable tok to the registry us-west1-docker.pkg.dev:

    cat $tok | docker login -u _json_key_base64 --password-stdin https://sample-registry.com
    
  2. Set up the compose.yaml file.

    IMPORTANT NOTE: Edge deployments with Edge Observability enabled with the EDGE_BUNDLE option include an authentication token that only authenticates once. To store the token long term, include the persistent volume with the volumes: tag.

    services:
    engine:
        image: sample-registry.com/engine:v2023.3.0-main-3707
        ports:
            - 8080:8080
        volumes:
            - ./data:/persist
    environment:
        PIPELINE_URL: sample-registry.com/pipelines/edge-cv-retail:bf70eaf7-8c11-4b46-b751-916a43b1a555
        EDGE_BUNDLE: ZXhwb3J0IEJVTkRMRV9WRVJTSU9OPTEKZXhwb3J0IEVER0VfTkFNRT1lZGdlLWNjZnJhdWQtb2JzZXJ2YWJpbGl0eXlhaWcKZXhwb3J0IEpPSU5fVE9LRU49MjZmYzFjYjgtMjUxMi00YmU3LTk0ZGUtNjQ2NGI1MGQ2MzhiCmV4cG9ydCBPUFNDRU5URVJfSE9TVD1kb2MtdGVzdC5lZGdlLndhbGxhcm9vY29tbXVuaXR5Lm5pbmphCmV4cG9ydCBQSVBFTElORV9VUkw9Z2hjci5pby93YWxsYXJvb2xhYnMvZG9jLXNhbXBsZXMvcGlwZWxpbmVzL2VkZ2Utb2JzZXJ2YWJpbGl0eS1waXBlbGluZTozYjQ5ZmJhOC05NGQ4LTRmY2EtYWVjYy1jNzUyNTdmZDE2YzYKZXhwb3J0IFdPUktTUEFDRV9JRD03
        OCI_REGISTRY: sample-registry.com
        OCI_USERNAME:  _json_key_base64
        OCI_PASSWORD: abc123
        CONFIG_CPUS: 4
    
  3. Then deploy with docker compose up.

Docker Compose Deployment Example

The deployment and undeployment is then just a simple docker compose up and docker compose down. The following shows an example of deploying the Wallaroo edge pipeline using docker compose.

docker compose up
[+] Running 1/1
 ✔ Container cv_data-engine-1  Recreated                                                                                                                                                                 0.5s
Attaching to cv_data-engine-1
cv_data-engine-1  | Wallaroo Engine - Standalone mode
cv_data-engine-1  | Login Succeeded
cv_data-engine-1  | Fetching manifest and config for pipeline: sample-registry.com/pipelines/edge-cv-retail:bf70eaf7-8c11-4b46-b751-916a43b1a555
cv_data-engine-1  | Fetching model layers
cv_data-engine-1  | digest: sha256:c6c8869645962e7711132a7e17aced2ac0f60dcdc2c7faa79b2de73847a87984
cv_data-engine-1  |   filename: c6c8869645962e7711132a7e17aced2ac0f60dcdc2c7faa79b2de73847a87984
cv_data-engine-1  |   name: resnet-50
cv_data-engine-1  |   type: model
cv_data-engine-1  |   runtime: onnx
cv_data-engine-1  |   version: 693e19b5-0dc7-4afb-9922-e3f7feefe66d
cv_data-engine-1  |
cv_data-engine-1  | Fetched
cv_data-engine-1  | Starting engine
cv_data-engine-1  | Looking for preexisting `yaml` files in //modelconfigs
cv_data-engine-1  | Looking for preexisting `yaml` files in //pipelines

Helm Deployment

Published pipelines can be deployed through the use of helm charts.

Helm deployments take up to two steps - the first step is in retrieving the required values.yaml and making updates to override.

IMPORTANT NOTE: Edge deployments with Edge Observability enabled with the EDGE_BUNDLE option include an authentication token that only authenticates once. Helm chart installations automatically add a persistent volume during deployment to store the authentication session data for future deployments.

  1. Login to the registry service with helm registry login. For example, if the token is stored in the variable tok:

    helm registry login sample-registry.com --username _json_key_base64 --password $tok
    
  2. Pull the helm charts from the published pipeline. The two fields are the Helm Chart URL and the Helm Chart version to specify the OCI . This typically takes the format of:

    helm pull oci://{published.helm_chart_url} --version {published.helm_chart_version}
    
  3. Extract the tgz file and copy the values.yaml and copy the values used to edit engine allocations, etc. The following are required for the deployment to run:

    ociRegistry:
        registry: {your registry service}
        username:  {registry username here}
        password: {registry token here}
    

    For Wallaroo Server deployments with edge location set, the values include edgeBundle as generated when the edge was added to the pipeline publish.

    ociRegistry:
        registry: {your registry service}
        username:  {registry username here}
        password: {registry token here}
    edgeBundle: abcdefg
    

Store this into another file, suc as local-values.yaml.

  1. Create the namespace to deploy the pipeline to. For example, the namespace wallaroo-edge-pipeline would be:

    kubectl create -n wallaroo-edge-pipeline
    
  2. Deploy the helm installation with helm install through one of the following options:

    1. Specify the tgz file that was downloaded and the local values file. For example:

      helm install --namespace {namespace} --values {local values file} {helm install name} {tgz path}
      
    2. Specify the expended directory from the downloaded tgz file.

      helm install --namespace {namespace} --values {local values file} {helm install name} {helm directory path}
      
    3. Specify the Helm Pipeline Helm Chart and the Pipeline Helm Version.

      helm install --namespace {namespace} --values {local values file} {helm install name} oci://{published.helm_chart_url} --version {published.helm_chart_version}
      
  3. Once deployed, the DevOps engineer will have to forward the appropriate ports to the svc/engine-svc service in the specific pipeline. For example, using kubectl port-forward to the namespace ccfraud that would be:

    kubectl port-forward svc/engine-svc -n ccfraud01 8080 --address 0.0.0.0`
    

Edge Deployment Endpoints

The following endpoints are available for API calls to the edge deployed pipeline.

List Pipelines

The endpoint /pipelines returns:

  • id (String): The name of the pipeline.
  • status (String): The status as either Running, or Error if there are any issues.

List Pipelines Example

curl localhost:8080/pipelines
{"pipelines":[{"id":"edge-cv-retail","status":"Running"}]}

List Models

The endpoint /models returns a List of models with the following fields:

  • name (String): The model name.
  • sha (String): The sha hash value of the ML model.
  • status (String): The status of either Running or Error if there are any issues.
  • version (String): The model version. This matches the version designation used by Wallaroo to track model versions in UUID format.

List Models Example

curl localhost:8080/models
{"models":[{"name":"resnet-50","sha":"c6c8869645962e7711132a7e17aced2ac0f60dcdc2c7faa79b2de73847a87984","status":"Running","version":"693e19b5-0dc7-4afb-9922-e3f7feefe66d"}]}

Edge Inference Endpoint

The inference endpoint takes the following pattern:

  • /pipelines/{pipeline-name}: The pipeline-name is the same as returned from the /pipelines endpoint as id.

Wallaroo inference endpoint URLs accept the following data inputs through the Content-Type header:

  • Content-Type: application/vnd.apache.arrow.file: For Apache Arrow tables.
  • Content-Type: application/json; format=pandas-records: For pandas DataFrame in record format.

Once deployed, we can perform an inference through the deployment URL.

The endpoint returns Content-Type: application/json; format=pandas-records by default with the following fields:

  • check_failures (List[Integer]): Whether any validation checks were triggered. For more information, see Wallaroo SDK Essentials Guide: Pipeline Management: Anomaly Testing.
  • elapsed (List[Integer]): A list of time in nanoseconds for:
    • [0] The time to serialize the input.
    • [1…n] How long each step took.
  • model_name (String): The name of the model used.
  • model_version (String): The version of the model in UUID format.
  • original_data: The original input data. Returns null if the input may be too long for a proper return.
  • outputs (List): The outputs of the inference result separated by data type, where each data type includes:
    • data: The returned values.
    • dim (List[Integer]): The dimension shape returned.
    • v (Integer): The vector shape of the data.
  • pipeline_name (String): The name of the pipeline.
  • shadow_data: Any shadow deployed data inferences in the same format as outputs.
  • time (Integer): The time since UNIX epoch.

Edge Inference Endpoint Example

The following example demonstrates sending an Apache Arrow table to the Edge deployed pipeline, requesting the inference results back in a pandas DataFrame records format.

curl -X POST localhost:8080/pipelines/edge-cv-retail -H "Content-Type: application/vnd.apache.arrow.file" -H 'Accept: application/json; format=pandas-records'  --data-binary @./data/image_224x224.arrow

Returns:

[{"check_failures":[],"elapsed":[1067541,21209776],"model_name":"resnet-50","model_version":"2e05e1d0-fcb3-4213-bba8-4bac13f53e8d","original_data":null,"outputs":[{"Int64":{"data":[535],"dim":[1],"v":1}},{"Float":{"data":[0.00009498586587142199,0.00009141524787992239,0.0004606838047038764,0.00007667174941161647,0.00008047101437114179,...],"dim":[1,1001],"v":1}}],"pipeline_name":"edge-cv-demo","shadow_data":{},"time":1694205578428}]

Edge Bundle Token Time To Live

When an edge is added to a pipeline publish, the field docker_run_variables contains a JSON value for edge devices to connect to the Wallaroo Ops instance.

The settings are stored in the key EDGE_BUNDLE as a base64 encoded value that include the following:

  • BUNDLE_VERSION: The current version of the bundled Wallaroo pipeline.
  • EDGE_NAME: The edge name as defined when created and added to the pipeline publish.
  • JOIN_TOKEN_: The one time authentication token for authenticating to the Wallaroo Ops instance.
  • OPSCENTER_HOST: The hostname of the Wallaroo Ops edge service. See Edge Deployment Registry Guide for full details on enabling pipeline publishing and edge observability to Wallaroo.
  • PIPELINE_URL: The OCI registry URL to the containerized pipeline.
  • WORKSPACE_ID: The numerical ID of the workspace.

For example:

{'edgeBundle': 'ZXhwb3J0IEJVTkRMRV9WRVJTSU9OPTEKZXhwb3J0IEVER0VfTkFNRT14Z2ItY2NmcmF1ZC1lZGdlLXRlc3QKZXhwb3J0IEpPSU5fVE9LRU49MzE0OGFkYTUtMjg1YS00ZmNhLWIzYjgtYjUwYTQ4ZDc1MTFiCmV4cG9ydCBPUFNDRU5URVJfSE9TVD1kb2MtdGVzdC5lZGdlLndhbGxhcm9vY29tbXVuaXR5Lm5pbmphCmV4cG9ydCBQSVBFTElORV9VUkw9Z2hjci5pby93YWxsYXJvb2xhYnMvZG9jLXNhbXBsZXMvcGlwZWxpbmVzL2VkZ2UtcGlwZWxpbmU6ZjM4OGMxMDktOGQ1Ny00ZWQyLTk4MDYtYWExM2Y4NTQ1NzZiCmV4cG9ydCBXT1JLU1BBQ0VfSUQ9NQ=='}
base64 -D
ZXhwb3J0IEJVTkRMRV9WRVJTSU9OPTEKZXhwb3J0IEVER0VfTkFNRT14Z2ItY2NmcmF1ZC1lZGdlLXRlc3QKZXhwb3J0IEpPSU5fVE9LRU49MzE0OGFkYTUtMjg1YS00ZmNhLWIzYjgtYjUwYTQ4ZDc1MTFiCmV4cG9ydCBPUFNDRU5URVJfSE9TVD1kb2MtdGVzdC5lZGdlLndhbGxhcm9vY29tbXVuaXR5Lm5pbmphCmV4cG9ydCBQSVBFTElORV9VUkw9Z2hjci5pby93YWxsYXJvb2xhYnMvZG9jLXNhbXBsZXMvcGlwZWxpbmVzL2VkZ2UtcGlwZWxpbmU6ZjM4OGMxMDktOGQ1Ny00ZWQyLTk4MDYtYWExM2Y4NTQ1NzZiCmV4cG9ydCBXT1JLU1BBQ0VfSUQ9NQ==^D
export BUNDLE_VERSION=1
export EDGE_NAME=xgb-ccfraud-edge-test
export JOIN_TOKEN=3148ada5-285a-4fca-b3b8-b50a48d7511b
export OPSCENTER_HOST=doc-test.edge.wallaroocommunity.ninja
export PIPELINE_URL=ghcr.io/wallaroolabs/doc-samples/pipelines/edge-pipeline:f388c109-8d57-4ed2-9806-aa13f854576b
export WORKSPACE_ID=5

The JOIN_TOKEN is a one time access token. Once used, a JOIN_TOKEN expires. The authentication session data is stored in persistent volumes. Persistent volumes must be specified for docker and docker compose based deployments of Wallaroo pipelines; helm based deployments automatically provide persistent volumes to store authentication credentials.

The JOIN_TOKEN has the following time to live (TTL) parameters.

  • Once created, the JOIN_TOKEN is valid for 24 hours. After it expires the edge will not be allowed to contact the OpsCenter the first time and a new edge bundle will have to be created.
  • After an Edge joins to Wallaroo Ops for the first time with persistent storage, the edge must contact the Wallaroo Ops instance at least once every 7 days.
    • If this period is exceeded, the authentication credentials will expire and a new edge bundle must be created with a new and valid JOIN_TOKEN.

Wallaroo edges require unique names. To create a new edge bundle with the same name:

  • Use the Remove Edge to remove the edge by name.
  • Use Add Edge to add the edge with the same name. A new EDGE_BUNDLE is generated with a new JOIN_TOKEN.

8 - Wallaroo MLOps API Essentials Guide: Pipeline Log Management

How to use the Wallaroo API for Pipeline Log Management

Pipeline logs are retrieved through the Wallaroo MLOps API with the following request.

Get Pipeline Logs

Pipeline logs are retrieved through the Wallaroo MLOps API with the following request.

  • REQUEST URL
    • v1/api/pipelines/get_logs
  • Headers
    • Accept:
      • application/json; format=pandas-records: For the logs returned as pandas DataFrame
      • application/vnd.apache.arrow.file: for the logs returned as Apache Arrow
  • PARAMETERS
    • pipeline_name (String Required): The name of the pipeline.
    • workspace_id (Integer Required): The numerical identifier of the workspace.
    • cursor (String Optional): Cursor returned with a previous page of results from a pipeline log request, used to retrieve the next page of information.
    • order (String Optional Default: Desc): The order for log inserts returned. Valid values are:
      • Asc: In chronological order of inserts.
      • Desc: In reverse chronological order of inserts.
    • page_size (Integer Optional Default: 1000.): Max records per page.
    • start_time (String Optional): The start time of the period to retrieve logs for in RFC 3339 format for DateTime. Must be combined with end_time.
    • end_time (String Optional): The end time of the period to retrieve logs for in RFC 3339 format for DateTime. Must be combined with start_time.
  • RETURNS
    • The logs are returned by default as 'application/json; format=pandas-records' format. To request the logs as Apache Arrow tables, set the submission header Accept to application/vnd.apache.arrow.file.
    • Headers:
      • x-iteration-cursor: Used to retrieve the next page of results. This is not included if x-iteration-status is All.
      • x-iteration-status: Informs whether there are more records available outside of this log request parameters.
        • All: This page includes all logs available from this request. If x-iteration-status is All, then x-iteration-cursor is not provided.
        • SchemaChange: A change in the log schema caused by actions such as pipeline version, etc.
        • RecordLimited: The number of records exceeded from the page size, more records can be requested as the next page. There may be more records available to retrieve OR the record limit was reached for this request even if no more records are available in next cursor request.
        • ByteLimited: The number of records exceeded the pipeline log limit which is around 100K.

Get Pipeline Logs Example

For our example, we will retrieve the pipeline logs. FIrst by specifying the date and time, then we will request the logs and continue to show them as long as the cursor has another log to display. Because of the size of the input and outputs, most logs may be constrained by the x-iteration-status as ByteLimited.

# retrieve the authorization token
headers = wl.auth.auth_header()

url = f"{wl.api_endpoint}/v1/api/pipelines/get_logs"

# Standard log retrieval

data = {
    'pipeline_name': main_pipeline_name,
    'workspace_id': workspace_id,
    'start_time': dataframe_start.isoformat(),
    'end_time': dataframe_end.isoformat()
}

response = requests.post(url, headers=headers, json=data)
standard_logs = pd.DataFrame.from_records(response.json())

display(len(standard_logs))
display(standard_logs.loc[:, ["time", "out"]])
1
timeout
01703267760538{'output0': [17.09787, 16.459343, 17.259743, 19.960602, 43.600235, 59.986958, 62.826073, 68.24793, 77.43261, 80.82158, 89.44183, 96.168915, 99.22421, 112.584045, 126.75803, 131.9707, 137.1645, 141.93822, 146.29594, 152.00876, 155.94037, 165.20976, 175.27249, 184.05307, 193.66891, 201.51189, 215.04979, 223.80424, 227.24472, 234.19638, 244.9743, 248.5781, 252.42526, 264.95795, 278.48563, 285.758, 293.1897, 300.48227, 305.47742, 314.46085, 319.89404, 324.83658, 335.99536, 345.1116, 350.31964, 352.41107, 365.44934, 381.30008, 391.52316, 399.29163, 405.78503, 411.33804, 415.93207, 421.6868, 431.67108, 439.9069, 447.71542, 459.38522, 474.13187, 479.32642, 484.49884, 493.5153, 501.29932, 507.7967, 514.26044, 523.1473, 531.3479, 542.5094, 555.619, 557.7229, 564.6408, 571.5525, 572.8373, 587.95703, 604.2997, 609.452, 616.31714, 623.5797, 624.13153, 634.47266, 16.970057, 16.788723, 17.441803, 17.900642, 36.188023, 57.277973, 61.664352, 62.556896, 63.43486, 79.50621, 83.844, 95.983765, 106.166, 115.368454, 123.09253, 124.5821, 128.65866, 139.16113, 142.02315, 143.69855, ...]}
# retrieve the authorization token
headers = wl.auth.auth_header()

url = f"{wl.api_endpoint}/v1/api/pipelines/get_logs"

# datetime set back one day to get more values

data = {
    'pipeline_name': main_pipeline_name,
    'workspace_id': workspace_id
}

response = requests.post(url, headers=headers, json=data)
standard_logs = pd.DataFrame.from_records(response.json())

display(standard_logs.loc[:, ["time", "out"]])
cursor = response.headers['x-iteration-cursor']

# if there's another record, get the next one

while 'x-iteration-cursor' in response.headers:
    # retrieve the authorization token
    headers = wl.auth.auth_header()

    url = f"{wl.api_endpoint}/v1/api/pipelines/get_logs"

    # datetime set back one day to get more values

    data = {
        'pipeline_name': main_pipeline_name,
        'workspace_id': workspace_id,
        'cursor': response.headers['x-iteration-cursor']
    }

    response = requests.post(url, headers=headers, json=data)
    # if there's no response, the logs are done
    if response.json() != []:
        standard_logs = pd.DataFrame.from_records(response.json())
        display(standard_logs.head(5).loc[:, ["time", "out"]])
timeout
01703267304746{'output0': [17.09787, 16.459343, 17.259743, 19.960602, 43.600235, 59.986958, 62.826073, 68.24793, 77.43261, 80.82158, 89.44183, 96.168915, 99.22421, 112.584045, 126.75803, 131.9707, 137.1645, 141.93822, 146.29594, 152.00876, 155.94037, 165.20976, 175.27249, 184.05307, 193.66891, 201.51189, 215.04979, 223.80424, 227.24472, 234.19638, 244.9743, 248.5781, 252.42526, 264.95795, 278.48563, 285.758, 293.1897, 300.48227, 305.47742, 314.46085, 319.89404, 324.83658, 335.99536, 345.1116, 350.31964, 352.41107, 365.44934, 381.30008, 391.52316, 399.29163, 405.78503, 411.33804, 415.93207, 421.6868, 431.67108, 439.9069, 447.71542, 459.38522, 474.13187, 479.32642, 484.49884, 493.5153, 501.29932, 507.7967, 514.26044, 523.1473, 531.3479, 542.5094, 555.619, 557.7229, 564.6408, 571.5525, 572.8373, 587.95703, 604.2997, 609.452, 616.31714, 623.5797, 624.13153, 634.47266, 16.970057, 16.788723, 17.441803, 17.900642, 36.188023, 57.277973, 61.664352, 62.556896, 63.43486, 79.50621, 83.844, 95.983765, 106.166, 115.368454, 123.09253, 124.5821, 128.65866, 139.16113, 142.02315, 143.69855, ...]}
11703267414439{'output0': [17.09787, 16.459343, 17.259743, 19.960602, 43.600235, 59.986958, 62.826073, 68.24793, 77.43261, 80.82158, 89.44183, 96.168915, 99.22421, 112.584045, 126.75803, 131.9707, 137.1645, 141.93822, 146.29594, 152.00876, 155.94037, 165.20976, 175.27249, 184.05307, 193.66891, 201.51189, 215.04979, 223.80424, 227.24472, 234.19638, 244.9743, 248.5781, 252.42526, 264.95795, 278.48563, 285.758, 293.1897, 300.48227, 305.47742, 314.46085, 319.89404, 324.83658, 335.99536, 345.1116, 350.31964, 352.41107, 365.44934, 381.30008, 391.52316, 399.29163, 405.78503, 411.33804, 415.93207, 421.6868, 431.67108, 439.9069, 447.71542, 459.38522, 474.13187, 479.32642, 484.49884, 493.5153, 501.29932, 507.7967, 514.26044, 523.1473, 531.3479, 542.5094, 555.619, 557.7229, 564.6408, 571.5525, 572.8373, 587.95703, 604.2997, 609.452, 616.31714, 623.5797, 624.13153, 634.47266, 16.970057, 16.788723, 17.441803, 17.900642, 36.188023, 57.277973, 61.664352, 62.556896, 63.43486, 79.50621, 83.844, 95.983765, 106.166, 115.368454, 123.09253, 124.5821, 128.65866, 139.16113, 142.02315, 143.69855, ...]}
21703267420527{'output0': [17.09787, 16.459343, 17.259743, 19.960602, 43.600235, 59.986958, 62.826073, 68.24793, 77.43261, 80.82158, 89.44183, 96.168915, 99.22421, 112.584045, 126.75803, 131.9707, 137.1645, 141.93822, 146.29594, 152.00876, 155.94037, 165.20976, 175.27249, 184.05307, 193.66891, 201.51189, 215.04979, 223.80424, 227.24472, 234.19638, 244.9743, 248.5781, 252.42526, 264.95795, 278.48563, 285.758, 293.1897, 300.48227, 305.47742, 314.46085, 319.89404, 324.83658, 335.99536, 345.1116, 350.31964, 352.41107, 365.44934, 381.30008, 391.52316, 399.29163, 405.78503, 411.33804, 415.93207, 421.6868, 431.67108, 439.9069, 447.71542, 459.38522, 474.13187, 479.32642, 484.49884, 493.5153, 501.29932, 507.7967, 514.26044, 523.1473, 531.3479, 542.5094, 555.619, 557.7229, 564.6408, 571.5525, 572.8373, 587.95703, 604.2997, 609.452, 616.31714, 623.5797, 624.13153, 634.47266, 16.970057, 16.788723, 17.441803, 17.900642, 36.188023, 57.277973, 61.664352, 62.556896, 63.43486, 79.50621, 83.844, 95.983765, 106.166, 115.368454, 123.09253, 124.5821, 128.65866, 139.16113, 142.02315, 143.69855, ...]}
31703267426032{'output0': [17.09787, 16.459343, 17.259743, 19.960602, 43.600235, 59.986958, 62.826073, 68.24793, 77.43261, 80.82158, 89.44183, 96.168915, 99.22421, 112.584045, 126.75803, 131.9707, 137.1645, 141.93822, 146.29594, 152.00876, 155.94037, 165.20976, 175.27249, 184.05307, 193.66891, 201.51189, 215.04979, 223.80424, 227.24472, 234.19638, 244.9743, 248.5781, 252.42526, 264.95795, 278.48563, 285.758, 293.1897, 300.48227, 305.47742, 314.46085, 319.89404, 324.83658, 335.99536, 345.1116, 350.31964, 352.41107, 365.44934, 381.30008, 391.52316, 399.29163, 405.78503, 411.33804, 415.93207, 421.6868, 431.67108, 439.9069, 447.71542, 459.38522, 474.13187, 479.32642, 484.49884, 493.5153, 501.29932, 507.7967, 514.26044, 523.1473, 531.3479, 542.5094, 555.619, 557.7229, 564.6408, 571.5525, 572.8373, 587.95703, 604.2997, 609.452, 616.31714, 623.5797, 624.13153, 634.47266, 16.970057, 16.788723, 17.441803, 17.900642, 36.188023, 57.277973, 61.664352, 62.556896, 63.43486, 79.50621, 83.844, 95.983765, 106.166, 115.368454, 123.09253, 124.5821, 128.65866, 139.16113, 142.02315, 143.69855, ...]}
41703267432037{'output0': [17.09787, 16.459343, 17.259743, 19.960602, 43.600235, 59.986958, 62.826073, 68.24793, 77.43261, 80.82158, 89.44183, 96.168915, 99.22421, 112.584045, 126.75803, 131.9707, 137.1645, 141.93822, 146.29594, 152.00876, 155.94037, 165.20976, 175.27249, 184.05307, 193.66891, 201.51189, 215.04979, 223.80424, 227.24472, 234.19638, 244.9743, 248.5781, 252.42526, 264.95795, 278.48563, 285.758, 293.1897, 300.48227, 305.47742, 314.46085, 319.89404, 324.83658, 335.99536, 345.1116, 350.31964, 352.41107, 365.44934, 381.30008, 391.52316, 399.29163, 405.78503, 411.33804, 415.93207, 421.6868, 431.67108, 439.9069, 447.71542, 459.38522, 474.13187, 479.32642, 484.49884, 493.5153, 501.29932, 507.7967, 514.26044, 523.1473, 531.3479, 542.5094, 555.619, 557.7229, 564.6408, 571.5525, 572.8373, 587.95703, 604.2997, 609.452, 616.31714, 623.5797, 624.13153, 634.47266, 16.970057, 16.788723, 17.441803, 17.900642, 36.188023, 57.277973, 61.664352, 62.556896, 63.43486, 79.50621, 83.844, 95.983765, 106.166, 115.368454, 123.09253, 124.5821, 128.65866, 139.16113, 142.02315, 143.69855, ...]}
51703267437567{'output0': [17.09787, 16.459343, 17.259743, 19.960602, 43.600235, 59.986958, 62.826073, 68.24793, 77.43261, 80.82158, 89.44183, 96.168915, 99.22421, 112.584045, 126.75803, 131.9707, 137.1645, 141.93822, 146.29594, 152.00876, 155.94037, 165.20976, 175.27249, 184.05307, 193.66891, 201.51189, 215.04979, 223.80424, 227.24472, 234.19638, 244.9743, 248.5781, 252.42526, 264.95795, 278.48563, 285.758, 293.1897, 300.48227, 305.47742, 314.46085, 319.89404, 324.83658, 335.99536, 345.1116, 350.31964, 352.41107, 365.44934, 381.30008, 391.52316, 399.29163, 405.78503, 411.33804, 415.93207, 421.6868, 431.67108, 439.9069, 447.71542, 459.38522, 474.13187, 479.32642, 484.49884, 493.5153, 501.29932, 507.7967, 514.26044, 523.1473, 531.3479, 542.5094, 555.619, 557.7229, 564.6408, 571.5525, 572.8373, 587.95703, 604.2997, 609.452, 616.31714, 623.5797, 624.13153, 634.47266, 16.970057, 16.788723, 17.441803, 17.900642, 36.188023, 57.277973, 61.664352, 62.556896, 63.43486, 79.50621, 83.844, 95.983765, 106.166, 115.368454, 123.09253, 124.5821, 128.65866, 139.16113, 142.02315, 143.69855, ...]}
61703267442644{'output0': [17.09787, 16.459343, 17.259743, 19.960602, 43.600235, 59.986958, 62.826073, 68.24793, 77.43261, 80.82158, 89.44183, 96.168915, 99.22421, 112.584045, 126.75803, 131.9707, 137.1645, 141.93822, 146.29594, 152.00876, 155.94037, 165.20976, 175.27249, 184.05307, 193.66891, 201.51189, 215.04979, 223.80424, 227.24472, 234.19638, 244.9743, 248.5781, 252.42526, 264.95795, 278.48563, 285.758, 293.1897, 300.48227, 305.47742, 314.46085, 319.89404, 324.83658, 335.99536, 345.1116, 350.31964, 352.41107, 365.44934, 381.30008, 391.52316, 399.29163, 405.78503, 411.33804, 415.93207, 421.6868, 431.67108, 439.9069, 447.71542, 459.38522, 474.13187, 479.32642, 484.49884, 493.5153, 501.29932, 507.7967, 514.26044, 523.1473, 531.3479, 542.5094, 555.619, 557.7229, 564.6408, 571.5525, 572.8373, 587.95703, 604.2997, 609.452, 616.31714, 623.5797, 624.13153, 634.47266, 16.970057, 16.788723, 17.441803, 17.900642, 36.188023, 57.277973, 61.664352, 62.556896, 63.43486, 79.50621, 83.844, 95.983765, 106.166, 115.368454, 123.09253, 124.5821, 128.65866, 139.16113, 142.02315, 143.69855, ...]}
71703267449561{'output0': [17.09787, 16.459343, 17.259743, 19.960602, 43.600235, 59.986958, 62.826073, 68.24793, 77.43261, 80.82158, 89.44183, 96.168915, 99.22421, 112.584045, 126.75803, 131.9707, 137.1645, 141.93822, 146.29594, 152.00876, 155.94037, 165.20976, 175.27249, 184.05307, 193.66891, 201.51189, 215.04979, 223.80424, 227.24472, 234.19638, 244.9743, 248.5781, 252.42526, 264.95795, 278.48563, 285.758, 293.1897, 300.48227, 305.47742, 314.46085, 319.89404, 324.83658, 335.99536, 345.1116, 350.31964, 352.41107, 365.44934, 381.30008, 391.52316, 399.29163, 405.78503, 411.33804, 415.93207, 421.6868, 431.67108, 439.9069, 447.71542, 459.38522, 474.13187, 479.32642, 484.49884, 493.5153, 501.29932, 507.7967, 514.26044, 523.1473, 531.3479, 542.5094, 555.619, 557.7229, 564.6408, 571.5525, 572.8373, 587.95703, 604.2997, 609.452, 616.31714, 623.5797, 624.13153, 634.47266, 16.970057, 16.788723, 17.441803, 17.900642, 36.188023, 57.277973, 61.664352, 62.556896, 63.43486, 79.50621, 83.844, 95.983765, 106.166, 115.368454, 123.09253, 124.5821, 128.65866, 139.16113, 142.02315, 143.69855, ...]}
81703267455734{'output0': [17.09787, 16.459343, 17.259743, 19.960602, 43.600235, 59.986958, 62.826073, 68.24793, 77.43261, 80.82158, 89.44183, 96.168915, 99.22421, 112.584045, 126.75803, 131.9707, 137.1645, 141.93822, 146.29594, 152.00876, 155.94037, 165.20976, 175.27249, 184.05307, 193.66891, 201.51189, 215.04979, 223.80424, 227.24472, 234.19638, 244.9743, 248.5781, 252.42526, 264.95795, 278.48563, 285.758, 293.1897, 300.48227, 305.47742, 314.46085, 319.89404, 324.83658, 335.99536, 345.1116, 350.31964, 352.41107, 365.44934, 381.30008, 391.52316, 399.29163, 405.78503, 411.33804, 415.93207, 421.6868, 431.67108, 439.9069, 447.71542, 459.38522, 474.13187, 479.32642, 484.49884, 493.5153, 501.29932, 507.7967, 514.26044, 523.1473, 531.3479, 542.5094, 555.619, 557.7229, 564.6408, 571.5525, 572.8373, 587.95703, 604.2997, 609.452, 616.31714, 623.5797, 624.13153, 634.47266, 16.970057, 16.788723, 17.441803, 17.900642, 36.188023, 57.277973, 61.664352, 62.556896, 63.43486, 79.50621, 83.844, 95.983765, 106.166, 115.368454, 123.09253, 124.5821, 128.65866, 139.16113, 142.02315, 143.69855, ...]}
91703267462641{'output0': [17.09787, 16.459343, 17.259743, 19.960602, 43.600235, 59.986958, 62.826073, 68.24793, 77.43261, 80.82158, 89.44183, 96.168915, 99.22421, 112.584045, 126.75803, 131.9707, 137.1645, 141.93822, 146.29594, 152.00876, 155.94037, 165.20976, 175.27249, 184.05307, 193.66891, 201.51189, 215.04979, 223.80424, 227.24472, 234.19638, 244.9743, 248.5781, 252.42526, 264.95795, 278.48563, 285.758, 293.1897, 300.48227, 305.47742, 314.46085, 319.89404, 324.83658, 335.99536, 345.1116, 350.31964, 352.41107, 365.44934, 381.30008, 391.52316, 399.29163, 405.78503, 411.33804, 415.93207, 421.6868, 431.67108, 439.9069, 447.71542, 459.38522, 474.13187, 479.32642, 484.49884, 493.5153, 501.29932, 507.7967, 514.26044, 523.1473, 531.3479, 542.5094, 555.619, 557.7229, 564.6408, 571.5525, 572.8373, 587.95703, 604.2997, 609.452, 616.31714, 623.5797, 624.13153, 634.47266, 16.970057, 16.788723, 17.441803, 17.900642, 36.188023, 57.277973, 61.664352, 62.556896, 63.43486, 79.50621, 83.844, 95.983765, 106.166, 115.368454, 123.09253, 124.5821, 128.65866, 139.16113, 142.02315, 143.69855, ...]}
101703267468362{'output0': [17.09787, 16.459343, 17.259743, 19.960602, 43.600235, 59.986958, 62.826073, 68.24793, 77.43261, 80.82158, 89.44183, 96.168915, 99.22421, 112.584045, 126.75803, 131.9707, 137.1645, 141.93822, 146.29594, 152.00876, 155.94037, 165.20976, 175.27249, 184.05307, 193.66891, 201.51189, 215.04979, 223.80424, 227.24472, 234.19638, 244.9743, 248.5781, 252.42526, 264.95795, 278.48563, 285.758, 293.1897, 300.48227, 305.47742, 314.46085, 319.89404, 324.83658, 335.99536, 345.1116, 350.31964, 352.41107, 365.44934, 381.30008, 391.52316, 399.29163, 405.78503, 411.33804, 415.93207, 421.6868, 431.67108, 439.9069, 447.71542, 459.38522, 474.13187, 479.32642, 484.49884, 493.5153, 501.29932, 507.7967, 514.26044, 523.1473, 531.3479, 542.5094, 555.619, 557.7229, 564.6408, 571.5525, 572.8373, 587.95703, 604.2997, 609.452, 616.31714, 623.5797, 624.13153, 634.47266, 16.970057, 16.788723, 17.441803, 17.900642, 36.188023, 57.277973, 61.664352, 62.556896, 63.43486, 79.50621, 83.844, 95.983765, 106.166, 115.368454, 123.09253, 124.5821, 128.65866, 139.16113, 142.02315, 143.69855, ...]}
timeout
01703267760538{'output0': [17.09787, 16.459343, 17.259743, 19.960602, 43.600235, 59.986958, 62.826073, 68.24793, 77.43261, 80.82158, 89.44183, 96.168915, 99.22421, 112.584045, 126.75803, 131.9707, 137.1645, 141.93822, 146.29594, 152.00876, 155.94037, 165.20976, 175.27249, 184.05307, 193.66891, 201.51189, 215.04979, 223.80424, 227.24472, 234.19638, 244.9743, 248.5781, 252.42526, 264.95795, 278.48563, 285.758, 293.1897, 300.48227, 305.47742, 314.46085, 319.89404, 324.83658, 335.99536, 345.1116, 350.31964, 352.41107, 365.44934, 381.30008, 391.52316, 399.29163, 405.78503, 411.33804, 415.93207, 421.6868, 431.67108, 439.9069, 447.71542, 459.38522, 474.13187, 479.32642, 484.49884, 493.5153, 501.29932, 507.7967, 514.26044, 523.1473, 531.3479, 542.5094, 555.619, 557.7229, 564.6408, 571.5525, 572.8373, 587.95703, 604.2997, 609.452, 616.31714, 623.5797, 624.13153, 634.47266, 16.970057, 16.788723, 17.441803, 17.900642, 36.188023, 57.277973, 61.664352, 62.556896, 63.43486, 79.50621, 83.844, 95.983765, 106.166, 115.368454, 123.09253, 124.5821, 128.65866, 139.16113, 142.02315, 143.69855, ...]}
11703267788432{'output0': [17.09787, 16.459343, 17.259743, 19.960602, 43.600235, 59.986958, 62.826073, 68.24793, 77.43261, 80.82158, 89.44183, 96.168915, 99.22421, 112.584045, 126.75803, 131.9707, 137.1645, 141.93822, 146.29594, 152.00876, 155.94037, 165.20976, 175.27249, 184.05307, 193.66891, 201.51189, 215.04979, 223.80424, 227.24472, 234.19638, 244.9743, 248.5781, 252.42526, 264.95795, 278.48563, 285.758, 293.1897, 300.48227, 305.47742, 314.46085, 319.89404, 324.83658, 335.99536, 345.1116, 350.31964, 352.41107, 365.44934, 381.30008, 391.52316, 399.29163, 405.78503, 411.33804, 415.93207, 421.6868, 431.67108, 439.9069, 447.71542, 459.38522, 474.13187, 479.32642, 484.49884, 493.5153, 501.29932, 507.7967, 514.26044, 523.1473, 531.3479, 542.5094, 555.619, 557.7229, 564.6408, 571.5525, 572.8373, 587.95703, 604.2997, 609.452, 616.31714, 623.5797, 624.13153, 634.47266, 16.970057, 16.788723, 17.441803, 17.900642, 36.188023, 57.277973, 61.664352, 62.556896, 63.43486, 79.50621, 83.844, 95.983765, 106.166, 115.368454, 123.09253, 124.5821, 128.65866, 139.16113, 142.02315, 143.69855, ...]}
21703267794241{'output0': [17.09787, 16.459343, 17.259743, 19.960602, 43.600235, 59.986958, 62.826073, 68.24793, 77.43261, 80.82158, 89.44183, 96.168915, 99.22421, 112.584045, 126.75803, 131.9707, 137.1645, 141.93822, 146.29594, 152.00876, 155.94037, 165.20976, 175.27249, 184.05307, 193.66891, 201.51189, 215.04979, 223.80424, 227.24472, 234.19638, 244.9743, 248.5781, 252.42526, 264.95795, 278.48563, 285.758, 293.1897, 300.48227, 305.47742, 314.46085, 319.89404, 324.83658, 335.99536, 345.1116, 350.31964, 352.41107, 365.44934, 381.30008, 391.52316, 399.29163, 405.78503, 411.33804, 415.93207, 421.6868, 431.67108, 439.9069, 447.71542, 459.38522, 474.13187, 479.32642, 484.49884, 493.5153, 501.29932, 507.7967, 514.26044, 523.1473, 531.3479, 542.5094, 555.619, 557.7229, 564.6408, 571.5525, 572.8373, 587.95703, 604.2997, 609.452, 616.31714, 623.5797, 624.13153, 634.47266, 16.970057, 16.788723, 17.441803, 17.900642, 36.188023, 57.277973, 61.664352, 62.556896, 63.43486, 79.50621, 83.844, 95.983765, 106.166, 115.368454, 123.09253, 124.5821, 128.65866, 139.16113, 142.02315, 143.69855, ...]}
31703267802339{'output0': [17.09787, 16.459343, 17.259743, 19.960602, 43.600235, 59.986958, 62.826073, 68.24793, 77.43261, 80.82158, 89.44183, 96.168915, 99.22421, 112.584045, 126.75803, 131.9707, 137.1645, 141.93822, 146.29594, 152.00876, 155.94037, 165.20976, 175.27249, 184.05307, 193.66891, 201.51189, 215.04979, 223.80424, 227.24472, 234.19638, 244.9743, 248.5781, 252.42526, 264.95795, 278.48563, 285.758, 293.1897, 300.48227, 305.47742, 314.46085, 319.89404, 324.83658, 335.99536, 345.1116, 350.31964, 352.41107, 365.44934, 381.30008, 391.52316, 399.29163, 405.78503, 411.33804, 415.93207, 421.6868, 431.67108, 439.9069, 447.71542, 459.38522, 474.13187, 479.32642, 484.49884, 493.5153, 501.29932, 507.7967, 514.26044, 523.1473, 531.3479, 542.5094, 555.619, 557.7229, 564.6408, 571.5525, 572.8373, 587.95703, 604.2997, 609.452, 616.31714, 623.5797, 624.13153, 634.47266, 16.970057, 16.788723, 17.441803, 17.900642, 36.188023, 57.277973, 61.664352, 62.556896, 63.43486, 79.50621, 83.844, 95.983765, 106.166, 115.368454, 123.09253, 124.5821, 128.65866, 139.16113, 142.02315, 143.69855, ...]}
41703267810346{'output0': [17.09787, 16.459343, 17.259743, 19.960602, 43.600235, 59.986958, 62.826073, 68.24793, 77.43261, 80.82158, 89.44183, 96.168915, 99.22421, 112.584045, 126.75803, 131.9707, 137.1645, 141.93822, 146.29594, 152.00876, 155.94037, 165.20976, 175.27249, 184.05307, 193.66891, 201.51189, 215.04979, 223.80424, 227.24472, 234.19638, 244.9743, 248.5781, 252.42526, 264.95795, 278.48563, 285.758, 293.1897, 300.48227, 305.47742, 314.46085, 319.89404, 324.83658, 335.99536, 345.1116, 350.31964, 352.41107, 365.44934, 381.30008, 391.52316, 399.29163, 405.78503, 411.33804, 415.93207, 421.6868, 431.67108, 439.9069, 447.71542, 459.38522, 474.13187, 479.32642, 484.49884, 493.5153, 501.29932, 507.7967, 514.26044, 523.1473, 531.3479, 542.5094, 555.619, 557.7229, 564.6408, 571.5525, 572.8373, 587.95703, 604.2997, 609.452, 616.31714, 623.5797, 624.13153, 634.47266, 16.970057, 16.788723, 17.441803, 17.900642, 36.188023, 57.277973, 61.664352, 62.556896, 63.43486, 79.50621, 83.844, 95.983765, 106.166, 115.368454, 123.09253, 124.5821, 128.65866, 139.16113, 142.02315, 143.69855, ...]}
timeout
01703266176951{'output0': [17.09787, 16.459343, 17.259743, 19.960602, 43.600235, 59.986958, 62.826073, 68.24793, 77.43261, 80.82158, 89.44183, 96.168915, 99.22421, 112.584045, 126.75803, 131.9707, 137.1645, 141.93822, 146.29594, 152.00876, 155.94037, 165.20976, 175.27249, 184.05307, 193.66891, 201.51189, 215.04979, 223.80424, 227.24472, 234.19638, 244.9743, 248.5781, 252.42526, 264.95795, 278.48563, 285.758, 293.1897, 300.48227, 305.47742, 314.46085, 319.89404, 324.83658, 335.99536, 345.1116, 350.31964, 352.41107, 365.44934, 381.30008, 391.52316, 399.29163, 405.78503, 411.33804, 415.93207, 421.6868, 431.67108, 439.9069, 447.71542, 459.38522, 474.13187, 479.32642, 484.49884, 493.5153, 501.29932, 507.7967, 514.26044, 523.1473, 531.3479, 542.5094, 555.619, 557.7229, 564.6408, 571.5525, 572.8373, 587.95703, 604.2997, 609.452, 616.31714, 623.5797, 624.13153, 634.47266, 16.970057, 16.788723, 17.441803, 17.900642, 36.188023, 57.277973, 61.664352, 62.556896, 63.43486, 79.50621, 83.844, 95.983765, 106.166, 115.368454, 123.09253, 124.5821, 128.65866, 139.16113, 142.02315, 143.69855, ...]}
11703266192757{'output0': [17.09787, 16.459343, 17.259743, 19.960602, 43.600235, 59.986958, 62.826073, 68.24793, 77.43261, 80.82158, 89.44183, 96.168915, 99.22421, 112.584045, 126.75803, 131.9707, 137.1645, 141.93822, 146.29594, 152.00876, 155.94037, 165.20976, 175.27249, 184.05307, 193.66891, 201.51189, 215.04979, 223.80424, 227.24472, 234.19638, 244.9743, 248.5781, 252.42526, 264.95795, 278.48563, 285.758, 293.1897, 300.48227, 305.47742, 314.46085, 319.89404, 324.83658, 335.99536, 345.1116, 350.31964, 352.41107, 365.44934, 381.30008, 391.52316, 399.29163, 405.78503, 411.33804, 415.93207, 421.6868, 431.67108, 439.9069, 447.71542, 459.38522, 474.13187, 479.32642, 484.49884, 493.5153, 501.29932, 507.7967, 514.26044, 523.1473, 531.3479, 542.5094, 555.619, 557.7229, 564.6408, 571.5525, 572.8373, 587.95703, 604.2997, 609.452, 616.31714, 623.5797, 624.13153, 634.47266, 16.970057, 16.788723, 17.441803, 17.900642, 36.188023, 57.277973, 61.664352, 62.556896, 63.43486, 79.50621, 83.844, 95.983765, 106.166, 115.368454, 123.09253, 124.5821, 128.65866, 139.16113, 142.02315, 143.69855, ...]}
21703266198591{'output0': [17.09787, 16.459343, 17.259743, 19.960602, 43.600235, 59.986958, 62.826073, 68.24793, 77.43261, 80.82158, 89.44183, 96.168915, 99.22421, 112.584045, 126.75803, 131.9707, 137.1645, 141.93822, 146.29594, 152.00876, 155.94037, 165.20976, 175.27249, 184.05307, 193.66891, 201.51189, 215.04979, 223.80424, 227.24472, 234.19638, 244.9743, 248.5781, 252.42526, 264.95795, 278.48563, 285.758, 293.1897, 300.48227, 305.47742, 314.46085, 319.89404, 324.83658, 335.99536, 345.1116, 350.31964, 352.41107, 365.44934, 381.30008, 391.52316, 399.29163, 405.78503, 411.33804, 415.93207, 421.6868, 431.67108, 439.9069, 447.71542, 459.38522, 474.13187, 479.32642, 484.49884, 493.5153, 501.29932, 507.7967, 514.26044, 523.1473, 531.3479, 542.5094, 555.619, 557.7229, 564.6408, 571.5525, 572.8373, 587.95703, 604.2997, 609.452, 616.31714, 623.5797, 624.13153, 634.47266, 16.970057, 16.788723, 17.441803, 17.900642, 36.188023, 57.277973, 61.664352, 62.556896, 63.43486, 79.50621, 83.844, 95.983765, 106.166, 115.368454, 123.09253, 124.5821, 128.65866, 139.16113, 142.02315, 143.69855, ...]}
31703266205430{'output0': [17.09787, 16.459343, 17.259743, 19.960602, 43.600235, 59.986958, 62.826073, 68.24793, 77.43261, 80.82158, 89.44183, 96.168915, 99.22421, 112.584045, 126.75803, 131.9707, 137.1645, 141.93822, 146.29594, 152.00876, 155.94037, 165.20976, 175.27249, 184.05307, 193.66891, 201.51189, 215.04979, 223.80424, 227.24472, 234.19638, 244.9743, 248.5781, 252.42526, 264.95795, 278.48563, 285.758, 293.1897, 300.48227, 305.47742, 314.46085, 319.89404, 324.83658, 335.99536, 345.1116, 350.31964, 352.41107, 365.44934, 381.30008, 391.52316, 399.29163, 405.78503, 411.33804, 415.93207, 421.6868, 431.67108, 439.9069, 447.71542, 459.38522, 474.13187, 479.32642, 484.49884, 493.5153, 501.29932, 507.7967, 514.26044, 523.1473, 531.3479, 542.5094, 555.619, 557.7229, 564.6408, 571.5525, 572.8373, 587.95703, 604.2997, 609.452, 616.31714, 623.5797, 624.13153, 634.47266, 16.970057, 16.788723, 17.441803, 17.900642, 36.188023, 57.277973, 61.664352, 62.556896, 63.43486, 79.50621, 83.844, 95.983765, 106.166, 115.368454, 123.09253, 124.5821, 128.65866, 139.16113, 142.02315, 143.69855, ...]}
41703266212538{'output0': [17.09787, 16.459343, 17.259743, 19.960602, 43.600235, 59.986958, 62.826073, 68.24793, 77.43261, 80.82158, 89.44183, 96.168915, 99.22421, 112.584045, 126.75803, 131.9707, 137.1645, 141.93822, 146.29594, 152.00876, 155.94037, 165.20976, 175.27249, 184.05307, 193.66891, 201.51189, 215.04979, 223.80424, 227.24472, 234.19638, 244.9743, 248.5781, 252.42526, 264.95795, 278.48563, 285.758, 293.1897, 300.48227, 305.47742, 314.46085, 319.89404, 324.83658, 335.99536, 345.1116, 350.31964, 352.41107, 365.44934, 381.30008, 391.52316, 399.29163, 405.78503, 411.33804, 415.93207, 421.6868, 431.67108, 439.9069, 447.71542, 459.38522, 474.13187, 479.32642, 484.49884, 493.5153, 501.29932, 507.7967, 514.26044, 523.1473, 531.3479, 542.5094, 555.619, 557.7229, 564.6408, 571.5525, 572.8373, 587.95703, 604.2997, 609.452, 616.31714, 623.5797, 624.13153, 634.47266, 16.970057, 16.788723, 17.441803, 17.900642, 36.188023, 57.277973, 61.664352, 62.556896, 63.43486, 79.50621, 83.844, 95.983765, 106.166, 115.368454, 123.09253, 124.5821, 128.65866, 139.16113, 142.02315, 143.69855, ...]}

Pipeline Log Storage

Pipeline logs have a set allocation of storage space and data requirements.

Pipeline Log Storage Warnings

To prevent storage and performance issues, inference result data may be dropped from pipeline logs by the following standards:

  • Columns are progressively removed from the row starting with the largest input data size and working to the smallest, then the same for outputs.

For example, Computer Vision ML Models typically have large inputs and output values - a single pandas DataFrame inference request may be over 13 MB in size, and the inference results nearly as large. To prevent pipeline log storage issues, the input may be dropped from the pipeline logs, and if additional space is needed, the inference outputs would follow. The time column is preserved.

If a pipeline has dropped columns for space purposes, this will be displayed when a log request is made with the following warning, with {columns} replaced with the dropped columns.

The inference log is above the allowable limit and the following columns may have been suppressed for various rows in the logs: {columns}. To review the dropped columns for an individual inferences suppressed data, include dataset=["metadata"] in the log request.

Review Dropped Columns

To review what columns are dropped from pipeline logs for storage reasons, include the dataset metadata in the request to view the column metadata.dropped. This metadata field displays a List of any columns dropped from the pipeline logs.

For example:

# retrieve the authorization token
headers = wl.auth.auth_header()

url = f"{APIURL}/v1/api/pipelines/get_logs"

# Standard log retrieval

data = {
    'pipeline_name': main_pipeline_name,
    'workspace_id': workspace_id
}

response = requests.post(url, headers=headers, json=data)
standard_logs = pd.DataFrame.from_records(response.json())

display(len(standard_logs))
display(standard_logs.head(5).loc[:, ["time", "metadata"]])
cursor = response.headers['x-iteration-cursor']
 timemetadata
01688760035752{’last_model’: ‘{“model_name”:“logapicontrol”,“model_sha”:“e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6”}’, ‘pipeline_version’: ‘’, ’elapsed’: [112967, 267146], ‘dropped’: []}
11688760036054{’last_model’: ‘{“model_name”:“logapicontrol”,“model_sha”:“e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6”}’, ‘pipeline_version’: ‘’, ’elapsed’: [37127, 594183], ‘dropped’: []}
21688759857040{’last_model’: ‘{“model_name”:“logapicontrol”,“model_sha”:“e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6”}’, ‘pipeline_version’: ‘’, ’elapsed’: [111082, 253184], ‘dropped’: []}
31688759857526{’last_model’: ‘{“model_name”:“logapicontrol”,“model_sha”:“e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6”}’, ‘pipeline_version’: ‘’, ’elapsed’: [43962, 265740], ‘dropped’: []}

Suppressed Data Elements

Data elements that do not fit the supported data types below, such as None or Null values, are not supported in pipeline logs. When present, undefined data will be written in the place of the null value, typically zeroes. Any null list values will present an empty list.

9 - Wallaroo MLOps API Essentials Guide: Enablement Management

How to use the Wallaroo API for Enablement Management

Enablement Management

Enablement Management allows users to see what Wallaroo features have been activated.

List Enablement Features

Lists the enablement features for the Wallaroo instance.

  • PARAMETERS
    • null: An empty set {}
  • RETURNS
    • features - (string): Enabled features.
    • name - (string): Name of the Wallaroo instance.
    • is_auth_enabled - (bool): Whether authentication is enabled.
# List enablement features
# Retrieve the token 
headers = wl.auth.auth_header()
api_request = f"{APIURL}/v1/api/features/list"

data = {
}

response = requests.post(api_request, json=data, headers=headers, verify=True).json()
response
{'features': {'plateau': 'true'},
 'name': 'Wallaroo Dev',
 'is_auth_enabled': True}

10 - Wallaroo MLOps API Essentials Guide: Assays Management

How to use the Wallaroo API for Assays Management

Enablement Management

Enablement Management allows users to see what Wallaroo features have been activated.

List Enablement Features

Lists the enablement features for the Wallaroo instance.

List Enablement Features Parameters

An empty set {}

List Enablement Features Returns

FieldTypeDescription
featuresStringEnabled features.
nameStringName of the Wallaroo instance.
is_auth_enabledBooleanWhether authentication is enabled.
# List enablement features
# Retrieve the token 
headers = wl.auth.auth_header()

endpoint = f"{wl.api_endpoint}/v1/api/features/list"

data = {
}

response = requests.post(endpoint, json=data, headers=headers, verify=True).json()
response
{'features': {'plateau': 'true'},
 'name': 'Wallaroo Dev',
 'is_auth_enabled': True}
!curl {wl.api_endpoint}/v1/api/features/list \
    -H "Authorization: {wl.auth.auth_header()['Authorization']}" \
    -H "Content-Type: application/json" \
    --data '{{}}'
{"features":{"plateau":"true"},"name":"Wallaroo Dev","is_auth_enabled":true}

Assays

Create Baseline Summary Statistics

  • Endpoint: /v1/api/assays/summarize

Generate static baseline summary stats for a vector or windowed data. For windows data, this requires that inferences have been performed in the target pipeline. For vector data, a numpy array is used.

Create Baseline Summary Statistics Parameters

Field  TypeDescription
baseline  Baseline (Required)The baseline for the assay consisting of the following options. Options are vector and fixed_window.
 fixed_window (AssayFixConfiguration) (Required)The fixed configuration for the assay window baseline with the set baseline start and end dates.
  intervalString (Required)The interval of data to gather for the baseline in the format {Integer} {Unit}, with Unit being minutes, hours, and days.
  model_nameString (Required)The name of the model to track inference data for the baseline from.
  pathString (Required)The input/output path in the format `{input
  pipeline_nameString (Required)The name of the pipeline with the baseline data.
  startString (Optional)The DateTime of the baseline start date.
  widthString (Required)The amount of data from inference results to gather in the format {Integer} {Unit}, with Unit being minutes, hours, and days.
  workspace_idInteger (Required)The numerical identifier of the workspace.
  workspace_idList[String] (Optional)A list of locations, including edge locations. An empty list [] defaults to the Wallaroo Ops pipeline location.
 vector List[Float]A numpy array of values for the baseline.
summarizer  (AssaySummerizer) (REQUIRED)The summarizer type for the array aka “advanced settings” in the Wallaroo Dashboard UI.
 type String (Required)Type of summarizer.
  bin_modeString (Required)The binning model type. Values are: Quantile and Equal
  aggregationString (Required)Aggregation type. Values are: Cumulative, Density, and Edges.
  metricString (Required)Metric type. Values are: PSI, MaxDiff (Maximum Difference of Bins), and SumDiff (Sum of the Difference of Bins)
  num_bins*Integer (Required)The number of bins. Recommended values are between 5 and 14.
  typeString (Required)Bin type. Values are: UnivariateContinuous
  bin_weights*List[float] (Optional)The weights assigned to the assay bins.
  bin_widthList[float] (Optional)The width assigned to the assay bins.
  provided_edgesList[float] (Optional)The edges used for the assay bins.

Create Baseline Summary Statistics Returns

FieldTypeDescription
countIntegerThe number of inference results collected.
minFloatThe minimum values from the baseline.
maxFloatThe maximum values from the baseline.
meanFloatThe mean values from the baseline.
medianFloatThe median values from the baseline.
stdFloatThe standard deviation score.
edgesList[Float]The defined bin edges.
edgesList[String]The bin names.
aggregated_valuesList[Float]The aggregated values of the baseline.
aggregationStringAggregation type. Values are: Cumulative, Density, and Edges.
startStringThe datetime stamp of when the baseline data started.
endStringThe datetime stamp of when the baseline data started.

Create Baseline Summary Statistics Examples

The following example code will:

  • Create a workspace for our assay values.
  • Upload a model and create a pipeline with the model as a pipeline step.
  • Perform sample inference and use the inference results create the baseline summary data.
# create a new workspace

# Retrieve the token 
headers = wl.auth.auth_header()

endpoint = f"{wl.api_endpoint}/v1/api/workspaces/create"

data = {
  "workspace_name": "test-api-workspace-assays"
}

response = requests.post(endpoint, json=data, headers=headers, verify=True).json()
display(response)

# Stored for future examples
workspace_id = response['workspace_id']
{'workspace_id': 7}
framework='onnx'

example_model_name = f"api-sample-model"

pipeline_name = "api-pipeline-with-models"

# Create pipeline in a workspace with models

# First upload a model
# Retrieve the token 
headers = wl.auth.auth_header()

endpoint = f"{wl.api_endpoint}/v1/api/models/upload_and_convert"

metadata = {
    "name": example_model_name,
    "visibility": "public",
    "workspace_id": workspace_id,
    "conversion": {
        "framework": framework,
        "python_version": "3.8",
        "requirements": []
    }
}

files = {
    "metadata": (None, json.dumps(metadata), "application/json"),
    'file': (example_model_name, open('./models/ccfraud.onnx', 'rb'), "application/octet-stream")
    }

response = requests.post(endpoint, files=files, headers=headers).json()

example_model_id = response['insert_models']['returning'][0]['models'][0]['id']

# Second, get the model version

# Retrieve the token 
headers = wl.auth.auth_header()
endpoint = f"{wl.api_endpoint}/v1/api/models/list_versions"

data = {
  "model_id": example_model_name,
  "models_pk_id": example_model_id
}

response = requests.post(endpoint, json=data, headers=headers, verify=True).json()
example_model_sha = response[-1]['sha']
example_model_version = response[-1]['model_version']

# Now create the pipeline with the new model
# Retrieve the token 
headers = wl.auth.auth_header()

endpoint = f"{wl.api_endpoint}/v1/api/pipelines/create"

data = {
  "pipeline_id": pipeline_name,
  "workspace_id": workspace_id,
  "definition": {
      'steps': [
          {
            'ModelInference': 
              {
                  'models': 
                    [
                        {
                            'name': example_model_name, 
                            'version': example_model_version, 
                            'sha': example_model_sha
                        }
                    ]
              }
          }
        ]
      }
    }

response = requests.post(endpoint, 
                         json=data, 
                         headers=headers, 
                         verify=True).json()
display(json.dumps(response))

# saved for later steps

model_pipeline_id = response['pipeline_pk_id']
model_pipeline_variant_id=response['pipeline_variant_pk_id']
model_pipeline_variant_version=['pipeline_variant_version']
'{"pipeline_pk_id": 6, "pipeline_variant_pk_id": 6, "pipeline_variant_version": "2f4bf3ac-48bb-4798-b59a-81de844248d1"}'
# deploy that pipeline

# Deploy a pipeline with models

# Retrieve the token 
headers = wl.auth.auth_header()
endpoint = f"{wl.api_endpoint}/v1/api/pipelines/deploy"

# verify this matches the pipeline with model created earlier
pipeline_with_models_id = "api-pipeline-with-models"

data = {
    "deploy_id": pipeline_with_models_id,
    "pipeline_version_pk_id": model_pipeline_variant_id,
    "models": [
        {
            "name": example_model_name,
            "version":example_model_version,
            "sha":example_model_sha
        }
    ],
    "pipeline_id": model_pipeline_id
}

response = requests.post(endpoint, json=data, headers=headers, verify=True).json()
display(response)
model_deployment_id=response['id']
{'id': 3}
# get the inference url

## Retrieve the pipeline's External Inference URL

# Retrieve the token 
headers = wl.auth.auth_header()

endpoint = f"{wl.api_endpoint}/v1/api/admin/get_pipeline_external_url"

data = {
    "workspace_id": workspace_id,
    "pipeline_name": pipeline_with_models_id
}

response = requests.post(endpoint, json=data, headers=headers, verify=True).json()
print(response)
deployurl = response['url']
{'url': 'https://doc-test.api.wallarooexample.ai/v1/api/pipelines/infer/api-pipeline-with-models-3/api-pipeline-with-models'}
# check the deployment status - must be 'Running'

# Retrieve the token 
headers = wl.auth.auth_header()

# Get model pipeline deployment

status = ""

while status != 'Running':
  time.sleep(15)
  endpoint = f"{wl.api_endpoint}/v1/api/status/get_deployment"

  data = {
    "name": f"{pipeline_with_models_id}-{model_deployment_id}"
  }

  response = requests.post(endpoint, json=data, headers=headers, verify=True).json()
  status = response['status']
  print(status)
Running
# sample inference 

assay_start = datetime.datetime.now(datetime.timezone.utc)

# Retrieve the token
headers = wl.auth.auth_header()

# set Content-Type type
headers['Content-Type']='application/json; format=pandas-records'

## Inference through external URL using dataframe

# retrieve the json data to submit
data = pd.read_json('./data/cc_data_api_1k.df.json')

response = (requests.post(
        deployurl, 
        data=data.to_json(orient="records"), 
        headers=headers)
        .json())

df = pd.DataFrame.from_records(response)

display(df.head(5))

# sleep for 10 seconds to accommodate any time difference

time.sleep(10)
assay_end = datetime.datetime.now(datetime.timezone.utc)
timeinoutcheck_failuresmetadata
01704827280654{'dense_input': [-1.0603297501, 2.3544967095, ...{'dense_1': [0.99300325]}[]{'last_model': '{"model_name":"api-sample-mode...
11704827280654{'dense_input': [-1.0603297501, 2.3544967095, ...{'dense_1': [0.99300325]}[]{'last_model': '{"model_name":"api-sample-mode...
21704827280654{'dense_input': [-1.0603297501, 2.3544967095, ...{'dense_1': [0.99300325]}[]{'last_model': '{"model_name":"api-sample-mode...
31704827280654{'dense_input': [-1.0603297501, 2.3544967095, ...{'dense_1': [0.99300325]}[]{'last_model': '{"model_name":"api-sample-mode...
41704827280654{'dense_input': [0.5817662108, 0.097881551, 0....{'dense_1': [0.0010916889]}[]{'last_model': '{"model_name":"api-sample-mode...

Create the summary via Requests.

## create the summary via requests

# Retrieve the token 
headers = wl.auth.auth_header()

endpoint = f"{wl.api_endpoint}/v1/api/assays/summarize"

data = {
    "summarizer": {
        "type": "UnivariateContinuous",
        "bin_mode": "Quantile",
        "aggregation": "Density",
        "metric": "PSI",
        "num_bins": 5,
        "bin_weights": None,
        "bin_width": None,
        "provided_edges": None
    },
    "baseline": {
        "fixed_window": {
            "interval": "24 hours",
            "model_name": example_model_name,
            "path": "output dense_1 0",
            "pipeline_name": pipeline_name,
            "start": assay_start.isoformat(),
            "width": "24 hours",
            "workspace_id": workspace_id,
            "locations": []
        }
    }
}

# print(data)

response = requests.post(endpoint, json=data, headers=headers, verify=True)
baseline_summary = response.json()
display(baseline_summary)
{'count': 1001,
 'min': 1.519918441772461e-06,
 'max': 1.0,
 'mean': 0.0065982516233499475,
 'median': 0.0005419552326202393,
 'std': 0.07661715908141875,
 'edges': [1.519918441772461e-06,
  0.00026360154151916504,
  0.0004509389400482178,
  0.0006800591945648193,
  0.000969529151916504,
  1.0,
  None],
 'edge_names': ['left_outlier',
  'q_20',
  'q_40',
  'q_60',
  'q_80',
  'q_100',
  'right_outlier'],
 'aggregated_values': [0.0,
  0.1998001998001998,
  0.1998001998001998,
  0.2007992007992008,
  0.1998001998001998,
  0.1998001998001998,
  0.0],
 'aggregation': 'Density',
 'start': '2024-01-09T19:07:59.566516Z',
 'end': '2024-01-10T19:07:59.566516Z'}

Create Assay

  • Endpoint: /v1/api/assays/create

Create a new array in a specified pipeline. Assays required at least one inference request is completed.

Create Assay Parameters

Field  TypeDescription
id  Integer (Optional)The numerical identifier for the assay.
name  String (Required)The name of the assay.
pipeline_id  String (Required)The numerical identifier the assay will be placed into.
pipeline_name  String (Required)The name of the pipeline
active  Boolean (Required)Indicates whether the assay will be active upon creation or not.
status  String (Required)The status of the assay upon creation. Options are: active: the Assay is created and active. created: The assay is created but inactive.
iopath  String (Required)The iopath of the assay in the format `“input
baseline  Baseline (Required)The baseline for the assay consisting of the following options. Options are fixed, calculated and static.
 fixed (AssayFixConfiguration) (Required)The fixed configuration for the assay with the set baseline start and end dates.
  pipelineString (Required)The name of the pipeline with the baseline data.
  modelString (Required)The name of the model used.
  start_atString (Required)The DateTime of the baseline start date.
  end_atString (Required)The DateTime of the baseline end date.
 calculated (AssaysCreateJsonBodyBaselineType0CalculatedType0) (Required)The calculated baseline submitted with numpy arrays. Declared with calculated. This includes the following fields.
  vector(List[float]) (Required)The numpy array of baseline values.
 static AssaysCreateJsonBodyBaselineType1Static (Required)The static baseline. These values are generated from the fixed and calculated baseline methods listed above, and can be used to achieve more granularity in the assay baseline creation.
  countInteger (Required)The number values used.
  minFloat (Required)The minimum value of baseline values.
  maxFloat (Required)The maximum value of baseline values.
  meanFloat (Required)The mean of the of baseline values.
  stdFloat (Required)The standard deviation of baseline values.
  edgesList[Float] (Required)The edges defined from the baseline values or manually defined.
  aggregated_valuesList[Float] (Required)The aggregated values of each edge.
  aggregationString (Required)The type of baseline aggregation. Values are: Cumulative, Density, and Edges.
  startString (Optional)The DateTime of the baseline inference start date.
  endString (Optional) The DateTime of the baseline inference end date.
window  AssayWindow (Required)Assay window.
 pipeline String (Required)The name of the pipeline for the assay window.
 model String (Required)The name of the model used for the assay window.
 width String (Required)The width of the assay window.
 start String (Required)The DateTime of when to start the assay window.
 interval String (Required)The assay window interval.
 locations List[String] (Required)The locations included in the assay. If set to an empty set [] the Wallaroo Ops pipeline will be the only location used.
summarizer  (AssaySummerizer) (REQUIRED)The summarizer type for the array aka “advanced settings” in the Wallaroo Dashboard UI.
 type String (Required)Type of summarizer.
  bin_modeString (Required)The binning model type. Values are: Quantile and Equal
  aggregationString (Required)Aggregation type. Values are: Cumulative, Density, and Edges.
  metricString (Required)Metric type. Values are: PSI, MaxDiff (Maximum Difference of Bins), and SumDiff (Sum of the Difference of Bins)
  num_bins*Integer (Required)The number of bins. Recommended values are between 5 and 14.
  typeString (Required)Bin type. Values are: UnivariateContinuous
  bin_weights*List[float] (Optional)The weights assigned to the assay bins.
  bin_widthList[float] (Optional)The width assigned to the assay bins.
  provided_edgesList[float] (Optional)The edges used for the assay bins.
  add_outlier_edgesBoolean (Required)Indicates whether to add outlier edges or not.
warning_threshold  Float (Required)Optional warning threshold.
alert_threshold  Float (Required)Alert threshold.
run_until  String (OPTIONAL)DateTime of when to end the assay.
workspace_id  Integer (Required)The workspace the assay is part of.
model_insights_url  String (OPTIONAL)URL for model insights.

Create Assay Returns

FieldTypeDescription
assay_idIntegerThe id of the new assay.

Create Assay Examples

The following example code will:

  • Create an assay from the endpoint /v1/api/assays/create with baseline generated from Create Baseline Summary Statistics.
    • For our example, we will be using the output of the field dense_2 at the index 0 for the iopath.
  • The assay id is stored for later examples.

Create assay via Requests.

import datetime

assay_window_end = datetime.datetime.now(datetime.timezone.utc)

## Create assay

# Retrieve the token 
headers = wl.auth.auth_header()

endpoint = f"{wl.api_endpoint}/v1/api/assays/create"

# generate random values so the assay is always unique
import string
import random

# # make a random 4 character suffix to prevent overwriting other user's workspaces
suffix= ''.join(random.choice(string.ascii_lowercase) for i in range(4))

exampleAssayName = f"api assay example {suffix}"
# display(exampleAssayName)

data = {
    "name": exampleAssayName,
    "pipeline_id": model_pipeline_id,
    "pipeline_name": pipeline_name,
    "active": True,
    "status": "created",
    "baseline": {
        "static": baseline_summary
    },
    "window": {
        "pipeline_name": pipeline_name,
        "model_name": example_model_name,
        "width": "1 minute",
        "start": assay_start.isoformat(),
        "interval": "1 minute",
        "path": "output dense_1 0",
        "workspace_id": workspace_id,
        "locations": []
    },
    "summarizer": {
        "type": "UnivariateContinuous",
        "bin_mode": "Quantile",
        "aggregation": "Density",
        "metric": "PSI",
        "num_bins": 5,
        "bin_weights": None,
        "bin_width": None,
        "provided_edges": None,
        "add_outlier_edges": None
    },
    "warning_threshold": None,
    "alert_threshold": 0.25,
    "run_until": assay_window_end.isoformat(),
    "workspace_id": workspace_id
}

print(data)

response = requests.post(endpoint, json=data, headers=headers, verify=True).json()
assay_id = response['assay_id']
response
{'name': 'api assay example fjii', 'pipeline_id': 6, 'pipeline_name': 'api-pipeline-with-models', 'active': True, 'status': 'created', 'baseline': {'static': {'count': 1001, 'min': 1.519918441772461e-06, 'max': 1.0, 'mean': 0.0065982516233499475, 'median': 0.0005419552326202393, 'std': 0.07661715908141875, 'edges': [1.519918441772461e-06, 0.00026360154151916504, 0.0004509389400482178, 0.0006800591945648193, 0.000969529151916504, 1.0, None], 'edge_names': ['left_outlier', 'q_20', 'q_40', 'q_60', 'q_80', 'q_100', 'right_outlier'], 'aggregated_values': [0.0, 0.1998001998001998, 0.1998001998001998, 0.2007992007992008, 0.1998001998001998, 0.1998001998001998, 0.0], 'aggregation': 'Density', 'start': '2024-01-09T19:07:59.566516Z', 'end': '2024-01-10T19:07:59.566516Z'}}, 'window': {'pipeline_name': 'api-pipeline-with-models', 'model_name': 'api-sample-model', 'width': '1 minute', 'start': '2024-01-09T19:07:59.566516+00:00', 'interval': '1 minute', 'path': 'output dense_1 0', 'workspace_id': 7, 'locations': []}, 'summarizer': {'type': 'UnivariateContinuous', 'bin_mode': 'Quantile', 'aggregation': 'Density', 'metric': 'PSI', 'num_bins': 5, 'bin_weights': None, 'bin_width': None, 'provided_edges': None, 'add_outlier_edges': None}, 'warning_threshold': None, 'alert_threshold': 0.25, 'run_until': '2024-01-09T19:09:34.063643+00:00', 'workspace_id': 7}

{‘assay_id’: 9}

Create assay via curl.

exampleAssayName = f"api assay curl examples {suffix}"

data = {
    "name": exampleAssayName,
    "pipeline_id": model_pipeline_id,
    "pipeline_name": pipeline_name,
    "active": True,
    "status": "created",
    "baseline": {
        "static": baseline_summary
    },
    "window": {
        "pipeline_name": pipeline_name,
        "model_name": example_model_name,
        "width": "24 hours",
        "start": assay_start.isoformat(),
        "interval": "24 hours",
        "path": "output dense_1 0",
        "workspace_id": workspace_id,
        "locations": []
    },
    "summarizer": {
        "type": "UnivariateContinuous",
        "bin_mode": "Quantile",
        "aggregation": "Density",
        "metric": "PSI",
        "num_bins": 5,
        "bin_weights": None,
        "bin_width": None,
        "provided_edges": None,
        "add_outlier_edges": None
    },
    "warning_threshold": None,
    "alert_threshold": 0.25,
    "run_until": assay_window_end.isoformat(),
    "workspace_id": workspace_id
}

!curl {wl.api_endpoint}/v1/api/assays/create \
    -H "Authorization: {wl.auth.auth_header()['Authorization']}" \
    -H "Content-Type: application/json" \
    --data '{json.dumps(data)}'
{"assay_id":10}

List Assays

  • Endpoint: /v1/api/assays/list

Lists all assays in the specified pipeline.

List Assays Parameters

FieldTypeDescription
pipeline_idInteger (Required)The numerical ID of the pipeline.

List Assays Returns

A list of assays with the following fields.

Field  TypeDescription
id  IntegerThe numerical identifier for the assay.
name  StringThe name of the assay.
active  BooleanWhether the assay is Active or not.
status  StringThe status of the assay. Options are: active: the Assay is created and active. created: The assay is created but inactive.
warning_threshold  Float (Required)Optional warning threshold.
alert_threshold  Float (Required)Alert threshold.
pipeline_id  StringThe numerical identifier the assay will be placed into.
pipeline_name  StringThe name of the pipeline
last_run  StringThe DateTime for the last time the assay ran.
next_run  StringThe DateTime for the next schedule assay run.
run_until  StringThe DateTime when the assay will stop running.
updated_at  StringThe DateTime when the assay was last updated.
iopath  StringThe iopath of the assay in the format `“input
baseline  BaselineThe baseline for the assay. This is determined by how the assay was generated. See Create Assay Parameters for complete values.
window  AssayWindowAssay window.
 model_name StringThe name of the model used for the assay window.
 pipeline_name StringThe name of the pipeline for the assay window.
 path StringThe iopath of the assay window.
 start StringThe DateTime of when to start the assay window.
 width StringThe width of the assay window.
 workspace_id StringThe width of the assay window.
 locations List[String]The locations included in the assay. An empty set indicates only the Wallaroo Ops deployment of the pipeline is used.
summarizer  (AssaySummerizer)The summarizer type for the array aka “advanced settings” in the Wallaroo Dashboard UI.
 type StringType of summarizer.
  bin_modeStringThe binning model type. Values are: Quantile and Equal
  aggregationStringAggregation type. Values are: Cumulative, Density, and Edges.
  metricStringMetric type. Values are: PSI, MaxDiff (Maximum Difference of Bins), and SumDiff (Sum of the Difference of Bins)
  num_bins*IntegerThe number of bins. Recommended values are between 5 and 14.
  bin_weights*List[float]The weights assigned to the assay bins.
  provided_edgesList[float]The edges used for the assay bins.

List Assays Examples

Display a list of all assays in a related to a pipeline.

List all pipeline assays via Requests.

# Get assays
# Retrieve the token 
headers = wl.auth.auth_header()

endpoint = f"{wl.api_endpoint}/v1/api/assays/list"

data = {
    "pipeline_id": model_pipeline_id
}

response = requests.post(endpoint, json=data, headers=headers, verify=True).json()
response[0]
{'id': 10,
 'name': 'api assay curl examples fjii',
 'active': True,
 'status': 'created',
 'warning_threshold': None,
 'alert_threshold': 0.25,
 'pipeline_id': 6,
 'pipeline_name': 'api-pipeline-with-models',
 'last_run': None,
 'next_run': '2024-01-09T19:10:00.429482+00:00',
 'run_until': '2024-01-09T19:09:34.063643+00:00',
 'updated_at': '2024-01-09T19:10:00.432239+00:00',
 'baseline': {'static': {'count': 1001,
   'min': 1.519918441772461e-06,
   'max': 1.0,
   'mean': 0.0065982516233499475,
   'median': 0.0005419552326202393,
   'std': 0.07661715908141875,
   'edges': [1.519918441772461e-06,
    0.00026360154151916504,
    0.0004509389400482178,
    0.0006800591945648193,
    0.000969529151916504,
    1.0,
    None],
   'edge_names': ['left_outlier',
    'q_20',
    'q_40',
    'q_60',
    'q_80',
    'q_100',
    'right_outlier'],
   'aggregated_values': [0.0,
    0.1998001998001998,
    0.1998001998001998,
    0.2007992007992008,
    0.1998001998001998,
    0.1998001998001998,
    0.0],
   'aggregation': 'Density',
   'start': '2024-01-09T19:07:59.566516Z',
   'end': '2024-01-10T19:07:59.566516Z'}},
 'window': {'interval': '24 hours',
  'model_name': 'api-sample-model',
  'path': 'output dense_1 0',
  'pipeline_name': 'api-pipeline-with-models',
  'start': '2024-01-09T19:07:59.566516Z',
  'width': '24 hours',
  'workspace_id': 7,
  'locations': []},
 'summarizer': {'type': 'UnivariateContinuous',
  'bin_mode': 'Quantile',
  'aggregation': 'Density',
  'metric': 'PSI',
  'num_bins': 5,
  'bin_weights': None,
  'provided_edges': None}}

List all pipeline assays via Curl.

data = {
    "pipeline_id": model_pipeline_id
}

!curl {wl.api_endpoint}/v1/api/assays/list \
    -H "Authorization: {wl.auth.auth_header()['Authorization']}" \
    -H "Content-Type: application/json" \
    --data '{json.dumps(data)}'
[{"id":10,"name":"api assay curl examples fjii","active":true,"status":"created","warning_threshold":null,"alert_threshold":0.25,"pipeline_id":6,"pipeline_name":"api-pipeline-with-models","last_run":null,"next_run":"2024-01-09T19:10:00.429482+00:00","run_until":"2024-01-09T19:09:34.063643+00:00","updated_at":"2024-01-09T19:10:00.432239+00:00","baseline":{"static":{"count":1001,"min":1.519918441772461e-6,"max":1.0,"mean":0.0065982516233499475,"median":0.0005419552326202393,"std":0.07661715908141875,"edges":[1.519918441772461e-6,0.00026360154151916504,0.0004509389400482178,0.0006800591945648193,0.000969529151916504,1.0,null],"edge_names":["left_outlier","q_20","q_40","q_60","q_80","q_100","right_outlier"],"aggregated_values":[0.0,0.1998001998001998,0.1998001998001998,0.2007992007992008,0.1998001998001998,0.1998001998001998,0.0],"aggregation":"Density","start":"2024-01-09T19:07:59.566516Z","end":"2024-01-10T19:07:59.566516Z"}},"window":{"interval":"24 hours","model_name":"api-sample-model","path":"output dense_1 0","pipeline_name":"api-pipeline-with-models","start":"2024-01-09T19:07:59.566516Z","width":"24 hours","workspace_id":7,"locations":[]},"summarizer":{"type":"UnivariateContinuous","bin_mode":"Quantile","aggregation":"Density","metric":"PSI","num_bins":5,"bin_weights":null,"provided_edges":null}},{"id":9,"name":"api assay example fjii","active":true,"status":"{\"run_at\": \"2024-01-09T19:10:01.369980660+00:00\",  \"num_ok\": 0, \"num_warnings\": 0, \"num_alerts\": 0}","warning_threshold":null,"alert_threshold":0.25,"pipeline_id":6,"pipeline_name":"api-pipeline-with-models","last_run":"2024-01-09T19:10:01.369981+00:00","next_run":"2024-01-09T19:09:59.566516+00:00","run_until":"2024-01-09T19:09:34.063643+00:00","updated_at":"2024-01-09T19:09:34.741218+00:00","baseline":{"static":{"count":1001,"min":1.519918441772461e-6,"max":1.0,"mean":0.0065982516233499475,"median":0.0005419552326202393,"std":0.07661715908141875,"edges":[1.519918441772461e-6,0.00026360154151916504,0.0004509389400482178,0.0006800591945648193,0.000969529151916504,1.0,null],"edge_names":["left_outlier","q_20","q_40","q_60","q_80","q_100","right_outlier"],"aggregated_values":[0.0,0.1998001998001998,0.1998001998001998,0.2007992007992008,0.1998001998001998,0.1998001998001998,0.0],"aggregation":"Density","start":"2024-01-09T19:07:59.566516Z","end":"2024-01-10T19:07:59.566516Z"}},"window":{"interval":"1 minute","model_name":"api-sample-model","path":"output dense_1 0","pipeline_name":"api-pipeline-with-models","start":"2024-01-09T19:07:59.566516Z","width":"1 minute","workspace_id":7,"locations":[]},"summarizer":{"type":"UnivariateContinuous","bin_mode":"Quantile","aggregation":"Density","metric":"PSI","num_bins":5,"bin_weights":null,"provided_edges":null}}]

Activate or Deactivate Assay

  • Endpoint: /v1/api/assays/set_active

Activates or deactivates an existing assay.

Activate or Deactivate Assay Parameters

FieldTypeDescription
idInteger (Required)The id of the new assay.
activeBoolean (Required)True to activate the assay, False to deactivate it.

Activate or Deactivate Assay Returns

FieldTypeDescription
idThe numerical id of the assay.
activeBooleanTrue to activate the assay, False to deactivate it.

Activate or Deactivate Assay Examples

The recently created assay will be deactivated then activated.

Deactivate and activate an assay via Requests

# Deactivate assay

# Retrieve the token 
headers = wl.auth.auth_header()

endpoint = f"{wl.api_endpoint}/v1/api/assays/set_active"

data = {
    'id': assay_id,
    'active': False
}

response = requests.post(endpoint, json=data, headers=headers, verify=True).json()
response
{'id': 9, 'active': False}
# Activate assay

# Retrieve the token 
headers = wl.auth.auth_header()

endpoint = f"{wl.api_endpoint}/v1/api/assays/set_active"

data = {
    'id': assay_id,
    'active': True
}

response = requests.post(endpoint, json=data, headers=headers, verify=True).json()
response
{'id': 9, 'active': True}

Deactivate and activate an assay via curl

data = {
    'id': assay_id,
    'active': False
}

!curl {wl.api_endpoint}/v1/api/assays/set_active \
    -H "Authorization: {wl.auth.auth_header()['Authorization']}" \
    -H "Content-Type: application/json" \
    --data '{json.dumps(data)}'
{"id":9,"active":false}
data = {
    'id': assay_id,
    'active': True
}

!curl {wl.api_endpoint}/v1/api/assays/set_active \
    -H "Authorization: {wl.auth.auth_header()['Authorization']}" \
    -H "Content-Type: application/json" \
    --data '{json.dumps(data)}'
{"id":9,"active":true}

Get Assay Baseline

  • Endpoint: /v1/api/assays/get_baseline

Get Assay Baseline Parameters

FieldTypeDescription
workspace_idInteger (Required)Numerical id for the workspace the assay is in.
pipeline_nameString (Required)Name of the pipeline the assay is in.
startString (Optional)DateTime for when the baseline starts.
endString (Optional)DateTime for when the baseline ends.
model_nameString (Optional)Name of the model.
limitInteger (Optional)Maximum number of baselines to return.

Get Assay Baseline Returns

Returns the assay baseline as defined in Create Assay.

Get Assay Baseline Examples

The following examples show retrieving the first three baselines from the recently created baseline.

Retrieve assay baseline via Requests.

## Get Assay Baseline

# Retrieve the token 
headers = wl.auth.auth_header()

endpoint = f"{wl.api_endpoint}/v1/api/assays/get_baseline"

data = {
    'workspace_id': workspace_id,
    'pipeline_name': pipeline_with_models_id,
    'limit': 3
}

response = requests.post(endpoint, json=data, headers=headers, verify=True).json()
response[0]
{'time': 1704827224734,
 'in.dense_input': [-1.0603297501,
  2.3544967095,
  -3.5638788326,
  5.1387348926,
  -1.2308457019,
  -0.7687824608,
  -3.5881228109,
  1.8880837663,
  -3.2789674274,
  -3.9563254554,
  4.0993439118,
  -5.6539176395,
  -0.8775733373,
  -9.131571192,
  -0.6093537873,
  -3.7480276773,
  -5.0309125017,
  -0.8748149526,
  1.9870535692,
  0.7005485718,
  0.9204422758,
  -0.1041491809,
  0.3229564351,
  -0.7418141657,
  0.0384120159,
  1.0993439146,
  1.2603409756,
  -0.1466244739,
  -1.4463212439],
 'out.dense_1': [0.99300325],
 'check_failures': [],
 'metadata.last_model': '{"model_name":"api-sample-model","model_sha":"bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507"}',
 'metadata.pipeline_version': '2f4bf3ac-48bb-4798-b59a-81de844248d1',
 'metadata.elapsed': [5039007, 602225],
 'metadata.dropped': [],
 'metadata.partition': 'engine-b4789c964-cnjjb'}

Retrieve assay baseline via curl.

data = {
    'workspace_id': workspace_id,
    'pipeline_name': pipeline_with_models_id,
    'limit': 3
}

!curl {wl.api_endpoint}/v1/api/assays/get_baseline \
    -H "Authorization: {wl.auth.auth_header()['Authorization']}" \
    -H "Content-Type: application/json" \
    --data '{json.dumps(data)}'
[{"time":1704827224734,"in.dense_input":[-1.0603297501,2.3544967095,-3.5638788326,5.1387348926,-1.2308457019,-0.7687824608,-3.5881228109,1.8880837663,-3.2789674274,-3.9563254554,4.0993439118,-5.6539176395,-0.8775733373,-9.131571192,-0.6093537873,-3.7480276773,-5.0309125017,-0.8748149526,1.9870535692,0.7005485718,0.9204422758,-0.1041491809,0.3229564351,-0.7418141657,0.0384120159,1.0993439146,1.2603409756,-0.1466244739,-1.4463212439],"out.dense_1":[0.99300325],"check_failures":[],"metadata.last_model":"{\"model_name\":\"api-sample-model\",\"model_sha\":\"bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507\"}","metadata.pipeline_version":"2f4bf3ac-48bb-4798-b59a-81de844248d1","metadata.elapsed":[5039007,602225],"metadata.dropped":[],"metadata.partition":"engine-b4789c964-cnjjb"},{"time":1704827224734,"in.dense_input":[-1.0603297501,2.3544967095,-3.5638788326,5.1387348926,-1.2308457019,-0.7687824608,-3.5881228109,1.8880837663,-3.2789674274,-3.9563254554,4.0993439118,-5.6539176395,-0.8775733373,-9.131571192,-0.6093537873,-3.7480276773,-5.0309125017,-0.8748149526,1.9870535692,0.7005485718,0.9204422758,-0.1041491809,0.3229564351,-0.7418141657,0.0384120159,1.0993439146,1.2603409756,-0.1466244739,-1.4463212439],"out.dense_1":[0.99300325],"check_failures":[],"metadata.last_model":"{\"model_name\":\"api-sample-model\",\"model_sha\":\"bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507\"}","metadata.pipeline_version":"2f4bf3ac-48bb-4798-b59a-81de844248d1","metadata.elapsed":[5039007,602225],"metadata.dropped":[],"metadata.partition":"engine-b4789c964-cnjjb"},{"time":1704827224734,"in.dense_input":[-1.0603297501,2.3544967095,-3.5638788326,5.1387348926,-1.2308457019,-0.7687824608,-3.5881228109,1.8880837663,-3.2789674274,-3.9563254554,4.0993439118,-5.6539176395,-0.8775733373,-9.131571192,-0.6093537873,-3.7480276773,-5.0309125017,-0.8748149526,1.9870535692,0.7005485718,0.9204422758,-0.1041491809,0.3229564351,-0.7418141657,0.0384120159,1.0993439146,1.2603409756,-0.1466244739,-1.4463212439],"out.dense_1":[0.99300325],"check_failures":[],"metadata.last_model":"{\"model_name\":\"api-sample-model\",\"model_sha\":\"bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507\"}","metadata.pipeline_version":"2f4bf3ac-48bb-4798-b59a-81de844248d1","metadata.elapsed":[5039007,602225],"metadata.dropped":[],"metadata.partition":"engine-b4789c964-cnjjb"},{"time":1704827224734,"in.dense_input":[-1.0603297501,2.3544967095,-3.5638788326,5.1387348926,-1.2308457019,-0.7687824608,-3.5881228109,1.8880837663,-3.2789674274,-3.9563254554,4.0993439118,-5.6539176395,-0.8775733373,-9.131571192,-0.6093537873,-3.7480276773,-5.0309125017,-0.8748149526,1.9870535692,0.7005485718,0.9204422758,-0.1041491809,0.3229564351,-0.7418141657,0.0384120159,1.0993439146,1.2603409756,-0.1466244739,-1.4463212439],"out.dense_1":[0.99300325],"check_failures":[],"metadata.last_model":"{\"model_name\":\"api-sample-model\",\"model_sha\":\"bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507\"}","metadata.pipeline_version":"2f4bf3ac-48bb-4798-b59a-81de844248d1","metadata.elapsed":[5039007,602225],"metadata.dropped":[],"metadata.partition":"engine-b4789c964-cnjjb"},{"time":1704827224734,"in.dense_input":[0.5817662108,0.097881551,0.1546819424,0.4754101949,-0.1978862306,-0.4504344854,0.0166540447,-0.0256070551,0.0920561602,-0.2783917153,0.0593299441,-0.0196585416,-0.4225083157,-0.1217538877,1.5473094894,0.2391622864,0.3553974881,-0.7685165301,-0.7000849355,-0.1190043285,-0.3450517133,-1.1065114108,0.2523411195,0.0209441826,0.2199267436,0.2540689265,-0.0450225094,0.1086773898,0.2547179311],"out.dense_1":[0.0010916889],"check_failures":[],"metadata.last_model":"{\"model_name\":\"api-sample-model\",\"model_sha\":\"bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507\"}","metadata.pipeline_version":"2f4bf3ac-48bb-4798-b59a-81de844248d1","metadata.elapsed":[5039007,602225],"metadata.dropped":[],"metadata.partition":"engine-b4789c964-cnjjb"},{"time":1704827224734,"in.dense_input":[-0.7621273681,0.8854701414,0.5235808652,-0.8139743551,0.3793240544,0.1560653365,0.5451299665,0.0785927242,0.4143968543,0.4914052348,0.0774391022,1.050105026,0.9901440217,-0.6142483131,-1.5260740653,0.2053324703,-1.0185637854,0.0490986919,0.6964184879,0.5948331722,-0.3934362922,-0.592249266,-0.3953093077,-1.3310427025,0.6287441287,0.8665525996,0.7974673604,1.1174342262,-0.6700716551],"out.dense_1":[0.00047266483],"check_failures":[],"metadata.last_model":"{\"model_name\":\"api-sample-model\",\"model_sha\":\"bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507\"}","metadata.pipeline_version":"2f4bf3ac-48bb-4798-b59a-81de844248d1","metadata.elapsed":[5039007,602225],"metadata.dropped":[],"metadata.partition":"engine-b4789c964-cnjjb"},{"time":1704827224734,"in.dense_input":[-0.2836830107,0.2281341608,1.0358808685,1.0311413647,0.6485053917,0.6993339,0.1827667194,0.0989746212,-0.5734487733,0.5928927597,0.3085637362,0.1533869918,-0.3628347923,-0.2865054499,-1.1380446536,-0.2207117601,-0.1206033931,-0.2325246936,0.8675179232,-0.0081323034,-0.015330415,0.4169237822,-0.4249025314,-0.9834451977,-1.1175903578,2.1076701885,-0.3361950073,-0.3469573431,0.019307669],"out.dense_1":[0.00082170963],"check_failures":[],"metadata.last_model":"{\"model_name\":\"api-sample-model\",\"model_sha\":\"bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507\"}","metadata.pipeline_version":"2f4bf3ac-48bb-4798-b59a-81de844248d1","metadata.elapsed":[5039007,602225],"metadata.dropped":[],"metadata.partition":"engine-b4789c964-cnjjb"},{"time":1704827224734,"in.dense_input":[1.0379636346,-0.152987302,-1.0912561862,-0.0033339828,0.4804281836,0.1120708475,0.0233157709,0.0009213038,0.4021730182,0.2120753712,-0.1462804223,0.4424477027,-0.4641602117,0.498425643,-0.8230270969,0.3168388184,-0.9050440977,0.0710365039,1.1111388587,-0.2157914054,-0.373759129,-1.0330075347,0.3144720913,-0.5109243112,-0.1685910498,0.5918324406,-0.2231792825,-0.2287153377,-0.0868944762],"out.dense_1":[0.0011294782],"check_failures":[],"metadata.last_model":"{\"model_name\":\"api-sample-model\",\"model_sha\":\"bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507\"}","metadata.pipeline_version":"2f4bf3ac-48bb-4798-b59a-81de844248d1","metadata.elapsed":[5039007,602225],"metadata.dropped":[],"metadata.partition":"engine-b4789c964-cnjjb"},{"time":1704827224734,"in.dense_input":[0.1517283662,0.6589966337,-0.3323713647,0.7285871979,0.6430271573,-0.0361051306,0.2201530504,-1.4928731939,-0.5895806487,0.2227251103,0.4443729713,0.8411555815,-0.241291302,0.898682875,-0.9866307096,-0.8919301767,-0.0878875914,0.1163332461,1.1469566646,-0.5417470436,2.2321360563,-0.1679271382,-0.8071223667,-0.6379226787,1.9121889391,-0.5565545169,0.6528273964,0.8163897966,-0.2281615017],"out.dense_1":[0.0018743575],"check_failures":[],"metadata.last_model":"{\"model_name\":\"api-sample-model\",\"model_sha\":\"bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507\"}","metadata.pipeline_version":"2f4bf3ac-48bb-4798-b59a-81de844248d1","metadata.elapsed":[5039007,602225],"metadata.dropped":[],"metadata.partition":"engine-b4789c964-cnjjb"},{"time":1704827224734,"in.dense_input":[-0.1683100246,0.7070470317,0.1875234948,-0.3885406952,0.8190382137,-0.2264929889,0.9204469154,-0.1362740974,-0.3336344399,-0.3174281686,1.1903478934,0.1774292097,-0.5681631429,-0.8907063935,-0.5603225649,0.0897831737,0.4187525906,0.3406269046,0.7358794384,0.2162316926,-0.4090832915,-0.8736089461,-0.1128706509,1.0027861774,-0.9404916154,0.3447144641,0.0908233867,0.0338594886,-1.529552268],"out.dense_1":[0.0011520088],"check_failures":[],"metadata.last_model":"{\"model_name\":\"api-sample-model\",\"model_sha\":\"bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507\"}","metadata.pipeline_version":"2f4bf3ac-48bb-4798-b59a-81de844248d1","metadata.elapsed":[5039007,602225],"metadata.dropped":[],"metadata.partition":"engine-b4789c964-cnjjb"},{"time":1704827224734,"in.dense_input":[0.6066235674,0.0631839305,-0.0802961973,0.6955262345,-0.1775255859,-0.3757158261,-0.1003478381,-0.0020206974,0.6859442462,-0.6582840559,-0.9995187666,-0.5340094458,-1.1303344302,-1.4048093395,-0.0953316119,0.3428650708,1.1376277711,0.4248309202,0.2316384963,-0.1145370746,-0.301586357,-0.6731341245,-0.2723217481,-0.3925227831,1.1115261431,0.9205381913,-0.0280590004,0.1311643902,0.215202258],"out.dense_1":[0.0016568303],"check_failures":[],"metadata.last_model":"{\"model_name\":\"api-sample-model\",\"model_sha\":\"bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507\"}","metadata.pipeline_version":"2f4bf3ac-48bb-4798-b59a-81de844248d1","metadata.elapsed":[5039007,602225],"metadata.dropped":[],"metadata.partition":"engine-b4789c964-cnjjb"},{"time":1704827224734,"in.dense_input":[0.6022605286,0.0335418852,0.073849277,0.1851178536,-0.3054855539,-0.7940218337,0.1654941906,-0.1303600246,-0.1884158694,0.0665975781,1.4810974231,0.6472122045,-0.6703196484,0.7565686747,0.2134058731,0.1505475751,-0.4312378589,-0.0182951925,0.2851995511,-0.1076509026,0.0068242826,-0.1076589048,-0.0788026491,0.9475328125,0.8413388083,1.1769860739,-0.2026212206,-0.0006311265,0.1851559549],"out.dense_1":[0.0010267198],"check_failures":[],"metadata.last_model":"{\"model_name\":\"api-sample-model\",\"model_sha\":\"bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507\"}","metadata.pipeline_version":"2f4bf3ac-48bb-4798-b59a-81de844248d1","metadata.elapsed":[5039007,602225],"metadata.dropped":[],"metadata.partition":"engine-b4789c964-cnjjb"},{"time":1704827224734,"in.dense_input":[-1.2004162236,-0.0293424715,0.6002673903,-1.0581165764,0.8618826503,0.9173564432,0.0753151511,0.2206189225,1.2188735091,-0.388652383,-0.609512583,0.1965043267,-0.2661495952,-0.6379133677,0.483398342,-0.4985531207,-0.3064243289,-1.4524496793,-3.1140699631,-1.0750208206,0.3341223842,1.5687760942,-0.5201671364,-0.5317761577,-0.3832949469,-0.9846864507,-2.8976275684,-0.507351229,-0.3252693381],"out.dense_1":[0.00019043684],"check_failures":[],"metadata.last_model":"{\"model_name\":\"api-sample-model\",\"model_sha\":\"bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507\"}","metadata.pipeline_version":"2f4bf3ac-48bb-4798-b59a-81de844248d1","metadata.elapsed":[5039007,602225],"metadata.dropped":[],"metadata.partition":"engine-b4789c964-cnjjb"},{"time":1704827224734,"in.dense_input":[-2.8427357031,2.8260142811,-1.595334492,-0.2991672886,-1.5495220405,1.6401772163,-3.2821951849,0.4863028451,0.3576801276,0.3222372163,0.2710846269,2.0025589608,-1.2151492104,2.2835338744,-0.4148465662,-0.6786230741,2.6106103198,-1.6468850705,-1.6080123756,0.9995474223,-0.4665201753,0.8316050292,1.2855678736,-2.3561879048,0.2107938402,1.1256706306,1.1538189945,1.033206103,-1.4715524921],"out.dense_1":[0.00032365322],"check_failures":[],"metadata.last_model":"{\"model_name\":\"api-sample-model\",\"model_sha\":\"bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507\"}","metadata.pipeline_version":"2f4bf3ac-48bb-4798-b59a-81de844248d1","metadata.elapsed":[5039007,602225],"metadata.dropped":[],"metadata.partition":"engine-b4789c964-cnjjb"},{"time":1704827224734,"in.dense_input":[0.3702516863,-0.559565197,0.6757255982,0.6920586122,-1.0975950837,-0.3864326068,-0.2464826411,-0.0304654993,0.6888634288,-0.2568824022,-0.3126212674,0.4177154049,0.0865827265,-0.235865361,0.7111355283,0.2937196012,-0.2150665771,-0.037999824,-0.5299304856,0.4921897196,0.3506791137,0.4934112482,-0.3645305682,1.304678649,0.3883791808,1.0901335639,-0.098187589,0.2221143885,1.2586007559],"out.dense_1":[0.00062185526],"check_failures":[],"metadata.last_model":"{\"model_name\":\"api-sample-model\",\"model_sha\":\"bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507\"}","metadata.pipeline_version":"2f4bf3ac-48bb-4798-b59a-81de844248d1","metadata.elapsed":[5039007,602225],"metadata.dropped":[],"metadata.partition":"engine-b4789c964-cnjjb"},{"time":1704827224734,"in.dense_input":[1.0740600995,-0.5004444123,-0.6665104459,-0.5137954506,-0.221802514,0.1734011149,-0.6004987142,0.0105770193,-0.4816997217,0.8861570874,0.1852976978,0.8741913982,1.1965272846,-0.1099339687,-0.6057668522,-1.1254346438,-0.7830480708,1.9148497747,-0.3837603969,-0.6387752587,-0.4853295483,-0.5961116066,0.4123371083,0.1760369794,-0.5173803145,1.1181808797,-0.0934318755,-0.1756922307,-0.2551430327],"out.dense_1":[0.00033929944],"check_failures":[],"metadata.last_model":"{\"model_name\":\"api-sample-model\",\"model_sha\":\"bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507\"}","metadata.pipeline_version":"2f4bf3ac-48bb-4798-b59a-81de844248d1","metadata.elapsed":[5039007,602225],"metadata.dropped":[],"metadata.partition":"engine-b4789c964-cnjjb"},{"time":1704827224734,"in.dense_input":[-0.3389501954,0.4600633973,1.5422134203,0.0267389924,0.1158931731,0.504544689,0.0516362685,0.2645286362,1.2691985832,-1.0147580415,0.7932464418,-2.0832021155,1.2439082485,1.1713997667,-2.0616089222,0.7783038412,-0.374691853,1.3187633761,-0.5760757781,-0.2563693397,-0.2008462631,-0.1523985678,-0.5701108144,-0.9852939717,0.3654040112,-1.1771122816,0.2181815201,0.2822386297,-0.6710689243],"out.dense_1":[0.00094124675],"check_failures":[],"metadata.last_model":"{\"model_name\":\"api-sample-model\",\"model_sha\":\"bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507\"}","metadata.pipeline_version":"2f4bf3ac-48bb-4798-b59a-81de844248d1","metadata.elapsed":[5039007,602225],"metadata.dropped":[],"metadata.partition":"engine-b4789c964-cnjjb"},{"time":1704827224734,"in.dense_input":[-0.1743934044,0.5772185632,0.8490726828,-0.1038906591,0.0252342973,-0.4811598747,0.4933987162,0.0493502938,-0.1311069416,-0.2281717828,-1.0658145716,-1.0302882811,-1.434383579,0.4349549767,1.1462514329,0.4591260283,-0.5472935959,0.0172749694,0.2631614897,0.0145841831,-0.3874654386,-1.1470861804,-0.1080720714,-0.3041743294,-0.3364262981,0.2217097324,0.6161928746,0.3643105274,-1.1844813719],"out.dense_1":[0.00040614605],"check_failures":[],"metadata.last_model":"{\"model_name\":\"api-sample-model\",\"model_sha\":\"bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507\"}","metadata.pipeline_version":"2f4bf3ac-48bb-4798-b59a-81de844248d1","metadata.elapsed":[5039007,602225],"metadata.dropped":[],"metadata.partition":"engine-b4789c964-cnjjb"},{"time":1704827224734,"in.dense_input":[-0.2332974393,0.6800384852,1.5082537441,1.902665731,-0.4182576747,0.6195860859,0.2622512804,-0.0835581768,-0.4142605796,0.5189395936,0.3581939017,-2.8619533966,3.0848026022,1.055798472,0.7858524275,0.3180247098,1.0839174042,0.2026623381,1.7253719223,0.4795673018,-0.0043843842,0.4299095507,-0.2194886598,0.1370990841,-0.6193370967,5.7056185317,-0.3816283254,0.0847421447,0.820767133],"out.dense_1":[0.00014156103],"check_failures":[],"metadata.last_model":"{\"model_name\":\"api-sample-model\",\"model_sha\":\"bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507\"}","metadata.pipeline_version":"2f4bf3ac-48bb-4798-b59a-81de844248d1","metadata.elapsed":[5039007,602225],"metadata.dropped":[],"metadata.partition":"engine-b4789c964-cnjjb"},{"time":1704827224734,"in.dense_input":[-0.1576920792,0.5255560704,0.5355833824,-0.3921080777,0.5239687507,0.3137754019,0.4081535059,0.2258116533,-0.3212348791,-0.0856048598,-0.4439710421,-0.6280442182,-1.0677764986,0.5136290082,0.4487000903,1.0692428642,-1.3535798799,1.0473038479,1.196566953,0.1138327061,-0.4203194737,-1.2798508059,-0.3799819591,-2.444457856,-0.0622016625,0.3411696338,0.6030329737,0.2957255243,-0.6235003345],"out.dense_1":[0.0006403029],"check_failures":[],"metadata.last_model":"{\"model_name\":\"api-sample-model\",\"model_sha\":\"bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507\"}","metadata.pipeline_version":"2f4bf3ac-48bb-4798-b59a-81de844248d1","metadata.elapsed":[5039007,602225],"metadata.dropped":[],"metadata.partition":"engine-b4789c964-cnjjb"},{"time":1704827224734,"in.dense_input":[0.5821285026,0.1104055282,0.0005528572,0.35847768,0.0009258978,-0.5660804934,0.3656322942,-0.1789639774,-0.2540332045,-0.1110368122,0.1015306466,0.59766274,0.4702091968,0.44627195,1.0536294201,-0.3214604285,0.0087329979,-1.611796012,-0.3327904927,-0.0192282074,-0.4490871711,-1.4274565627,0.2396383569,0.1392294253,0.4135983881,0.3334319006,-0.1338884357,0.0567416816,0.4859567322],"out.dense_1":[0.0008019507],"check_failures":[],"metadata.last_model":"{\"model_name\":\"api-sample-model\",\"model_sha\":\"bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507\"}","metadata.pipeline_version":"2f4bf3ac-48bb-4798-b59a-81de844248d1","metadata.elapsed":[5039007,602225],"metadata.dropped":[],"metadata.partition":"engine-b4789c964-cnjjb"},{"time":1704827224734,"in.dense_input":[-0.0333590017,0.72533966,-0.3903269517,-0.5782608278,0.8241545254,-0.2370125504,0.7091515215,0.0588503252,-0.1144217342,-0.3106048523,0.7264094916,0.5107899987,0.091243163,-0.944940746,-1.0101438628,0.6696018979,-0.0273902198,0.4503850675,0.2018647107,0.2297016577,-0.4741291427,-1.130144259,0.0881564968,0.2847338898,-0.6596809902,0.2495021965,0.7838420374,0.4252710087,-1.529552268],"out.dense_1":[0.0011220276],"check_failures":[],"metadata.last_model":"{\"model_name\":\"api-sample-model\",\"model_sha\":\"bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507\"}","metadata.pipeline_version":"2f4bf3ac-48bb-4798-b59a-81de844248d1","metadata.elapsed":[5039007,602225],"metadata.dropped":[],"metadata.partition":"engine-b4789c964-cnjjb"},{"time":1704827224734,"in.dense_input":[0.996120637,-0.070714643,-0.075803509,0.375222792,-0.4103418483,-0.5206063577,-0.4036313477,-0.1552825032,2.2485933197,-0.5432778503,0.3780502341,-1.6682524862,2.3059322904,1.4362260723,0.3328187169,0.1123140088,0.1167606432,0.1679027741,-0.5310203866,-0.3375143864,-0.3548673107,-0.4578689929,0.640507891,-0.206852076,-0.7297064476,-1.9848061122,0.0908649427,-0.0923745124,-1.0974333077],"out.dense_1":[0.0007892251],"check_failures":[],"metadata.last_model":"{\"model_name\":\"api-sample-model\",\"model_sha\":\"bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507\"}","metadata.pipeline_version":"2f4bf3ac-48bb-4798-b59a-81de844248d1","metadata.elapsed":[5039007,602225],"metadata.dropped":[],"metadata.partition":"engine-b4789c964-cnjjb"},{"time":1704827224734,"in.dense_input":[0.6669037649,0.1705099542,-0.0474385403,-0.0406692165,0.0221413792,-0.7551039228,0.3490489213,-0.3029571738,-0.2439987474,-0.1523054091,-0.3251253997,0.8622392734,1.5689607531,0.117906692,0.7897991483,0.2109904707,-0.5235865272,-1.1954295371,0.5662182824,0.060217664,-0.6320622503,-1.8353984748,0.226614813,-0.1215965129,0.4161900406,1.3380167293,-0.2317966956,0.018291942,-0.2108207026],"out.dense_1":[0.00040519238],"check_failures":[],"metadata.last_model":"{\"model_name\":\"api-sample-model\",\"model_sha\":\"bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507\"}","metadata.pipeline_version":"2f4bf3ac-48bb-4798-b59a-81de844248d1","metadata.elapsed":[5039007,602225],"metadata.dropped":[],"metadata.partition":"engine-b4789c964-cnjjb"},{"time":1704827224734,"in.dense_input":[0.595425401,0.1748922456,0.3199916062,1.0054676941,-0.1558116713,-0.2865859824,-0.1008624849,-0.0397099148,1.3330975816,-0.2638554919,0.4846079641,-3.0550595925,0.1049865994,2.1513403248,0.6738207076,-0.1172208139,0.7147722707,-0.0442742046,-1.2792547588,-0.3996199419,-0.0939447634,0.0533509264,-0.0823611777,-0.0309503038,1.0073502749,-0.5888372618,-0.0129730027,0.0412812629,-0.3297778345],"out.dense_1":[0.000813812],"check_failures":[],"metadata.last_model":"{\"model_name\":\"api-sample-model\",\"model_sha\":\"bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507\"}","metadata.pipeline_version":"2f4bf3ac-48bb-4798-b59a-81de844248d1","metadata.elapsed":[5039007,602225],"metadata.dropped":[],"metadata.partition":"engine-b4789c964-cnjjb"},{"time":1704827224734,"in.dense_input":[-0.0744693675,0.3759375447,-0.0939607155,-0.6793985764,1.2705145551,-0.7267385735,0.9701613337,-0.3021118583,-0.046024169,-0.8712874369,-0.8927457215,-0.1990438604,0.1089836046,-1.2071697887,-0.7052919033,0.493177229,-0.0388577771,-0.212483322,-0.7632953293,-0.0241397226,-0.3506033969,-0.8706072175,0.1972542306,0.6212803788,-1.8726409047,-0.2064992992,-0.0389139704,0.1145916908,-1.1314557747],"out.dense_1":[0.00046765804],"check_failures":[],"metadata.last_model":"{\"model_name\":\"api-sample-model\",\"model_sha\":\"bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507\"}","metadata.pipeline_version":"2f4bf3ac-48bb-4798-b59a-81de844248d1","metadata.elapsed":[5039007,602225],"metadata.dropped":[],"metadata.partition":"engine-b4789c964-cnjjb"},{"time":1704827224734,"in.dense_input":[0.5147528799,-0.3092386086,0.2202983757,0.3659359406,-0.1553912974,0.5975108346,-0.2690881025,0.1078891231,0.3656514104,-0.1139340204,-0.0256032439,1.377567284,1.6224705773,-0.4696812501,-0.7431821922,0.5846581609,-1.0572315587,0.7425234994,0.8572045757,0.3786566877,0.1544498541,0.4263391768,-0.6959067546,-1.5441769846,1.2422346773,1.1549270147,-0.0404567279,0.0704368742,0.9527486644],"out.dense_1":[0.00020697713],"check_failures":[],"metadata.last_model":"{\"model_name\":\"api-sample-model\",\"model_sha\":\"bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507\"}","metadata.pipeline_version":"2f4bf3ac-48bb-4798-b59a-81de844248d1","metadata.elapsed":[5039007,602225],"metadata.dropped":[],"metadata.partition":"engine-b4789c964-cnjjb"},{"time":1704827224734,"in.dense_input":[-0.5004119071,0.5710679403,-0.7004768368,-1.5298941982,0.1226834696,-0.8023827375,0.5407325829,0.3167065547,-2.3045022923,-0.129750004,0.3272245451,0.189108887,0.842542912,0.9200364366,-1.4378879553,1.0624661174,0.158084957,-1.0738218615,1.243701082,-0.1363782132,0.7573872869,1.6388302051,-0.7735879367,-0.6830006547,1.1140363274,0.125792406,-0.762906004,-0.2095034943,0.5154124157],"out.dense_1":[0.00071662664],"check_failures":[],"metadata.last_model":"{\"model_name\":\"api-sample-model\",\"model_sha\":\"bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507\"}","metadata.pipeline_version":"2f4bf3ac-48bb-4798-b59a-81de844248d1","metadata.elapsed":[5039007,602225],"metadata.dropped":[],"metadata.partition":"engine-b4789c964-cnjjb"},{"time":1704827224734,"in.dense_input":[1.0361011127,-0.0008696652,-0.8793289149,0.3046015897,0.3029690048,-0.4764488549,0.2686106475,-0.2846177954,0.3363357135,0.032398869,-1.1434221195,0.960917481,1.5005986312,-0.0465846933,-0.3294647919,-0.2212449959,-0.5470982629,-1.0241194517,0.406600429,-0.1433458082,-0.3594085848,-0.7389766245,0.3019516292,-1.1330311333,-0.056791208,0.2800813723,-0.1277028101,-0.1886957034,-0.1227703777],"out.dense_1":[0.00077426434],"check_failures":[],"metadata.last_model":"{\"model_name\":\"api-sample-model\",\"model_sha\":\"bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507\"}","metadata.pipeline_version":"2f4bf3ac-48bb-4798-b59a-81de844248d1","metadata.elapsed":[5039007,602225],"metadata.dropped":[],"metadata.partition":"engine-b4789c964-cnjjb"},{"time":1704827224734,"in.dense_input":[-0.1342293161,0.5885043933,0.8013512628,-0.0749059894,0.3626415018,-0.1719553337,0.7418286317,-0.1110203028,-0.2338422399,-0.3124686598,-0.1941903724,-0.1848918535,0.1883050864,-0.6446637405,1.30464823,0.1200699111,0.0585841234,-0.5700200679,-0.3925486208,0.2726402386,-0.4164246671,-0.867875246,-0.0378090556,-0.5754678894,-0.3795274681,0.1956127413,0.2197846989,-0.4012862645,0.0558900763],"out.dense_1":[0.00072047114],"check_failures":[],"metadata.last_model":"{\"model_name\":\"api-sample-model\",\"model_sha\":\"bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507\"}","metadata.pipeline_version":"2f4bf3ac-48bb-4798-b59a-81de844248d1","metadata.elapsed":[5039007,602225],"metadata.dropped":[],"metadata.partition":"engine-b4789c964-cnjjb"},{"time":1704827224734,"in.dense_input":[1.0314610678,0.1866147519,-0.9675649751,0.4416746397,0.4012324483,-0.6225398051,0.1392907242,-0.2993184586,1.4711266523,-0.6794687485,0.5419127166,-1.9156076373,2.5038748264,0.4530600582,-0.6905031983,0.2636870451,1.3335654497,-0.1475923811,-0.2126411897,-0.1763170685,-0.6911300613,-1.4972259053,0.5978433471,0.8760902448,-0.4911312852,0.2635810572,-0.2134215466,-0.0958789049,-0.3778135637],"out.dense_1":[0.0011078417],"check_failures":[],"metadata.last_model":"{\"model_name\":\"api-sample-model\",\"model_sha\":\"bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507\"}","metadata.pipeline_version":"2f4bf3ac-48bb-4798-b59a-81de844248d1","metadata.elapsed":[5039007,602225],"metadata.dropped":[],"metadata.partition":"engine-b4789c964-cnjjb"},{"time":1704827224734,"in.dense_input":[-0.442037716,0.7099227108,0.8526703303,-0.3780673287,-0.1040647304,-0.7837050373,0.4876154094,0.2182253898,-0.7070515247,-0.9084344008,-0.442028478,0.6687432712,1.0360110324,0.2724678839,0.239193594,0.6503611227,-0.6246111339,-0.702817108,-0.7871962026,-0.1578322738,-0.2558803073,-1.0996249805,0.2053001265,0.6436521706,-0.7173240075,-0.1807012827,-0.1780590941,0.1765385302,0.0200562951],"out.dense_1":[0.00034046173],"check_failures":[],"metadata.last_model":"{\"model_name\":\"api-sample-model\",\"model_sha\":\"bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507\"}","metadata.pipeline_version":"2f4bf3ac-48bb-4798-b59a-81de844248d1","metadata.elapsed":[5039007,602225],"metadata.dropped":[],"metadata.partition":"engine-b4789c964-cnjjb"},{"time":1704827224734,"in.dense_input":[1.0306029645,-0.0987504781,-0.6829126331,0.3023232401,-0.1994088505,-0.9453840708,0.0849942154,-0.2079332162,0.659456217,0.1082157136,-0.9329141406,-0.3043515747,-1.3438600066,0.5728034752,0.2017053638,-0.0542533428,-0.228468972,-0.8118631833,0.1884637008,-0.4121132168,-0.4221404823,-1.1464951878,0.6295380124,0.0039466782,-0.6541091926,0.4131260899,-0.1994605585,-0.1881334502,-1.1816916919],"out.dense_1":[0.0009812713],"check_failures":[],"metadata.last_model":"{\"model_name\":\"api-sample-model\",\"model_sha\":\"bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507\"}","metadata.pipeline_version":"2f4bf3ac-48bb-4798-b59a-81de844248d1","metadata.elapsed":[5039007,602225],"metadata.dropped":[],"metadata.partition":"engine-b4789c964-cnjjb"},{"time":1704827224734,"in.dense_input":[1.0075769213,-1.2756438093,0.0521996529,-1.0390974975,-1.0037422762,1.5008410928,-1.8225067582,0.5232012879,-0.1395611951,1.0056699216,-0.8200283624,0.0569786717,0.9611161631,-1.4249620822,-0.4269038458,-1.7077422168,1.776168946,-1.6468299192,-1.3404514972,-0.5501627433,-0.164029694,0.6213204501,0.6916747795,-0.5768679352,-1.2334156564,-0.2610768736,0.3177277072,-0.0819241077,0.4355256115],"out.dense_1":[0.00014293194],"check_failures":[],"metadata.last_model":"{\"model_name\":\"api-sample-model\",\"model_sha\":\"bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507\"}","metadata.pipeline_version":"2f4bf3ac-48bb-4798-b59a-81de844248d1","metadata.elapsed":[5039007,602225],"metadata.dropped":[],"metadata.partition":"engine-b4789c964-cnjjb"},{"time":1704827224734,"in.dense_input":[-0.4858230218,-0.1472918704,0.3581659894,-1.6885973567,-0.4470823069,-0.7588063355,-0.0105622872,0.2224493411,-2.5852776759,0.6018766863,-0.4255550905,-0.4675146284,1.0372911653,0.0259019666,-0.2714938327,-0.7914221044,0.8986040726,-1.0924019978,-1.1911633609,-0.1370313007,-0.415262695,-1.0700475931,0.4559526717,0.0079893336,-0.484857517,-1.2643088032,0.5648047486,0.2477349128,0.6446368299],"out.dense_1":[0.00011792779],"check_failures":[],"metadata.last_model":"{\"model_name\":\"api-sample-model\",\"model_sha\":\"bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507\"}","metadata.pipeline_version":"2f4bf3ac-48bb-4798-b59a-81de844248d1","metadata.elapsed":[5039007,602225],"metadata.dropped":[],"metadata.partition":"engine-b4789c964-cnjjb"},{"time":1704827224734,"in.dense_input":[0.6719021761,-0.4446021252,0.792188472,-0.3198164137,-1.1308394945,-0.2776607632,-0.8503602566,-0.0261966251,-0.1782044773,0.3639999826,-0.4123664061,0.3382654375,1.4787013183,-1.0856503703,-0.2577362351,1.0686611073,0.4525187885,-1.7047869338,0.8341225764,0.2104337681,0.1873331929,0.7545139535,-0.0497730586,0.7401947642,0.7763305379,-0.4149761552,0.1439961625,0.1080152854,-0.0912402128],"out.dense_1":[0.0006335676],"check_failures":[],"metadata.last_model":"{\"model_name\":\"api-sample-model\",\"model_sha\":\"bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507\"}","metadata.pipeline_version":"2f4bf3ac-48bb-4798-b59a-81de844248d1","metadata.elapsed":[5039007,602225],"metadata.dropped":[],"metadata.partition":"engine-b4789c964-cnjjb"},{"time":1704827224734,"in.dense_input":[-0.0276449855,0.6927756366,-0.3744033832,-0.9896722102,1.2325598788,-0.7477087765,1.6407014648,-0.6010296785,-