.

.

Wallaroo Use Case Tutorials

Wallaroo Use Case Tutorials

The following tutorials demonstrate specific use case solutions using Wallaroo. These focus on the main pillars of Wallaroo:

  • Upload
  • Deploy
  • Observe
  • Optimize
  • Automate

These tutorials focus on use cases by ML model type and industry, and provide demonstrations in as many aspects of using ML models in a testing and production environment as possible.

Full tutorials with sample Jupyter notebooks and code are available from the Wallaroo Workshop Repository. Individual use cases are available with models and sample data from the Wallaroo Workshop release page.

Users are recommended to attempt to complete the workshop tutorials themselves with the free Wallaroo Community Edition.

1 - Classification

Wallaroo Use Case Tutorials focused on Classification models

1.1 - Classification: Financial Services

Wallaroo Use Case Tutorials focused on solving Financial problems with Classification models

1.1.1 - Financial Services: Upload and Deploy

How to upload and deploy a financial services classification model to Wallaroo.

Tutorial Notebook 1: Build and Deploy a Model

For this tutorial, let’s pretend that you work for financial institution that determines whether a transaction was more or less likely to be a fraudulent charge based on previous data.

In this set of exercises, you will build a model to predict house sale prices, and deploy it to Wallaroo.

Before we start, let’s load some libraries that we will need for this notebook (note that this may not be a complete list).

  • IMPORTANT NOTE: This tutorial is geared towards a Wallaroo 2023.2.1 instance.
# preload needed libraries 

import wallaroo
from wallaroo.object import EntityNotFoundError
from wallaroo.framework import Framework

from IPython.display import display

# used to display DataFrame information without truncating
from IPython.display import display
import pandas as pd
pd.set_option('display.max_colwidth', None)

import json

import datetime
import time

# used for unique connection names

import string
import random

Exercise: Build a model

This tutorial is geared towards a ML model that outputs a single array value. For example:

{
“prediction”: [0.75]
}

This represents a 0.75% chance that the financial transaction is fraudulent or not. Use data you have included to product a ML model to output a similar output, or one that you choose.

At the end of the exercise, you should have a notebook and possibly other artifacts to produce a model for predicting house prices. For the purposes of the exercise, please use a framework that can be converted to ONNX, such as scikit-learn or XGBoost.

For assistance converting a model to ONNX, see the Wallaroo Model Conversion Tutorials for some examples.

NOTE

If you prefer to shortcut this step, you can use one of the pre-trained model files in the models subdirectory.

## Blank space for training model, if needed

Getting Ready to deploy

Wallaroo natively supports models in the ONNX and Tensorflow frameworks, and other frameworks via containerization. For this exercise, we assume that you have a model that can be converted to the ONNX framework. The first steps to deploying in Wallaroo, then, is to convert your model to ONNX, and to add some extra functions to your processing modules so Wallaroo can call them.


Exercise: Convert your Model to ONNX

Take the model that you created in the previous exercises, and convert it to ONNX. If you need help, see the Wallaroo Conversion Tutorials, or other conversion documentation.

At the end of this exercise, you should have your model as a standalone artifact, for example, a file called model.onnx.

NOTE

If you prefer to shortcut this exercise, you can use one of the pre-converted onnx files in the models directory.

# Blank space to load for converting model, if needed

Get ready to work with Wallaroo

Now that you have a model ready to go, you can log into Wallaroo and set up a workspace to organize your deployment artifacts. A Wallaroo workspace is place to organize the deployment artifacts for a project, and to collaborate with other team members. For more information, see the Wallaroo 101.

Logging into Wallaroo via the cluster’s integrated JupyterLab is quite straightfoward:

# Login through local Wallaroo instance 
wl = wallaroo.Client()

See the documentation if you are logging into Wallaroo some other way.

Once you are logged in, you can create a workspace and set it as your working environment. To make the first exercise easier, here is a convenience function to get or create a workspace:

# return the workspace called <name>, or create it if it does not exist.
# this function assumes your connection to wallaroo is called wl
def get_workspace(name):
    workspace = None
    for ws in wl.list_workspaces():
        if ws.name() == name:
            workspace= ws
    if(workspace == None):
        workspace = wl.create_workspace(name)
    return workspace

Then logging in and creating a workspace looks something like this:

# Login through local Wallaroo instance 
wl = wallaroo.Client()

Setting up the workspace may resemble this. Verify that the workspace name is unique across the Wallaroo instance.

# workspace names need to be globally unique, so add a random suffix to insure this
# especially important if the "main" workspace name is potentially a common one

suffix= ''.join(random.choice(string.ascii_lowercase) for i in range(4))
workspace_name = "my-workspace"+suffix

workspace = get_workspace(workspace_name)

# set your current workspace to the workspace that you just created
wl.set_current_workspace(workspace)

# optionally, examine your current workspace
wl.get_current_workspace()

Exercise: Log in and create a workspace

Log into wallaroo, and create a workspace for this tutorial. Then set that new workspace to your current workspace.
Make sure you remember the name that you gave the workspace, as you will need it for later notebooks. Set that workspace to be your working environment.

Notes

  • Workspace names must be globally unique, so don’t pick something too common. The “random suffix” trick in the code snippet is one way to try to generate a unique workspace name, if you suspect you are using a common name.

At the end of the exercise, you should be in a new workspace to do further work.

For more information, see Wallaroo SDK Essentials Guide: Workspace Management.

# Login through local Wallaroo instance

wl = wallaroo.Client()
## Blank spot to connect to the workspace

suffix= ''.join(random.choice(string.ascii_lowercase) for i in range(4))
workspace_name = "classification-finserv"

workspace = get_workspace(workspace_name)

# set your current workspace to the workspace that you just created
wl.set_current_workspace(workspace)

# optionally, examine your current workspace
wl.get_current_workspace()
{'name': 'classification-finserv-jch', 'id': 21, 'archived': False, 'created_by': '0a36fba2-ad42-441b-9a8c-bac8c68d13fa', 'created_at': '2023-08-07T16:26:26.779098+00:00', 'models': [{'name': 'ccfraud-model-keras', 'versions': 1, 'owner_id': '""', 'last_update_time': datetime.datetime(2023, 8, 7, 16, 26, 36, 806125, tzinfo=tzutc()), 'created_at': datetime.datetime(2023, 8, 7, 16, 26, 36, 806125, tzinfo=tzutc())}], 'pipelines': [{'name': 'finserv-ccfraud', 'create_time': datetime.datetime(2023, 8, 7, 16, 26, 37, 485326, tzinfo=tzutc()), 'definition': '[]'}]}

Deploy a Simple Single-Step Pipeline

Once your model is in the ONNX format, and you have a workspace to work in, you can easily upload your model to Wallaroo’s production platform with just a few lines of code. For example, if you have a model called model.onnx, and you wish to upload it to Wallaroo with the name mymodel, then upload the model as follows (once you are in the appropriate workspace):

from wallaroo.framework import Framework
my_model = wl.upload_model("mymodel", "model.onnx", framework=Framework.ONNX).configure()

See Wallaroo SDK Essentials Guide: Model Uploads and Registrations: ONNX for full details.

The function upload_model() returns a handle to the uploaded model that you will continue to work with in the SDK.

Once the model has been uploaded, you can create a pipeline that contains the model. The pipeline is the mechanism that manages deployments. A pipeline contains a series of steps - sequential sets of models which take in the data from the preceding step, process it through the model, then return a result. Some pipelines can have just one step, while others may have multiple models with multiple steps or arranged for A/B testing. Deployed pipelines allocate resources and can then process data either through local files or through a deployment URL.

So for your model to accept inferences, you must add it to a pipeline. You can create a single step pipeline called mypipeline as follows.

# create the pipeline
my_pipeline = wl.build_pipeline("mypipeline").add_model_step(my_model)

# deploy the pipeline
my_pipeline = my_pipeline.deploy()

Deploying the pipeline means that resources from the cluster are allocated to the pipeline, and it is ready to accept inferences. You can “turn off” the pipeline with the call pipeline.undeploy(), which returns the resources back to the cluster. This is an important step - leaving pipeline deployed when they’re no longer needed takes up resources that may be needed by other pipelines or services.

See Wallaroo SDK Essentials Guide: Pipeline Management for full details.

More Hints

  • workspace = wl.get_current_workspace() gives you a handle to the current workspace
  • then workspace.models() will return a list of the models in the workspace
  • and workspace.pipelines() will return a list of the pipelines in the workspace

Exercise: Upload and deploy your model

Upload and deploy the ONNX model that you created in the previous exercise. For simplicity, do any needed pre-processing in the notebook.

At the end of the exercise, you should have a model and a deployed pipeline in your workspace.

## blank space to upload model, and create the pipeline

from wallaroo.framework import Framework

model = wl.upload_model('ccfraud-model-keras', '../models/keras_ccfraud.onnx', framework=Framework.ONNX)

pipeline = wl.build_pipeline("finserv-ccfraud").add_model_step(model)

pipeline.deploy()
namefinserv-ccfraud
created2023-08-07 16:26:37.485326+00:00
last_updated2023-08-07 16:28:48.278133+00:00
deployedTrue
tags
versions230d585a-52db-476d-ab28-7b4baed9d023, 192f92e9-9a97-4339-8c1d-f89541ff2cef, 5d2d9c84-13c2-4e35-a41f-ec3c4e8d297b, 41927bef-d8fb-49ee-914e-d106ffc304b3
stepsccfraud-model-keras

Sending Data to your Pipeline

ONNX models generally expect their input as an array in a dictionary, keyed by input name. In Wallaroo, the default input name is “tensor”. So (outside of Wallaroo), an ONNX model that expected three numeric values as its input would expect input data similar to the below: (Note: The below examples are only notional, they aren’t intended to work with our example models.)

# one datum
singleton = {'tensor': [[1, 2, 3]] }

# two datums
two_inputs = {'tensor': [[1, 2, 3], [4, 5, 6]] }

In the Wallaroo SDK, you can send a pandas DataFrame representation of this dictionary (pandas record format) to the pipeline, via the pipeline.infer() method.

import pandas as pd

# one datum (notional example)
sdf = pd.DataFrame(singleton)
sdf
#       tensor
# 0  [1, 2, 3]

# send the datum to a pipeline for inference
# notional example - not houseprice model!
result = my_pipeline.infer(sdf)

# two datums
# Note that the value of 'tensor' must be a list, not a numpy array 
twodf = pd.DataFrame(two_inputs)
twodf
#      tensor
# 0  [1, 2, 3]
# 1  [4, 5, 6]

# send the data to a pipeline for inference
# notional example, not houseprice model!
result = my_pipeline.infer(twodf)

To send data to a pipeline via the inference URL (for example, via CURL), you need the JSON representation of these data frames.

#
# notional examples, not houseprice model!
#
sdf.to_json(orient='records')
# '[{"tensor":[1,2,3]}]'

twodf.to_json(orient='records')
# '[{"tensor":[1,2,3]},{"tensor":[4,5,6]}]'

If the JSON data is in a file, you can send it to the pipeline from within the SDK via the pipeline.infer_from_file() method.

In either case, a successful inference will return a data frame of inference results. The model inference(s) will be in the column out.<outputname>.

For more details, see Wallaroo SDK Essentials Guide: Inference Management.

Converting from files

If your input data is in a pandas record format (like the cc_data_10k.df.json example data in the data directory), then you need to import it to pandas record format to send the data to your pipeline. See the pandas DataFrame documentation for methods on how to import CSV or JSON files into a pandas DataFrame.

To help with the following exercises, here are some convenience functions you might find useful for doing this conversion. These functions convert input data in standard tabular format (in a pandas DataFrame) to the pandas record format that the model expects.

# pull a single datum from a data frame 
# and convert it to the format the model expects
def get_singleton(df, i):
    singleton = df.iloc[i,:].to_numpy().tolist()
    sdict = {'tensor': [singleton]}
    return pd.DataFrame.from_dict(sdict)

# pull a batch of data from a data frame
# and convert to the format the model expects
def get_batch(df, first=0, nrows=1):
    last = first + nrows
    batch = df.iloc[first:last, :].to_numpy().tolist()
    return pd.DataFrame.from_dict({'tensor': batch})

Execute the following code block to see examples of what get_singleton and get_batch do.

# RUN ME!

print('''TOY data for a model that takes inputs var1, var2, var3.
The dataframe is called df.
Pretend the model is in a Wallaroo pipeline called "toypipeline"''')

df = pd.DataFrame({
    'var1': [1, 3, 5],
    'var2': [33, 88, 45],
    'var3': [6, 20, 5]
})

display(df)

# create a model input from the first row
# this is now in the format that a model would accept
singleton = get_singleton(df, 0)

print('''The command "singleton = get_singleton(df, 0)" converts
the first row of the data frame into the format that Wallaroo pipelines accept.
You could now get a prediction by: "toypipeline.infer(singleton)".
''')
display(singleton)

# create a batch of queries from the entire dataframe
batch = get_batch(df, nrows=2)

print('''The command "batch = get_batch(df, nrows=2)" converts
the the first two rows of the data frame into a batch format that Wallaroo pipelines accept.
You could now get a batch prediction by: "toypipeline.infer(batch)".
''')
display(batch)
TOY data for a model that takes inputs var1, var2, var3.
The dataframe is called df.
Pretend the model is in a Wallaroo pipeline called "toypipeline"
var1var2var3
01336
138820
25455
The command "singleton = get_singleton(df, 0)" converts
the first row of the data frame into the format that Wallaroo pipelines accept.
You could now get a prediction by: "toypipeline.infer(singleton)".
tensor
0[1, 33, 6]
The command "batch = get_batch(df, nrows=2)" converts
the the first two rows of the data frame into a batch format that Wallaroo pipelines accept.
You could now get a batch prediction by: "toypipeline.infer(batch)".
tensor
0[1, 33, 6]
1[3, 88, 20]

Exercise: Send data to your pipeline for inference.

Create some test data from the housing data and send it to the pipeline that you deployed in the previous exercise.

  • If you used the pre-provided models, then you can use cc_data_10k.df.json from the data directory. This can be loaded directly into your sample pandas DataFrame - check the pandas documentation for a handy function for doing that. (We mention yours because sometimes people try to use the example code above rather than their own data.)

  • Start easy, with just one datum; retrieve the inference results. You can try small batches, as well. Use the above example as a guide.

  • Examine the inference results; observe what the model prediction column is called; it should be of the form out.<outputname>.

For more hints about the different ways of sending data to the pipeline, and to see an example of the inference result format, see the Run Inference through Local Variable.

At the end of the exercise, you should have a set of inference results that you got through the Wallaroo pipeline.

##  blank space to create test data, and send some data to your model

df = pd.read_json('../data/cc_data_10k.df.json')
df.head(5)

singleton = get_singleton(df, 0)
display(singleton)

single_result = pipeline.infer(singleton)
display(single_result)

multiple_batch = get_batch(df, nrows=5)
display(multiple_batch)
multiple_result = pipeline.infer(multiple_batch)
display(multiple_result)
tensor
0[[-1.0603297501, 2.3544967095000002, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526000001, 1.9870535692, 0.7005485718000001, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756000001, -0.1466244739, -1.4463212439]]
timein.tensorout.dense_1check_failures
02023-08-07 17:11:20.715[[-1.0603297501, 2.3544967095, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526, 1.9870535692, 0.7005485718, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756, -0.1466244739, -1.4463212439]][0.99300325]0
tensor
0[[-1.0603297501, 2.3544967095000002, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526000001, 1.9870535692, 0.7005485718000001, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756000001, -0.1466244739, -1.4463212439]]
1[[-1.0603297501, 2.3544967095000002, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526000001, 1.9870535692, 0.7005485718000001, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756000001, -0.1466244739, -1.4463212439]]
2[[-1.0603297501, 2.3544967095000002, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526000001, 1.9870535692, 0.7005485718000001, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756000001, -0.1466244739, -1.4463212439]]
3[[-1.0603297501, 2.3544967095000002, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526000001, 1.9870535692, 0.7005485718000001, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756000001, -0.1466244739, -1.4463212439]]
4[[0.5817662108, 0.09788155100000001, 0.1546819424, 0.4754101949, -0.19788623060000002, -0.45043448540000003, 0.016654044700000002, -0.0256070551, 0.0920561602, -0.2783917153, 0.059329944100000004, -0.0196585416, -0.4225083157, -0.12175388770000001, 1.5473094894, 0.2391622864, 0.3553974881, -0.7685165301, -0.7000849355000001, -0.1190043285, -0.3450517133, -1.1065114108, 0.2523411195, 0.0209441826, 0.2199267436, 0.2540689265, -0.0450225094, 0.10867738980000001, 0.2547179311]]
timein.tensorout.dense_1check_failures
02023-08-07 17:11:21.172[[-1.0603297501, 2.3544967095, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526, 1.9870535692, 0.7005485718, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756, -0.1466244739, -1.4463212439]][0.99300325]0
12023-08-07 17:11:21.172[[-1.0603297501, 2.3544967095, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526, 1.9870535692, 0.7005485718, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756, -0.1466244739, -1.4463212439]][0.99300325]0
22023-08-07 17:11:21.172[[-1.0603297501, 2.3544967095, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526, 1.9870535692, 0.7005485718, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756, -0.1466244739, -1.4463212439]][0.99300325]0
32023-08-07 17:11:21.172[[-1.0603297501, 2.3544967095, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526, 1.9870535692, 0.7005485718, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756, -0.1466244739, -1.4463212439]][0.99300325]0
42023-08-07 17:11:21.172[[0.5817662108, 0.097881551, 0.1546819424, 0.4754101949, -0.1978862306, -0.4504344854, 0.0166540447, -0.0256070551, 0.0920561602, -0.2783917153, 0.0593299441, -0.0196585416, -0.4225083157, -0.1217538877, 1.5473094894, 0.2391622864, 0.3553974881, -0.7685165301, -0.7000849355, -0.1190043285, -0.3450517133, -1.1065114108, 0.2523411195, 0.0209441826, 0.2199267436, 0.2540689265, -0.0450225094, 0.1086773898, 0.2547179311]][0.0010916889]0

Undeploying Your Pipeline

You should always undeploy your pipelines when you are done with them, or don’t need them for a while. This releases the resources that the pipeline is using for other processes to use. You can always redeploy the pipeline when you need it again. As a reminder, here are the commands to deploy and undeploy a pipeline:


# when the pipeline is deployed, it's ready to receive data and infer
pipeline.deploy()

# "turn off" the pipeline and release its resources
pipeline.undeploy()

For more information on undeploying a pipeline, see Undeploy a Pipeline.

If you are continuing on to the next notebook now, you can leave the pipeline deployed to keep working; but if you are taking a break, then you should undeploy.

## blank space to undeploy the pipeline, if needed

pipeline.undeploy()
namefinserv-ccfraud
created2023-08-07 16:26:37.485326+00:00
last_updated2023-08-07 16:28:48.278133+00:00
deployedFalse
tags
versions230d585a-52db-476d-ab28-7b4baed9d023, 192f92e9-9a97-4339-8c1d-f89541ff2cef, 5d2d9c84-13c2-4e35-a41f-ec3c4e8d297b, 41927bef-d8fb-49ee-914e-d106ffc304b3
stepsccfraud-model-keras

Congratulations!

You have now

  • Successfully trained a model
  • Converted your model and uploaded it to Wallaroo
  • Created and deployed a simple single-step pipeline
  • Successfully send data to your pipeline for inference

In the next notebook, you will look at two different ways to evaluate your model against the real world environment.

1.1.2 - Financial Services: Model Experiments

How to use Wallaroo to experiment with different financial services classification models.

Tutorial Notebook 2: Vetting a Model With Production Experiments

So far, we’ve discussed practices and methods for transitioning an ML model and related artifacts from development to production. However, just the act of pushing a model into production is not the only consideration. In many situations, it’s important to vet a model’s performance in the real world before fully activating it. Real world vetting can surface issues that may not have arisen during the development stage, when models are only checked using hold-out data.

In this notebook, you will learn about two kinds of production ML model validation methods: A/B testing and Shadow Deployments. A/B tests and other types of experimentation are part of the ML lifecycle. The ability to quickly experiment and test new models in the real world helps data scientists to continually learn, innovate, and improve AI-driven decision processes.

Preliminaries

In the blocks below we will preload some required libraries; we will also redefine some of the convenience functions that you saw in the previous notebook.

After that, you should log into Wallaroo and set your working environment to the workspace that you created in the previous notebook.

# preload needed libraries 

import wallaroo
from wallaroo.object import EntityNotFoundError
from wallaroo.framework import Framework

from IPython.display import display

# used to display DataFrame information without truncating
from IPython.display import display
import pandas as pd
pd.set_option('display.max_colwidth', None)

import json
import datetime
import time

# used for unique connection names

import string
import random
## convenience functions from the previous notebook

# return the workspace called <name>, or create it if it does not exist.
# this function assumes your connection to wallaroo is called wl
def get_workspace(name):
    workspace = None
    for ws in wl.list_workspaces():
        if ws.name() == name:
            workspace= ws
    if(workspace == None):
        workspace = wl.create_workspace(name)
    return workspace

# pull a single datum from a data frame 
# and convert it to the format the model expects
def get_singleton(df, i):
    singleton = df.iloc[i,:].to_numpy().tolist()
    sdict = {'tensor': [singleton]}
    return pd.DataFrame.from_dict(sdict)

# pull a batch of data from a data frame
# and convert to the format the model expects
def get_batch(df, first=0, nrows=1):
    last = first + nrows
    batch = df.iloc[first:last, :].to_numpy().tolist()
    return pd.DataFrame.from_dict({'tensor': batch})

Pre-exercise

If needed, log into Wallaroo and go to the workspace that you created in the previous notebook. Please refer to Notebook 1 to refresh yourself on how to log in and set your working environment to the appropriate workspace.

## blank space to log in and go to the appropriate workspace

wl = wallaroo.Client()

workspace_name = "classification-finserv-jch"

workspace = get_workspace(workspace_name)

# set your current workspace to the workspace that you just created
wl.set_current_workspace(workspace)

# optionally, examine your current workspace
wl.get_current_workspace()
{'name': 'classification-finserv-jch', 'id': 21, 'archived': False, 'created_by': '0a36fba2-ad42-441b-9a8c-bac8c68d13fa', 'created_at': '2023-08-07T16:26:26.779098+00:00', 'models': [{'name': 'ccfraud-model-keras', 'versions': 2, 'owner_id': '""', 'last_update_time': datetime.datetime(2023, 8, 7, 16, 28, 46, 566311, tzinfo=tzutc()), 'created_at': datetime.datetime(2023, 8, 7, 16, 26, 36, 806125, tzinfo=tzutc())}], 'pipelines': [{'name': 'finserv-ccfraud', 'create_time': datetime.datetime(2023, 8, 7, 16, 26, 37, 485326, tzinfo=tzutc()), 'definition': '[]'}]}

A/B Testing

An A/B test, also called a controlled experiment or a randomized control trial, is a statistical method of determining which of a set of variants is the best. A/B tests allow organizations and policy-makers to make smarter, data-driven decisions that are less dependent on guesswork.

In the simplest version of an A/B test, subjects are randomly assigned to either the control group (group A) or the treatment group (group B). Subjects in the treatment group receive the treatment (such as a new medicine, a special offer, or a new web page design) while the control group proceeds as normal without the treatment. Data is then collected on the outcomes and used to study the effects of the treatment.

In data science, A/B tests are often used to choose between two or more candidate models in production, by measuring which model performs best in the real world. In this formulation, the control is often an existing model that is currently in production, sometimes called the champion. The treatment is a new model being considered to replace the old one. This new model is sometimes called the challenger. In our discussion, we’ll use the terms champion and challenger, rather than control and treatment.

When data is sent to a Wallaroo A/B test pipeline for inference, each datum is randomly sent to either the champion or challenger. After enough data has been sent to collect statistics on all the models in the A/B test pipeline, then those outcomes can be analyzed to determine the difference (if any) in the performance of the champion and challenger. Usually, the purpose of an A/B test is to decide whether or not to replace the champion with the challenger.

Keep in mind that in machine learning, the terms experiments and trials also often refer to the process of finding a training configuration that works best for the problem at hand (this is sometimes called hyperparameter optimization). In this guide, we will use the term experiment to refer to the use of A/B tests to compare the performance of different models in production.


Exercise: Create some challenger models and upload them to Wallaroo

Use the house price data from Notebook 1 to create at least one alternate house price prediction model. You can do this by varying the modeling algorithm, the inputs, the feature engineering, or all of the above.

For the purpose of these exercises, please make sure that the predictions from the new model(s) are in the same units as the (champion) model that you created in Chapter 3. For example, if the champion model predicts log price, then the challenger models should also predict log price. If the champion model predicts price in units of $10,000, then the challenger models should, also.

  • If you prefer to shortcut this step, you can use some of the pretrained model onnx files in the models directory
  • Upload your new model(s) to Wallaroo, into your workspace

At the end of this exercise, you should have at least one challenger model to compare to your champion model uploaded to your workspace.

# blank space to train, convert, and upload new model

challenger_model = wl.upload_model('ccfraud-model-xgboost', '../models/xgboost_ccfraud.onnx', framework=Framework.ONNX)

There are a number of considerations to designing an A/B test; you can check out the article The What, Why, and How of A/B Testing for more details. In these exercises, we will concentrate on the deployment aspects. You will need a champion model and at least one challenger model. You also need to decide on a data split: for example 50-50 between the champion and challenger, or a 2:1 ratio between champion and challenger (two-thirds of the data to the champion, one-third to the challenger).

As an example of creating an A/B test deployment, suppose you have a champion model called “champion” that you have been running in a one-step pipeline called “pipeline”. You now want to compare it to a challenger model called “challenger”. For your A/B test, you will send two-thirds of the data to the champion, and the other third to the challenger. Both models have already been uploaded.

To help you with the exercises, here some convenience functions to retrieve a models and pipelines that have been previously uploaded to your workspace (in this example, wl is your wallaroo.client() object).

# Get the most recent version of a model.
# Assumes that the most recent version is the first in the list of versions.
# wl.get_current_workspace().models() returns a list of models in the current workspace

def get_model(mname, modellist=wl.get_current_workspace().models()):
    model = [m.versions()[-1] for m in modellist if m.name() == mname]
    if len(model) <= 0:
        raise KeyError(f"model {mname} not found in this workspace")
    return model[0]

# get a pipeline by name in the workspace
def get_pipeline(pname, plist = wl.get_current_workspace().pipelines()):
    pipeline = [p for p in plist if p.name() == pname]
    if len(pipeline) <= 0:
        raise KeyError(f"pipeline {pname} not found in this workspace")
    return pipeline[0]
# use the space here for retrieving the models and pipeline

pipeline = get_pipeline('finserv-ccfraud')

ccfraud_keras_model = get_model('ccfraud-model-keras')

Pipelines may have already been issued with pipeline steps. Pipeline steps can be removed or replaced with other steps.

The easiest way to clear all pipeline steps is with the Pipeline clear() method.

To remove one step, use the Pipeline remove_step(index) method, where index is the step number ordered from zero. For example, if a pipeline has one step, then remove_step(0) would remove that step.

To replace a pipeline step, use the Pipeline replace_with_model_step(index, model), where index is the step number ordered from zero, and the model is the model to be replacing it with.

Updated pipeline steps are not saved until the pipeline is redeployed with the Pipeline deploy() method.

Reference: Wallaroo SDK Essentials Guide: Pipeline Management
.

For A/B testing, pipeline steps are added or replace an existing step.

To add a A/B testing step use the Pipeline add_random_split method with the following parameters:

ParameterTypeDescription
champion_weightFloat (Required)The weight for the champion model.
champion_modelWallaroo.Model (Required)The uploaded champion model.
challenger_weightFloat (Required)The weight of the challenger model.
challenger_modelWallaroo.Model (Required)The uploaded challenger model.
hash_keyString(Optional)A key used instead of a random number for model selection. This must be between 0.0 and 1.0.

Note that multiple challenger models with different weights can be added as the random split step.

In this example, a pipeline will be built with a 2:1 weighted ratio between the champion and a single challenger model.

pipeline.add_random_split([(2, control), (1, challenger)]))

To replace an existing pipeline step with an A/B testing step use the Pipeline replace_with_random_split method.

ParameterTypeDescription
indexInteger (Required)The pipeline step being replaced.
champion_weightFloat (Required)The weight for the champion model.
champion_modelWallaroo.Model (Required)The uploaded champion model.
challenger_weightFloat (Required)The weight of the challenger model.
challenger_modelWallaroo.Model (Required)The uploaded challenger model.
hash_keyString(Optional)A key used instead of a random number for model selection. This must be between 0.0 and 1.0.

This example replaces the first pipeline step with a 2:1 champion to challenger radio.

pipeline.replace_with_random_split(0,[(2, control), (1, challenger)]))

In either case, the random split will randomly send inference data to one model based on the weighted ratio. As more inferences are performed, the ratio between the champion and challengers will align more and more to the ratio specified.

Reference: Wallaroo SDK Essentials Guide: Pipeline Management A/B Testing
.

Then creating an A/B test deployment would look something like this:

First get the models used.

# retrieve handles to the most recent versions 
# of the champion and challenger models
champion = get_model("champion")
challenger = get_model("challenger")
# blank space to get the model(s)

ccfraud_xgboost_model = get_model('ccfraud-model-xgboost')

Second step is to retrieve the pipeline created in the previous Notebook, then redeploy it with the A/B testing split step.

Here’s some sample code:

# get an existing single-step pipeline and undeploy it
pipeline = get_pipeline("pipeline")
pipeline.undeploy()

# clear the pipeline and add a random split
pipeline.clear()
pipeline.add_random_split([(2, champion), (1, challenger)])
pipeline.deploy()

The above code clears out all the steps of the pipeline and adds a new step with a A/B test deployment, where the incoming data is randomly sent in a 2:1 ratio to the champion and the challenger, respectively.

You can add multiple challengers to an A/B test::

pipeline.add_random_split([ (2, champion), (1, challenger01), (1, challenger02) ])

This pipeline will distribute data in the ratio 2:1:1 (or half to the champion, a quarter each to the challlengers) to the champion and challenger models, respectively.

You can also create an A/B test deployment from scratch:

pipeline = wl.build_pipeline("pipeline")
pipeline.add_random_split([(2, champion), (1, challenger)])

Exercise: Create an A/B test deployment of your house price models

Use the champion and challenger models that you created in the previous exercises to create an A/B test deployment. You can either create one from scratch, or reconfigure an existing pipeline.

  • Send half the data to the champion, and distribute the rest among the challenger(s).

At the end of this exercise, you should have an A/B test deployment and be ready to compare multiple models.

# blank space to retrieve pipeline and redeploy with a/b testing step

pipeline.clear()
pipeline.add_random_split([(2, ccfraud_keras_model), (1, ccfraud_xgboost_model)])
pipeline.deploy()
namefinserv-ccfraud
created2023-08-07 16:26:37.485326+00:00
last_updated2023-08-07 17:22:20.920743+00:00
deployedTrue
tags
versions36faf126-dac5-419d-b0b1-7d7b698b587e, 230d585a-52db-476d-ab28-7b4baed9d023, 192f92e9-9a97-4339-8c1d-f89541ff2cef, 5d2d9c84-13c2-4e35-a41f-ec3c4e8d297b, 41927bef-d8fb-49ee-914e-d106ffc304b3
stepsccfraud-model-keras

The pipeline steps are displayed with the Pipeline steps() method. This is used to verify the current deployed steps in the pipeline.

  • IMPORTANT NOTE: Verify that the pipeline is deployed before checking for pipeline steps. Deploying the pipeline sets the steps into the Wallaroo system - until that happens, the steps only exist in the local system as potential steps.
# blank space to get the current pipeline steps

pipeline.steps()
[{'RandomSplit': {'hash_key': None, 'weights': [{'model': {'name': 'ccfraud-model-keras', 'version': '53dbee25-2e64-4acf-8f5b-44feab1de488', 'sha': 'bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507'}, 'weight': 2}, {'model': {'name': 'ccfraud-model-xgboost', 'version': 'e71c9c1d-9777-4967-b32c-a074ea8aa467', 'sha': '054810e3e3ebbdd34438d9c1a08ed6a6680ef10bf97b9223f78ebf38e14b3b52'}, 'weight': 1}]}}]

Please note that for batch inferences, the entire batch will be sent to the same model. So in order to verify that your pipeline is distributing inferences in the proportion you specified, you will need to send your queries one datum at a time.

To help with the next exercise, here is another convenience function you might find useful.

# get the names of the inferring models
# from a dataframe of a/b test results
def get_names(resultframe):
    modelcol = resultframe['out._model_split']
    jsonstrs = [mod[0]  for mod in modelcol]
    return [json.loads(jstr)['name'] for jstr in jsonstrs]

Here’s an example of how to send a large number of queries one at a time to your pipeline in the SDK

results = []

# get a list of result frames
for i in range(1000):
    query = get_singleton(testdata, i)
    results.append(pipeline.infer(query))

# make one data frame of all results    
allresults = pd.concat(results, ignore_index=True)

# add a column to indicate which model made the inference
allresults['modelname'] = get_names(allresults)

# get the counts of how many inferences were made by each model
allresults.modelname.value_counts()
  • NOTE: Performing 1,000 inferences sequentially may take several minutes to complete. Adjust the range for time as required.

As with the single-step pipeline, the model predictions will be in a column named out.<outputname>. In addition, there will be a column named out._model_split that contains information about the model that made a particular prediction. The get_names() convenience function above extracts the model name from the out._model_split column.


Exercise: Send some queries to your A/B test deployment

  1. Send a single datum to the A/B test pipeline you created in the previous exercise. You can use the same test data set that you created/downloaded in the previous notebook. Observe what the inference result looks like. If you send the singleton through the pipeline multiple times, you should observe that the model making the inference changes.
  2. Send a large number of queries (at least 100) one at a time to the pipeline.
  • Note that approximately half the inferences were made by the champion model.
  • The remaining inferences should be distributed as you specified.

The more queries you send, the closer the distribution should be to what you specified.

If you can align the actual house prices from your test data to the predictions, you can also compare the accuracy of the different models.

Don’t forget to undeploy your pipeline after you are done, to free up resources.

## blank space to test one inference

##  blank space to create test data, and send some data to your model

df = pd.read_json('../data/cc_data_10k.df.json')

singleton = get_singleton(df, 0)
display(singleton)

single_result = pipeline.infer(singleton)
display(single_result)

display(get_names(single_result))
tensor
0[[-1.0603297501, 2.3544967095000002, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526000001, 1.9870535692, 0.7005485718000001, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756000001, -0.1466244739, -1.4463212439]]
timein.tensorout._model_splitout.dense_1check_failures
02023-08-07 17:24:57.870[[-1.0603297501, 2.3544967095, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526, 1.9870535692, 0.7005485718, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756, -0.1466244739, -1.4463212439]][{"name":"ccfraud-model-keras","version":"53dbee25-2e64-4acf-8f5b-44feab1de488","sha":"bc85ce596945f876256f41515c7501c399fd97ebcb9ab3dd41bf03f8937b4507"}][0.99300325]0
['ccfraud-model-keras']
# blank space to send queries to A/B test pipeline and examine the results

results = []

# get a list of result frames
for i in range(20):
    query = get_singleton(df, i)
    results.append(pipeline.infer(query))

# make one data frame of all results    
allresults = pd.concat(results, ignore_index=True)

# add a column to indicate which model made the inference
allresults['modelname'] = get_names(allresults)

# get the counts of how many inferences were made by each model
allresults.modelname.value_counts()
ccfraud-model-keras      12
ccfraud-model-xgboost     8
Name: modelname, dtype: int64

Shadow Deployments

Another way to vet your new model is to set it up in a shadow deployment. With shadow deployments, all the models in the experiment pipeline get all the data, and all inferences are recorded. However, the pipeline returns only one “official” prediction: the one from default, or champion model.

Shadow deployments are useful for “sanity checking” a model before it goes truly live. For example, you might have built a smaller, leaner version of an existing model using knowledge distillation or other model optimization techniques, as discussed here. A shadow deployment of the new model alongside the original model can help ensure that the new model meets desired accuracy and performance requirements before it’s put into production.

As an example of creating a shadow deployment, suppose you have a champion model called “champion”, that you have been running in a one-step pipeline called “pipeline”. You now want to put a challenger model called “challenger” into a shadow deployment with the champion. Both models have already been uploaded.

Shadow deployments can be added as a pipeline step, or replace an existing pipeline step.

Shadow deployment steps are added with the add_shadow_deploy(champion, [model2, model3,...]) method, where the champion is the model that the inference results will be returned. The array of models listed after are the models where inference data is also submitted with their results displayed as as shadow inference results.

Shadow deployment steps replace an existing pipeline step with the replace_with_shadow_deploy(index, champion, [model2, model3,...]) method. The index is the step being replaced with pipeline steps starting at 0, and the champion is the model that the inference results will be returned. The array of models listed after are the models where inference data is also submitted with their results displayed as as shadow inference results.

Then creating a shadow deployment from a previously created (and deployed) pipeline could look something like this:

# retrieve handles to the most recent versions 
# of the champion and challenger models
# see the A/B test section for the definition of get_model()
champion = get_model("champion")
challenger = get_model("challenger")

# get the existing pipeline and undeploy it
# see the A/B test section for the definition of get_pipeline()
pipeline = get_pipeline("pipeline")
pipeline.undeploy()

# clear the pipeline and add a shadow deploy step
pipeline.clear()
pipeline.add_shadow_deploy(champion, [challenger])
pipeline.deploy()

The above code clears the pipeline and adds a shadow deployment. The pipeline will still only return the inferences from the champion model, but it will also run the challenger model in parallel and log the inferences, so that you can compare what all the models do on the same inputs.

You can add multiple challengers to a shadow deploy:

pipeline.add_shadow_deploy(champion, [challenger01, challenger02])

You can also create a shadow deployment from scratch with a new pipeline. This example just uses two models - one champion, one challenger.

newpipeline = wl.build_pipeline("pipeline")
newpipeline.add_shadow_deploy(champion, [challenger])

Exercise: Create a house price model shadow deployment

Use the champion and challenger models that you created in the previous exercises to create a shadow deployment. You can either create one from scratch, or reconfigure an existing pipeline.

At the end of this exercise, you should have a shadow deployment running multiple models in parallel.

For more information, see Pipeline Shadow Deployments

# blank space to create a shadow deployment

pipeline.clear()

pipeline.add_shadow_deploy(ccfraud_keras_model, [ccfraud_xgboost_model])

pipeline.deploy()
namefinserv-ccfraud
created2023-08-07 16:26:37.485326+00:00
last_updated2023-08-07 17:29:08.595377+00:00
deployedTrue
tags
versions8ff48a62-fd9d-43f5-ba3f-11a7f1bcc474, 36faf126-dac5-419d-b0b1-7d7b698b587e, 230d585a-52db-476d-ab28-7b4baed9d023, 192f92e9-9a97-4339-8c1d-f89541ff2cef, 5d2d9c84-13c2-4e35-a41f-ec3c4e8d297b, 41927bef-d8fb-49ee-914e-d106ffc304b3
stepsccfraud-model-keras

Since a shadow deployment returns multiple predictions for a single datum, its inference result will look a little different from those of an A/B test or a single-step pipelne. The next exercise will show you how to examine all the inferences from all the models.


Exercise: Examine shadow deployment inferences

Use the test data that you created in a previous exercise to send a single datum to the shadow deployment that you created in the previous exercise.

  • Observe the inference result
  • You should see a column called out.<outputname>; this is the prediction from the champion model. It is the “official” prediction from the pipeline. If you used the same champion model in the A/B test exercise above, and in the single-step pipeline from the previous notebook, you should see the inference results from all those pipelines was also called out.<outputname>.
  • You should also see a column called out_<challengermodel>.<outputname> (or more than one, if you had multiple challengers). These are the predictions from the challenger models.

For example, if your champion model is called “champion”, your challenger model is called “challenger”, and the outputname is “output”,
then you should see the “official” prediction out.output and the shadow prediction out_challenger.output.

Save the datum and the inference result from this exercise. You will need it for the next exercise.

# blank space to send an inference and examine the result

multiple_batch = get_batch(df, nrows=5)
multiple_result = pipeline.infer(multiple_batch)
display(multiple_result)
timein.tensorout.dense_1check_failuresout_ccfraud-model-xgboost.variable
02023-08-07 17:29:16.803[[-1.0603297501, 2.3544967095, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526, 1.9870535692, 0.7005485718, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756, -0.1466244739, -1.4463212439]][0.99300325]0[1.0094898]
12023-08-07 17:29:16.803[[-1.0603297501, 2.3544967095, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526, 1.9870535692, 0.7005485718, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756, -0.1466244739, -1.4463212439]][0.99300325]0[1.0094898]
22023-08-07 17:29:16.803[[-1.0603297501, 2.3544967095, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526, 1.9870535692, 0.7005485718, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756, -0.1466244739, -1.4463212439]][0.99300325]0[1.0094898]
32023-08-07 17:29:16.803[[-1.0603297501, 2.3544967095, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526, 1.9870535692, 0.7005485718, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756, -0.1466244739, -1.4463212439]][0.99300325]0[1.0094898]
42023-08-07 17:29:16.803[[0.5817662108, 0.097881551, 0.1546819424, 0.4754101949, -0.1978862306, -0.4504344854, 0.0166540447, -0.0256070551, 0.0920561602, -0.2783917153, 0.0593299441, -0.0196585416, -0.4225083157, -0.1217538877, 1.5473094894, 0.2391622864, 0.3553974881, -0.7685165301, -0.7000849355, -0.1190043285, -0.3450517133, -1.1065114108, 0.2523411195, 0.0209441826, 0.2199267436, 0.2540689265, -0.0450225094, 0.1086773898, 0.2547179311]][0.0010916889]0[-1.9073486e-06]

After the Experiment: Swapping in New Models

You have seen two methods to validate models in production with test (challenger) models.
The end result of an experiment is a decision about which model becomes the new champion. Let’s say that you have been running the shadow deployment that you created in the previous exercise, and you have decided that you want to replace the model “champion” with the model “challenger”. To do this, you will clear all the steps out of the pipeline, and add only “challenger” back in.

# retrieve a handle to the challenger model
# see the A/B test section for the definition of get_model()
challenger = get_model("challenger")

# get the existing pipeline and undeploy it
# see the A/B test section for the definition of get_pipeline()
pipeline = get_pipeline("pipeline")
pipeline.undeploy()

# clear out all the steps and add the champion back in 
pipeline.clear() 
pipeline.add_model_step(challenger).deploy()

Exercise: Set a challenger model as the new active model

Pick one of your challenger models as the new champion, and reconfigure your shadow deployment back into a single-step pipeline with the new chosen model.

  • Run the test datum from the previous exercise through the reconfigured pipeline.
  • Compare the results to the results from the previous exercise.
  • Notice that the pipeline predictions are different from the old champion, and consistent with the new one.

At the end of this exercise, you should have a single step pipeline, running a new model.

More information is available through Replace a Pipeline Step.

# Blank space - remove all steps, then redeploy with new champion model

pipeline.clear()

pipeline.add_model_step(ccfraud_xgboost_model)

pipeline.deploy()

singleton = get_singleton(df, 0)
display(singleton)

display(pipeline.steps())

single_result = pipeline.infer(singleton)
display(single_result)
tensor
0[[-1.0603297501, 2.3544967095000002, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526000001, 1.9870535692, 0.7005485718000001, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756000001, -0.1466244739, -1.4463212439]]
[{'ModelInference': {'models': [{'name': 'ccfraud-model-xgboost', 'version': 'e71c9c1d-9777-4967-b32c-a074ea8aa467', 'sha': '054810e3e3ebbdd34438d9c1a08ed6a6680ef10bf97b9223f78ebf38e14b3b52'}]}}]
timein.tensorout.dense_1check_failures
02023-08-07 17:30:17.358[[-1.0603297501, 2.3544967095, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526, 1.9870535692, 0.7005485718, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756, -0.1466244739, -1.4463212439]][0.99300325]0

Congratulations!

You have now

  • successfully trained new challenger models for the house price prediction problem
  • compared your models using an A/B test
  • compared your models using a shadow deployment
  • replaced your old model for a new one in the house price prediction pipeline

In the next notebook, you will learn how to monitor your production pipeline for “anomalous” or out-of-range behavior.

Cleaning up.

At this point, if you are not continuing on to the next notebook, undeploy your pipeline(s) to give the resources back to the environment.

## blank space to undeploy the pipelines

pipeline.undeploy()
namefinserv-ccfraud
created2023-08-07 16:26:37.485326+00:00
last_updated2023-08-07 17:30:15.093178+00:00
deployedFalse
tags
versions2513faec-2385-4733-a42d-56a8ae38761a, 10e495af-9b23-4973-913e-4cdebca73461, 782b9edc-d42a-4984-90ca-3bdf35922b87, e80d21a4-21c2-46b8-85d4-6652bbac9506, 8ff48a62-fd9d-43f5-ba3f-11a7f1bcc474, 36faf126-dac5-419d-b0b1-7d7b698b587e, 230d585a-52db-476d-ab28-7b4baed9d023, 192f92e9-9a97-4339-8c1d-f89541ff2cef, 5d2d9c84-13c2-4e35-a41f-ec3c4e8d297b, 41927bef-d8fb-49ee-914e-d106ffc304b3
stepsccfraud-model-keras

1.1.3 - Financial Services: Validation Rules

How to use set validation rules for model inference inputs and outputs with Wallaroo

Tutorial Notebook 3: Observability Part 1 - Validation Rules

In the previous notebooks you uploaded the models and artifacts, then deployed the models to production through provisioning workspaces and pipelines. Now you’re ready to put your feet up! But to keep your models operational, your work’s not done once the model is in production. You must continue to monitor the behavior and performance of the model to insure that the model provides value to the business.

In this notebook, you will learn about adding validation rules to pipelines.

Preliminaries

In the blocks below we will preload some required libraries; we will also redefine some of the convenience functions that you saw in the previous notebooks.

After that, you should log into Wallaroo and set your working environment to the workspace that you created for this tutorial. Please refer to Notebook 1 to refresh yourself on how to log in and set your working environment to the appropriate workspace.

# preload needed libraries 

import wallaroo
from wallaroo.object import EntityNotFoundError
from wallaroo.framework import Framework

from IPython.display import display

# used to display DataFrame information without truncating
from IPython.display import display
import pandas as pd
pd.set_option('display.max_colwidth', None)

import json
import datetime
import time

# used for unique connection names

import string
import random
## convenience functions from the previous notebooks
## these functions assume your connection to wallaroo is called wl

# return the workspace called <name>, or create it if it does not exist.
# this function assumes your connection to wallaroo is called wl
def get_workspace(name):
    workspace = None
    for ws in wl.list_workspaces():
        if ws.name() == name:
            workspace= ws
    if(workspace == None):
        workspace = wl.create_workspace(name)
    return workspace

# pull a single datum from a data frame 
# and convert it to the format the model expects
def get_singleton(df, i):
    singleton = df.iloc[i,:].to_numpy().tolist()
    sdict = {'tensor': [singleton]}
    return pd.DataFrame.from_dict(sdict)

# pull a batch of data from a data frame
# and convert to the format the model expects
def get_batch(df, first=0, nrows=1):
    last = first + nrows
    batch = df.iloc[first:last, :].to_numpy().tolist()
    return pd.DataFrame.from_dict({'tensor': batch})

# Get the most recent version of a model in the workspace
# Assumes that the most recent version is the first in the list of versions.
# wl.get_current_workspace().models() returns a list of models in the current workspace

def get_model(mname):
    modellist = wl.get_current_workspace().models()
    model = [m.versions()[-1] for m in modellist if m.name() == mname]
    if len(model) <= 0:
        raise KeyError(f"model {mname} not found in this workspace")
    return model[0]

# get a pipeline by name in the workspace
def get_pipeline(pname):
    plist = wl.get_current_workspace().pipelines()
    pipeline = [p for p in plist if p.name() == pname]
    if len(pipeline) <= 0:
        raise KeyError(f"pipeline {pname} not found in this workspace")
    return pipeline[0]
## blank space to log in and go to correct workspace

wl = wallaroo.Client()

workspace_name = "classification-finserv-jch"

workspace = get_workspace(workspace_name)

# set your current workspace to the workspace that you just created
wl.set_current_workspace(workspace)

# optionally, examine your current workspace
wl.get_current_workspace()
{'name': 'classification-finserv-jch', 'id': 21, 'archived': False, 'created_by': '0a36fba2-ad42-441b-9a8c-bac8c68d13fa', 'created_at': '2023-08-07T16:26:26.779098+00:00', 'models': [{'name': 'ccfraud-model-keras', 'versions': 2, 'owner_id': '""', 'last_update_time': datetime.datetime(2023, 8, 7, 16, 28, 46, 566311, tzinfo=tzutc()), 'created_at': datetime.datetime(2023, 8, 7, 16, 26, 36, 806125, tzinfo=tzutc())}, {'name': 'ccfraud-model-xgboost', 'versions': 1, 'owner_id': '""', 'last_update_time': datetime.datetime(2023, 8, 7, 17, 20, 51, 978426, tzinfo=tzutc()), 'created_at': datetime.datetime(2023, 8, 7, 17, 20, 51, 978426, tzinfo=tzutc())}], 'pipelines': [{'name': 'finserv-ccfraud', 'create_time': datetime.datetime(2023, 8, 7, 16, 26, 37, 485326, tzinfo=tzutc()), 'definition': '[]'}]}

Model Validation Rules

A simple way to try to keep your model’s behavior up to snuff is to make sure that it receives inputs that it expects, and that its output is something that downstream systems can handle. This can entail specifying rules that document what you expect, and either enforcing these rules (by refusing to make a prediction), or at least logging an alert that the expectations described by your validation rules have been violated. As the developer of the model, the data scientist (along with relevant subject matter experts) will often be the person in the best position to specify appropriate validation rules.

In our financial fraud example, supposed you know that confidences from a set of sample data shouldn’t go below 75% likelihood of fraud. Then you might want to set validation rules on your model pipeline to specify that you expect the model’s predictions to also be in that range. That way if the model predicts a value under that range, the pipeline will log that one of the validation checks has failed; this allows you to investigate that instance further.

Note that in this specific example, a model prediction outside the specified range may not necessarily be “wrong”; but out-of-range predictions are likely unusual enough that you may want to “sanity-check” the model’s behavior in these situations.

Wallaroo has functionality for specifying simple validation rules on model input and output values.

pipeline.add_validation(<rulename>, <expression>)

Here, <rulename> is the name of the rule, and <expression> is a simple logical expression that the data scientist expects to be true. This means if the expression proves false, then a check_failure flag is set in the inference results.

To add a validation step to a simple one-step pipeline, you need a handle to the pipeline (here called pipeline), and a handle to the model in the pipeline (here called model). Then you can specify an expected prediction range as follows:

  • Get the pipeline.
  • Depending on the steps, the pipeline can be cleared and the sample model added as a step.
  • Add the validation.
  • Deploy the pipeline to set the validation as part of its steps
# get the existing pipeline (in your workspace)
pipeline = get_pipeline("pipeline")

# you also need a handle to the model in this single-step pipeline.
# here are two ways to do it:
#
# (1) If you know the name of the model, you can also just use the get_model() convenience function above.
# In this example, the model has been uploaded to wallaroo with the name "mymodel"

model = get_model("mymodel") 

# (2) To get the model without knowing its name (for a single-step pipeline)
model = pipeline.model_configs()[0].model()

# specify the bounds
hi_bnd = 1500000.0 # 1.5M

#
# some examples of validation rules
#

# (1)  validation rule: prediction should be < 1.5 million
pipeline = pipeline.add_validation("less than 1.5m", model.outputs[0][0] < hi_bnd)

# deploy the pipeline to set the steps
pipeline.deploy()

When data is passed to the pipeline for inferences, the pipeline will log a check failure whenever one of the validation expressions evaluates to false. Here are some examples of inference results from a pipeline with the validation rule model.outputs[0][0] < 400000.0.

You can also find check failures in the logs:

logs = pipeline.logs()
logs.loc[logs['check_failures'] > 0]

Exercise: Add validation rules to your model pipeline

Add some simple validation rules to the model pipeline that you created in a previous exercise.

  • Add an upper bound or a lower bound to the model predictions
  • Try to create predictions that fall both in and out of the specified range
  • Look through the logs to find the check failures.

HINT 1: since the purpose of this exercise is try out validation rules, it might be a good idea to take a small data set and make predictions on that data set first, then set the validation rules based on those predictions, so that you can see the check failures trigger.

Don’t forget to undeploy your pipeline after you are done to free up resources.

See Anomaly Testing for more information.

## blank space to get your pipeline and run a small batch of data through it to see the range of predictions

pipeline = get_pipeline('finserv-ccfraud')

ccfraud_keras_model = get_model('ccfraud-model-keras')

pipeline.clear()
pipeline.add_model_step(ccfraud_keras_model)

pipeline.deploy()

df = pd.read_json('../data/cc_data_10k.df.json')

singleton = get_singleton(df, 0)
display(singleton)

single_result = pipeline.infer(singleton)
display(single_result)
tensor
0[[-1.0603297501, 2.3544967095000002, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526000001, 1.9870535692, 0.7005485718000001, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756000001, -0.1466244739, -1.4463212439]]
timein.tensorout.dense_1check_failures
02023-08-07 17:35:16.080[[-1.0603297501, 2.3544967095, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526, 1.9870535692, 0.7005485718, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756, -0.1466244739, -1.4463212439]][0.99300325]0
# blank space to set a validation rule on the pipeline and check if it triggers as expected

low_bnd = 0.75

pipeline = pipeline.add_validation("less than 75%", ccfraud_keras_model.outputs[0][0] > low_bnd)

pipeline.deploy()

multiple_batch = get_batch(df, nrows=5)
multiple_result = pipeline.infer(multiple_batch)
display(multiple_result)
timein.tensorout.dense_1check_failures
02023-08-07 17:35:27.089[[-1.0603297501, 2.3544967095, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526, 1.9870535692, 0.7005485718, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756, -0.1466244739, -1.4463212439]][0.99300325]0
12023-08-07 17:35:27.089[[-1.0603297501, 2.3544967095, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526, 1.9870535692, 0.7005485718, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756, -0.1466244739, -1.4463212439]][0.99300325]0
22023-08-07 17:35:27.089[[-1.0603297501, 2.3544967095, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526, 1.9870535692, 0.7005485718, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756, -0.1466244739, -1.4463212439]][0.99300325]0
32023-08-07 17:35:27.089[[-1.0603297501, 2.3544967095, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526, 1.9870535692, 0.7005485718, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756, -0.1466244739, -1.4463212439]][0.99300325]0
42023-08-07 17:35:27.089[[0.5817662108, 0.097881551, 0.1546819424, 0.4754101949, -0.1978862306, -0.4504344854, 0.0166540447, -0.0256070551, 0.0920561602, -0.2783917153, 0.0593299441, -0.0196585416, -0.4225083157, -0.1217538877, 1.5473094894, 0.2391622864, 0.3553974881, -0.7685165301, -0.7000849355, -0.1190043285, -0.3450517133, -1.1065114108, 0.2523411195, 0.0209441826, 0.2199267436, 0.2540689265, -0.0450225094, 0.1086773898, 0.2547179311]][0.0010916889]1

Congratulations!

In this tutorial you have

  • Set a validation rule on your financial fraud classification pipeline.
  • Detected model predictions that failed the validation rule.

In the next notebook, you will learn how to monitor the distribution of model outputs for drift away from expected behavior.

Cleaning up.

At this point, if you are not continuing on to the next notebook, undeploy your pipeline to give the resources back to the environment.

## blank space to undeploy the pipeline

pipeline.undeploy()
namefinserv-ccfraud
created2023-08-07 16:26:37.485326+00:00
last_updated2023-08-07 17:35:24.149368+00:00
deployedFalse
tags
versions095ef880-06f0-45d5-bfef-ac4e11d2b06c, f9004ecd-7a8c-4963-bebc-e40d1cde357c, 2513faec-2385-4733-a42d-56a8ae38761a, 10e495af-9b23-4973-913e-4cdebca73461, 782b9edc-d42a-4984-90ca-3bdf35922b87, e80d21a4-21c2-46b8-85d4-6652bbac9506, 8ff48a62-fd9d-43f5-ba3f-11a7f1bcc474, 36faf126-dac5-419d-b0b1-7d7b698b587e, 230d585a-52db-476d-ab28-7b4baed9d023, 192f92e9-9a97-4339-8c1d-f89541ff2cef, 5d2d9c84-13c2-4e35-a41f-ec3c4e8d297b, 41927bef-d8fb-49ee-914e-d106ffc304b3
stepsccfraud-model-keras

1.1.4 - Financial Services: Drift Detection

How to use Wallaroo to detect model drift with assays

Tutorial Notebook 4: Observability Part 2 - Drift Detection

In the previous notebook you learned how to add simple validation rules to a pipeline, to monitor whether outputs (or inputs) stray out of some expected range. In this notebook, you will monitor the distribution of the pipeline’s predictions to see if the model, or the environment that it runs it, has changed.

Preliminaries

In the blocks below we will preload some required libraries; we will also redefine some of the convenience functions that you saw in the previous notebooks.

After that, you should log into Wallaroo and set your working environment to the workspace that you created for this tutorial. Please refer to Notebook 1 to refresh yourself on how to log in and set your working environment to the appropriate workspace.

# preload needed libraries 

import wallaroo
from wallaroo.object import EntityNotFoundError

from IPython.display import display

# used to display DataFrame information without truncating
from IPython.display import display
import pandas as pd
pd.set_option('display.max_colwidth', None)

import json
import datetime
import time

# used for unique connection names

import string
import random
## convenience functions from the previous notebooks
## these functions assume your connection to wallaroo is called wl

# return the workspace called <name>, or create it if it does not exist.
# this function assumes your connection to wallaroo is called wl
def get_workspace(name):
    workspace = None
    for ws in wl.list_workspaces():
        if ws.name() == name:
            workspace= ws
    if(workspace == None):
        workspace = wl.create_workspace(name)
    return workspace

# pull a single datum from a data frame 
# and convert it to the format the model expects
def get_singleton(df, i):
    singleton = df.iloc[i,:].to_numpy().tolist()
    sdict = {'tensor': [singleton]}
    return pd.DataFrame.from_dict(sdict)

# pull a batch of data from a data frame
# and convert to the format the model expects
def get_batch(df, first=0, nrows=1):
    last = first + nrows
    batch = df.iloc[first:last, :].to_numpy().tolist()
    return pd.DataFrame.from_dict({'tensor': batch})

# Get the most recent version of a model in the workspace
# Assumes that the most recent version is the first in the list of versions.
# wl.get_current_workspace().models() returns a list of models in the current workspace

def get_model(mname):
    modellist = wl.get_current_workspace().models()
    model = [m.versions()[-1] for m in modellist if m.name() == mname]
    if len(model) <= 0:
        raise KeyError(f"model {mname} not found in this workspace")
    return model[0]

# get a pipeline by name in the workspace
def get_pipeline(pname):
    plist = wl.get_current_workspace().pipelines()
    pipeline = [p for p in plist if p.name() == pname]
    if len(pipeline) <= 0:
        raise KeyError(f"pipeline {pname} not found in this workspace")
    return pipeline[0]
## blank space to log in and go to correct workspace

wl = wallaroo.Client()

workspace_name = "classification-finserv-jch"

workspace = get_workspace(workspace_name)

# set your current workspace to the workspace that you just created
wl.set_current_workspace(workspace)

# optionally, examine your current workspace
wl.get_current_workspace()
{'name': 'classification-finserv-jch', 'id': 21, 'archived': False, 'created_by': '0a36fba2-ad42-441b-9a8c-bac8c68d13fa', 'created_at': '2023-08-07T16:26:26.779098+00:00', 'models': [{'name': 'ccfraud-model-keras', 'versions': 2, 'owner_id': '""', 'last_update_time': datetime.datetime(2023, 8, 7, 16, 28, 46, 566311, tzinfo=tzutc()), 'created_at': datetime.datetime(2023, 8, 7, 16, 26, 36, 806125, tzinfo=tzutc())}, {'name': 'ccfraud-model-xgboost', 'versions': 1, 'owner_id': '""', 'last_update_time': datetime.datetime(2023, 8, 7, 17, 20, 51, 978426, tzinfo=tzutc()), 'created_at': datetime.datetime(2023, 8, 7, 17, 20, 51, 978426, tzinfo=tzutc())}], 'pipelines': [{'name': 'finserv-ccfraud', 'create_time': datetime.datetime(2023, 8, 7, 16, 26, 37, 485326, tzinfo=tzutc()), 'definition': '[]'}]}

Monitoring for Drift: Shift Happens.

In machine learning, you use data and known answers to train a model to make predictions for new previously unseen data. You do this with the assumption that the future unseen data will be similar to the data used during training: the future will look somewhat like the past.
But the conditions that existed when a model was created, trained and tested can change over time, due to various factors.

A good model should be robust to some amount of change in the environment; however, if the environment changes too much, your models may no longer be making the correct decisions. This situation is known as concept drift; too much drift can obsolete your models, requiring periodic retraining.

Let’s consider the example we’ve been working on: credit card fraud prediction. You may start to notice a large number of transactions that that aren’t coming across as fraudulent compared to historical baselines.

Such a change could be due to many factors: a change in interest rates; the appearance or disappearance of major sources of employment; changes in travel patterns. Whatever the cause, detecting such a change quickly is crucial, so that the business can react quickly in the appropriate manner, whether that means simply retraining the model on fresher data, or a pivot in business strategy.

In Wallaroo you can monitor your housing model for signs of drift through the model monitoring and insight capability called Assays. Assays help you track changes in the environment that your model operates within, which can affect the model’s outcome. It does this by tracking the model’s predictions and/or the data coming into the model against an established baseline. If the distribution of monitored values in the current observation window differs too much from the baseline distribution, the assay will flag it. The figure below shows an example of a running scheduled assay.

Figure: A daily assay that’s been running for a month. The dots represent the difference between the distribution of values in the daily observation window, and the baseline. When that difference exceeds the specified threshold (indicated by a red dot) an alert is set.

This next set of exercises will walk you through setting up an assay to monitor the predictions of your house price model, in order to detect drift.

NOTE

An assay is a monitoring process that typically runs over an extended, ongoing period of time. For example, one might set up an assay that every day monitors the previous 24 hours’ worth of predictions and compares it to a baseline. For the purposes of these exercises, we’ll be compressing processes what normally would take hours or days into minutes.


Exercise Prep: Create some datasets for demonstrating assays

Because assays are designed to detect changes in distributions, let’s try to set up data with different distributions to test with. Take your financial transaction data and create two sets: a set with lower prices, and a set with higher prices. You can split however you choose.

The idea is we will pretend that the set of transactions less likely to trigger fraud warnings versus a set that is known to have a high likelihood of being evaluated as fraudulent.

  • If you are using the pre-provided models to do these exercises, you can use the provided data sets cc_data_low_confidence.df.json and cc_data_high_confidence.df.json. This is to establish our baseline as a set of known values, so the higher prices will trigger our assay alerts.
low_fraud_data = pd.read_json('../data/cc_data_low_confidence.df.json')
high_fraud_data = pd.read_json('../data/cc_data_high_confidence.df.json')

Note that the data in these files are already in the form expected by the models, so you don’t need to use the get_singleton or get_batch convenience functions to infer.

At the end of this exercise, you should have two sets of data to demonstrate assays. In the discussion below, we’ll refer to these sets as low_fraud_data and high_fraud_data.

See Wallaroo SDK Essentials Guide: Assays Management for more details.

# let's start with this - get the 10k of inferences.
# then split them into those < 0.25, and those > 0.75
# blank spot to split or download data

low_fraud_data = pd.read_json('../data/cc_data_low_confidence.df.json')
high_fraud_data = pd.read_json('../data/cc_data_high_confidence.df.json')

We will use this data to set up some “historical data” in the house price prediction pipeline that you build in the assay exercises.

Setting up a baseline for the assay

In order to know whether the distribution of your model’s predictions have changed, you need a baseline to compare them to. This baseline should represent how you expect the model to behave at the time it was trained. This might be approximated by the distribution of the model’s predictions over some “typical” period of time. For example, we might collect the predictions of our model over the first few days after it’s been deployed. For these exercises, we’ll compress that to a few minutes. Currently, to set up a wallaroo assay the pipeline must have been running for some period of time, and the assumption is that this period of time is “typical”, and that the distributions of the inputs and the outputs of the model during this period of time are “typical.”

Exercise Prep: Run some inferences and set some time stamps

Here, we simulate having a pipeline that’s been running for a long enough period of time to set up an assay.

To send enough data through the pipeline to create assays, you execute something like the following code (using the appropriate names for your pipelines and models). Note that this step will take a little while, because of the sleep interval.

You will need the timestamps baseline_start, and baseline_end, for the next exercises.

# get your pipeline (in this example named "mypipeline")

pipeline = get_pipeline("mypipeline")
pipeline.deploy()

## Run some baseline data
# Where the baseline data will start
baseline_start = datetime.datetime.now()

# the number of samples we'll use for the baseline
nsample = 500

# Wait 30 seconds to set this data apart from the rest
# then send the data in batch
time.sleep(30)

# get a sample
lowprice_data_sample = lowprice_data.sample(nsample, replace=True).reset_index(drop=True)
pipeline.infer(lowprice_data_sample)

# Set the baseline end
baseline_end = datetime.datetime.now()
# blank space to get pipeline and set up baseline data

pipeline = get_pipeline('finserv-ccfraud')

ccfraud_keras_model = get_model('ccfraud-model-keras')

pipeline.deploy()
namefinserv-ccfraud
created2023-08-07 16:26:37.485326+00:00
last_updated2023-08-07 17:55:48.136182+00:00
deployedTrue
tags
versionsaf6809a9-088f-47e5-88a4-2005599fde7a, 095ef880-06f0-45d5-bfef-ac4e11d2b06c, f9004ecd-7a8c-4963-bebc-e40d1cde357c, 2513faec-2385-4733-a42d-56a8ae38761a, 10e495af-9b23-4973-913e-4cdebca73461, 782b9edc-d42a-4984-90ca-3bdf35922b87, e80d21a4-21c2-46b8-85d4-6652bbac9506, 8ff48a62-fd9d-43f5-ba3f-11a7f1bcc474, 36faf126-dac5-419d-b0b1-7d7b698b587e, 230d585a-52db-476d-ab28-7b4baed9d023, 192f92e9-9a97-4339-8c1d-f89541ff2cef, 5d2d9c84-13c2-4e35-a41f-ec3c4e8d297b, 41927bef-d8fb-49ee-914e-d106ffc304b3
stepsccfraud-model-keras

Before setting up an assay on this pipeline’s output, we may want to look at the distribution of the predictions over our selected baseline period. To do that, we’ll create an assay_builder that specifies the pipeline, the model in the pipeline, and the baseline period.. We’ll also specify that we want to look at the output of the model, which in the example code is named variable, and would appear as out.variable in the logs.

# print out one of the logs to get the name of the output variable
display(pipeline.logs(limit=1))

# get the model name directly off the pipeline (you could just hard code this, if you know the name)

model_name = pipeline.model_configs()[0].model().name()

assay_builder = ( wl.build_assay(assay_name, pipeline, model_name, 
                     baseline_start, baseline_end)
                    .add_iopath("output variable 0") ) # specify that we are looking at the first output of the output variable "variable"

where baseline_start and baseline_end are the beginning and end of the baseline periods as datetime.datetime objects.

You can then examine the distribution of variable over the baseline period:

assay_builder.baseline_histogram()

Exercise: Create an assay builder and set a baseline

Create an assay builder to monitor the output of your house price pipeline. The baseline period should be from baseline_start to baseline_end.

  • You will need to know the name of your output variable, and the name of the model in the pipeline.

Examine the baseline distribution.

## Blank space to create an assay builder and examine the baseline distribution

display(pipeline.logs(limit=1))
model_name = pipeline.model_configs()[0].model().name()
display(model_name)

import datetime
import time
baseline_start = datetime.datetime.now()
time.sleep(5)

pipeline.infer(low_fraud_data)

time.sleep(5)

baseline_end = datetime.datetime.now()

assay_builder = ( wl.build_assay('sample finserv assay', pipeline, model_name, 
                     baseline_start, baseline_end)
                    .add_iopath("output dense_1 0") )

assay_builder.baseline_histogram()
Warning: There are more logs available. Please set a larger limit or request a file using export_logs.
timein.tensorout.dense_1check_failures
02023-08-07 17:11:21.172[[0.5817662108, 0.097881551, 0.1546819424, 0.4754101949, -0.1978862306, -0.4504344854, 0.0166540447, -0.0256070551, 0.0920561602, -0.2783917153, 0.0593299441, -0.0196585416, -0.4225083157, -0.1217538877, 1.5473094894, 0.2391622864, 0.3553974881, -0.7685165301, -0.7000849355, -0.1190043285, -0.3450517133, -1.1065114108, 0.2523411195, 0.0209441826, 0.2199267436, 0.2540689265, -0.0450225094, 0.1086773898, 0.2547179311]][0.0010916889]0
'ccfraud-model-keras'

An assay should detect if the distribution of model predictions changes from the above distribution over regularly sampled observation windows. This is called drift.

To show drift, we’ll run more data through the pipeline – first some data drawn from the same distribution as the baseline (lowprice_data). Then, we will gradually introduce more data from a different distribution (highprice_data). We should see the difference between the baseline distribution and the distribution in the observation window increase.

To set up the data, you should do something like the below. It will take a while to run, because of all the sleep intervals.

You will need the assay_window_end for a later exercise.

IMPORTANT NOTE: To generate the data for the assay, this process may take 4-5 minutes. Because the shortest period of time for an assay window is 1 minute, the intervals of inference data are spaced to fall within that time period.

# Set the start for our assay window period.
assay_window_start = datetime.datetime.now()

# Run a set of house values, spread across a "longer" period of time

# run "typical" data
for x in range(4):
    pipeline.infer(low_fraud_data.sample(2*nsample, replace=True).reset_index(drop=True))
    time.sleep(25)
    
# run a mix
for x in range(3):
    pipeline.infer(low_fraud_data.sample(nsample, replace=True).reset_index(drop=True))
    pipeline.infer(high_fraud_data.sample(nsample, replace=True).reset_index(drop=True))
    time.sleep(25)
    
# high price houses dominate the sample
for x in range(3):
    pipeline.infer(high_fraud_data.sample(2*nsample, replace=True).reset_index(drop=True))
    time.sleep(25)

# End our assay window period
assay_window_end = datetime.datetime.now()

Exercise Prep: Run some inferences and set some time stamps

Run more data through the pipeline, manifesting a drift, like the example above. It may around 10 minutes depending on how you stagger the inferences.

## Blank space to run more data

assay_window_start = datetime.datetime.now()

# Run a set of house values, spread across a "longer" period of time

nsample = 500

# run "typical" data
for x in range(4):
    pipeline.infer(df.sample(2*nsample, replace=True).reset_index(drop=True))
    time.sleep(25)
    
# run a mix
for x in range(3):
    pipeline.infer(low_fraud_data.sample(nsample, replace=True).reset_index(drop=True))
    pipeline.infer(high_fraud_data.sample(nsample, replace=True).reset_index(drop=True))
    time.sleep(25)
    
# high price houses dominate the sample
for x in range(3):
    pipeline.infer(high_fraud_data.sample(2*nsample, replace=True).reset_index(drop=True))
    time.sleep(25)

# End our assay window period
assay_window_end = datetime.datetime.now()

Defining the Assay Parameters

Now we’re finally ready to set up an assay!

The Observation Window

Once a baseline period has been established, you must define the window of observations that will be compared to the baseline. For instance, you might want to set up an assay that runs every 12 hours, collects the previous 24 hours’ predictions and compares the distribution of predictions within that 24 hour window to the baseline. To set such a comparison up would look like this:

assay_builder.window_builder().add_width(hours=24).add_interval(hours=12)

In other words width is the width of the observation window, and interval is how often an assay (comparison) is run. The default value of width is 24 hours; the default value of interval is to set it equal to width. The units can be specified in one of: minutes, hours, days, weeks.

The Comparison Threshold

Given an observation window and a baseline distribution, an assay computes the distribution of predictions in the observation window. It then calculates the “difference” (or “distance”) between the observed distribution and the baseline distribution. For the assay’s default distance metric (which we will use here), a good starting threshold is 0.1. Since a different value may work best for a specific situation, you can try interactive assay runs on historical data to find a good threshold, as we do in these exercises.

To set the assay threshold for the assays to 0.1:

assay_builder.add_alert_threshold(0.1)

Running an Assay on Historical Data

In this exercise, you will build an interactive assay over historical data. To do this, you need an end time (endtime).

Depending on the historical history, the window and interval may need adjusting. If using the previously generated information, an interval window as short as 1 minute may be useful.

Assuming you have an assay builder with the appropriate window parameters and threshold set, you can do an interactive run and look at the results would look like this.

# set the end of the interactive run
assay_builder.add_run_until(endtime)

# set the window

assay_builder.window_builder().add_width(hours=24).add_interval(hours=12)

assay_results = assay_builder.build().interactive_run()
df = assay_results.to_dataframe() # to return the results as a table
assay_results.chart_scores() # to plot the run

Exercise: Create an interactive assay

Use the assay_builder you created in the previous exercise to set up an interactive assay.

  • The assay should run every minute, on a window that is a minute wide.
  • Set the alert threshold to 0.1.
  • You can use assay_window_end (or a later timestamp) as the end of the interactive run.

Examine the assay results. Do you see any drift?

To try other ways of examining the assay results, see the “Interactive Assay Runs” section of the Model Insights tutorial.

# blank space for setting assay parameters, creating and examining an interactive assay

# set the end of the interactive run
assay_builder.add_run_until(assay_window_end)

# doing minutes to get our previous values in
assay_builder.window_builder().add_width(minutes=1).add_interval(minutes=1)
assay_builder.add_alert_threshold(0.1)
assay_results = assay_builder.build().interactive_run()
df = assay_results.to_dataframe() # to return the results as a table
assay_results.chart_scores() # to plot the run

Scheduling an Assay to run on ongoing data

(We won’t be doing an exercise here, this is for future reference).

Once you are satisfied with the parameters you have set, you can schedule an assay to run regularly .

# create a fresh assay builder with the correct parameters
assay_builder = ( wl.build_assay(assay_name, pipeline, model_name, 
                     baseline_start, baseline_end)
                    .add_iopath("output variable 0") )

# this assay runs every 24 hours on a 24 hour window
assay_builder.window_builder().add_width(hours=24)
assay_builder.add_alert_threshold(0.1)

# now schedule the assay
assay_id = assay_builder.upload()

You can use the assay id later to get the assay results.

You have now walked through setting up a basic assay and running it over historical data.

Congratulations!

In this tutorial you have:

  • Deployed a single step financial transaction classification pipeline and sent data to it.
  • Compared two financial transaction classification models in an A/B test
  • Compared two financial transaction classification models in a shadow deployment.
  • Swapped the “winner” of the comparisons into the financial transaction classification pipeline.
  • Set validation rules on the pipeline.
  • Set up an assay on the pipeline to monitor for drift in its predictions.

Great job!

Cleaning up.

Now that the tutorial is complete, don’t forget to undeploy your pipeline to free up the resources.

# blank space to undeploy your pipeline

pipeline.undeploy()
namefinserv-ccfraud
created2023-08-07 16:26:37.485326+00:00
last_updated2023-08-07 18:11:00.015882+00:00
deployedFalse
tags
versions405d7b12-309e-44a1-968c-744f765b42a2, a665ba59-bc76-4a87-a836-4dd65da8b115, af6809a9-088f-47e5-88a4-2005599fde7a, 095ef880-06f0-45d5-bfef-ac4e11d2b06c, f9004ecd-7a8c-4963-bebc-e40d1cde357c, 2513faec-2385-4733-a42d-56a8ae38761a, 10e495af-9b23-4973-913e-4cdebca73461, 782b9edc-d42a-4984-90ca-3bdf35922b87, e80d21a4-21c2-46b8-85d4-6652bbac9506, 8ff48a62-fd9d-43f5-ba3f-11a7f1bcc474, 36faf126-dac5-419d-b0b1-7d7b698b587e, 230d585a-52db-476d-ab28-7b4baed9d023, 192f92e9-9a97-4339-8c1d-f89541ff2cef, 5d2d9c84-13c2-4e35-a41f-ec3c4e8d297b, 41927bef-d8fb-49ee-914e-d106ffc304b3
stepsccfraud-model-keras

2 - Edge Deployment

Wallaroo Use Case Tutorials focused on Edge Deployments of ML Models.

Training on deploying Wallaroo pipelines to Edge devices. This includes:

  • Publishing pipelines and models to an OCI registry.
  • Deploying the Wallaroo pipelines to Edge devices.
  • Performing sample inferences and observing the results.

All samples are available from the Wallaroo Workshop GitHub Repository.

2.1 - Edge Deployment: Computer Vision Yolov8n

Wallaroo Use Case Tutorials focused on Edge Deployments of Computer Vision YoloV8n ML Models.

This workshop is available from the Wallaroo Workshop GitHub Repository.

Computer Vision Yolov8n Edge Deployment and Observability in Wallaroo

The Yolov8 computer vision model is used for fast recognition of objects in images. This tutorial demonstrates how to:

  • Deploy a Yolov8n pre-trained model into a Wallaroo Ops server and perform inferences on it.
  • Publish the pipeline to the OCI registry configured in the Wallaroo Ops server.
  • Add an edge location to the Wallaroo pipeline publish.
  • Deploy the pipeline as a Wallaroo Server on an edge device through Docker, and display the inference logs submitted to the Wallaroo Ops server.

Wallaroo Ops Center provides the ability to publish Wallaroo pipelines to an Open Continer Initative (OCI) compliant registry, then deploy those pipelines on edge devices as Docker container or Kubernetes pods. See Wallaroo SDK Essentials Guide: Pipeline Edge Publication for full details.

For this tutorial, the helper module CVDemoUtils and WallarooUtils are used to transform a sample image into a pandas DataFrame. This DataFrame is then submitted to the Yolov8n model deployed in Wallaroo.

References

  • Wallaroo Workspaces: Workspaces are environments were users upload models, create pipelines and other artifacts. The workspace should be considered the fundamental area where work is done. Workspaces are shared with other users to give them access to the same models, pipelines, etc.
  • Wallaroo Model Upload and Registration: ML Models are uploaded to Wallaroo through the SDK or the MLOps API to a workspace. ML models include default runtimes (ONNX, Python Step, and TensorFlow) that are run directly through the Wallaroo engine, and containerized runtimes (Hugging Face, PyTorch, etc) that are run through in a container through the Wallaroo engine.
  • Wallaroo Pipelines: Pipelines are used to deploy models for inferencing. Each model is a pipeline step in a pipelines, where the inputs of the previous step are fed into the next. Pipeline steps can be ML models, Python scripts, or Arbitrary Python (these contain necessary models and artifacts for running a model).
  • Wallaroo SDK Essentials Guide: Pipeline Edge Publication: Details on publishing a Wallaroo pipeline to an OCI Registry and deploying it as a Wallaroo Server instance.

Data Scientist Steps

The following details the steps a Data Scientist performs in uploading and verifying the model in a Wallaroo Ops server.

Load Libraries

The first step is loading the required libraries including the Wallaroo Python module.

# Import Wallaroo Python SDK
import wallaroo
from wallaroo.object import EntityNotFoundError
from wallaroo.framework import Framework
from CVDemoUtils import CVDemo
from WallarooUtils import Util
cvDemo = CVDemo()
util = Util()

# used to display DataFrame information without truncating
from IPython.display import display
import pandas as pd
pd.set_option('display.max_colwidth', None)

Connect to the Wallaroo Instance through the User Interface

The next step is to connect to Wallaroo through the Wallaroo client. The Python library is included in the Wallaroo install and available through the Jupyter Hub interface provided with your Wallaroo environment.

This is accomplished using the wallaroo.Client() command, which provides a URL to grant the SDK permission to your specific Wallaroo environment. When displayed, enter the URL into a browser and confirm permissions. Store the connection into a variable that can be referenced later.

If logging into the Wallaroo instance through the internal JupyterHub service, use wl = wallaroo.Client(). For more information on Wallaroo Client settings, see the Client Connection guide.

Connect to the Wallaroo Instance Exercise

Connect to the Wallaroo instance. If connecting through the JupyterHub service, then only the wallaroo.Client() is required. If connecting externally through the Wallaroo SDK, use the wallaroo.client(api_endpoint, auth_endpoint) method.

Sample code:

wl = wallaroo.Client()
# Connect to the Wallaroo instance here

wl = wallaroo.Client()

Create a New Workspace

We’ll use the SDK below to create our workspace , assign as our current workspace, then display all of the workspaces we have at the moment. We’ll also set up variables for our models and pipelines down the road, so we have one spot to change names to whatever fits your organization’s standards best.

To allow this tutorial to be run by multiple users in the same Wallaroo instance, update suffix with your first and last name. For example:

suffix = 'lazel-geth'

Create a New Workspace Exercise

Set the model name, file name, pipeline name, and workspace name.

Sample code:

suffix = ''

model_name = 'yolov8n'
model_filename = './models/cv-yolo/yolov8n.onnx'
pipeline_name = 'yolo8demonstration'
workspace_name = f'yolo8-edge-demonstration{suffix}'
# set helper variables here

suffix = ''

model_name = 'yolov8n'
model_filename = './models/cv-yolo/yolov8n.onnx'
pipeline_name = 'yolo8demonstration'
workspace_name = f'yolo8-edge-demonstration{suffix}'

Set the Current Workspace

Set the current workspace where the models are uploaded to and pipelines created.

Set the Current Workspace Exercise

Setting the workspace is performed with the wallaroo.client.set_current_workspace(workspace) method.

Sample code:

workspace = get_workspace(workspace_name, wl)
wl.set_current_workspace(workspace)
def get_workspace(name, client):
    workspace = None
    for ws in client.list_workspaces():
        if ws.name() == name:
            workspace= ws
    if(workspace == None):
        workspace = client.create_workspace(name)
    return workspace

workspace = get_workspace(workspace_name, wl)
wl.set_current_workspace(workspace)
{'name': 'yolo8-edge-demonstration', 'id': 8, 'archived': False, 'created_by': '93dc3434-cf57-47b4-a3e4-b2168b1fd7e1', 'created_at': '2023-12-11T20:30:18.268429+00:00', 'models': [], 'pipelines': []}

Upload the Model

When a model is uploaded to a Wallaroo cluster, it is optimized and packaged to make it ready to run as part of a pipeline. In many times, the Wallaroo Server can natively run a model without any Python overhead. In other cases, such as a Python script, a custom Python environment will be automatically generated. This is comparable to the process of “containerizing” a model by adding a small HTTP server and other wrapping around it.

Our pretrained model is in ONNX format, which is specified in the framework parameter. For this model, the tensor fields are set to images to match the input parameters, and the batch configuration is set to single - only one record will be submitted at a time.

Upload the Model Exercise

The model name and file name were set in the variables above. Use them to upload the model.

Sample code:

yolov8_model = (wl.upload_model(model_name, 
                               model_filename, 
                               framework=Framework.ONNX)
                               .configure(tensor_fields=['images'],
                                          batch_config="single"
                                          )
                )
# Upload Retrained Yolo8 Model 
yolov8_model = (wl.upload_model(model_name, 
                               model_filename, 
                               framework=Framework.ONNX)
                               .configure(tensor_fields=['images'],
                                          batch_config="single"
                                          )
                )

Pipeline Deployment Configuration

For our pipeline we set the deployment configuration to only use 1 cpu and 1 GiB of RAM.

Pipeline Deployment Configuration Exercise

Use the deployment configuration below.

deployment_config = wallaroo.DeploymentConfigBuilder() \
                    .replica_count(1) \
                    .cpus(1) \
                    .memory("1Gi") \
                    .build()

Build and Deploy the Pipeline

Now we build our pipeline and set our Yolo8 model as a pipeline step, then deploy the pipeline using the deployment configuration above.

Build and Deploy the Pipeline Exercise

We’ll do both commands in one step:

  • Build the pipeline with wallaroo.client.build_pipeline.
  • Set the model as a pipeline step with wallaroo.pipeline.add_model_step(model) method.

Sample code:

pipeline = wl.build_pipeline(pipeline_name) \
            .add_model_step(yolov8_model)        
pipeline = wl.build_pipeline(pipeline_name) \
            .add_model_step(yolov8_model)        

Deploy the Pipeline

We deploy the pipeline with the wallaroo.pipeline.deploy(deployment_config) command, using the deployment configuration set up in previous steps.

Deploy the Pipeline Exercise

Deploy the pipeline.

Sample code:

pipeline.deploy(deployment_config=deployment_config)
# deploy the pipeline

pipeline.deploy(deployment_config=deployment_config)
nameyolo8demonstration
created2023-12-11 20:30:22.268052+00:00
last_updated2023-12-11 20:30:23.212091+00:00
deployedTrue
archNone
tags
versions0c72211e-bc09-4026-b0ce-7c31db1a377d, 6069cb3a-ec03-44b4-8845-a4d8cafb2978
stepsyolov8n
publishedFalse

Convert Image to DataFrame

The sample image dogbike.png was converted to a DataFrame using the cvDemo helper modules. The converted DataFrame is stored as ./data/dogbike.df.json to save time.

The code sample below demonstrates how to use this module to convert the sample image to a DataFrame.

# convert the image to a tensor

width, height = 640, 640
tensor1, resizedImage1 = cvDemo.loadImageAndResize('dogbike.png', width, height)
tensor1.flatten()

# add the tensor to a DataFrame and save the DataFrame in pandas record format
df = util.convert_data(tensor1,'images')
df.to_json("data.json", orient = 'records')

Inference Request

We submit the DataFrame to the pipeline using wallaroo.pipeline.infer, and store the results in the variable inf1. A copy of the dataframe is stored in the file ./data/dogbike.df.json.

For this, we will use our cvDemo module to resize the image and retrieve the tensor values.

Inference Request Exercise

To use the cvDemo, we will:

  • Convert the image to a 640x640 size to fit the model’s inputs.
  • Create a pandas DataFrame from the image tensor data.
  • Submit the inference request and save the data as a variable.

Sample code:

width, height = 640, 640
tensor1, resizedImage1 = cvDemo.loadImageAndResize('./data/cv-yolo/dogbike.png', width, height)

# convert tensor1 to a pandas DataFrame

# add the tensor to a DataFrame and save the DataFrame in pandas record format
df = util.convert_data(tensor1,'images')
df.to_json("./data/cv-yolo/data.df.json", orient="records")

# inf1 = pipeline.infer_from_file('./data/cv-yolo/dogbike.df.json')
inf1 = pipeline.infer(df)
# inference code here

width, height = 640, 640
tensor1, resizedImage1 = cvDemo.loadImageAndResize('./data/cv-yolo/dogbike.png', width, height)

# convert tensor1 to a pandas DataFrame

# add the tensor to a DataFrame and save the DataFrame in pandas record format
df = util.convert_data(tensor1,'images')
df.to_json("./data/cv-yolo/data.df.json", orient="records")

# inf1 = pipeline.infer_from_file('./data/cv-yolo/dogbike.df.json')
inf1 = pipeline.infer(df)

Display Bounding Boxes

Using our helper method cvDemo we’ll identify the objects detected in the photo and their bounding boxes. Only objects with a confidence threshold of 50% or more are shown.

Note that the first inputs are the inference results from the previous inference request, and the second variable is the resized image. Note that the first two arguments are the inference results obtained from the inference request, and the resized image.

Display Bounding Boxes Exercise

Use the following code, modified based on the name of your inference results and resized image variables.

confidence_thres = 0.50
iou_thres = 0.25

cvDemo.drawYolo8Boxes(inf1, resizedImage1, width, height, confidence_thres, iou_thres, draw=True)
# draw the bounding boxes from the inference results

confidence_thres = 0.50
iou_thres = 0.25

cvDemo.drawYolo8Boxes(inf1, resizedImage1, width, height, confidence_thres, iou_thres, draw=True)
  Score: 86.47% | Class: Dog | Bounding Box: [108, 250, 149, 356]
  Score: 81.13% | Class: Bicycle | Bounding Box: [97, 149, 375, 323]
  Score: 63.16% | Class: Car | Bounding Box: [390, 85, 186, 108]
array([[[ 34,  34,  34],
        [ 35,  35,  35],
        [ 33,  33,  33],
        ...,
        [ 33,  33,  33],
        [ 33,  33,  33],
        [ 35,  35,  35]],

       [[ 33,  33,  33],
        [ 34,  34,  34],
        [ 34,  34,  34],
        ...,
        [ 34,  34,  34],
        [ 33,  33,  33],
        [ 34,  34,  34]],

       [[ 53,  54,  48],
        [ 54,  55,  49],
        [ 54,  55,  49],
        ...,
        [153, 178, 111],
        [151, 183, 108],
        [159, 176,  99]],

       ...,

       [[159, 167, 178],
        [159, 165, 177],
        [158, 163, 175],
        ...,
        [126, 127, 121],
        [127, 125, 120],
        [128, 120, 117]],

       [[160, 168, 179],
        [156, 162, 174],
        [152, 157, 169],
        ...,
        [126, 127, 121],
        [129, 127, 122],
        [127, 118, 116]],

       [[155, 163, 174],
        [155, 162, 174],
        [152, 158, 170],
        ...,
        [127, 127, 121],
        [130, 126, 122],
        [128, 119, 116]]], dtype=uint8)

Inference Through Pipeline API

Another method of performing an inference using the pipeline’s deployment url.

Performing an inference through an API requires the following:

  • The authentication token to authorize the connection to the pipeline.
  • The pipeline’s inference URL.
  • Inference data to sent to the pipeline - in JSON, DataFrame records format, or Apache Arrow.

Full details are available through the Wallaroo API Connection Guide on how retrieve an authorization token and perform inferences through the pipeline’s API.

For this demonstration we’ll submit the pandas record, request a pandas record as the return, and set the authorization header. The results will be stored in the file curl_response.df.

Inference Through Pipeline API Exercise

We’ll use the pipeline deployment URL to submit the inference request as an API call. The following is sample code for adding the authentication token and setting the Content-Type to pandas dataFrame.

!curl -X POST {pipeline._deployment._url()} \
    -H "Authorization:{wl.auth.auth_header()['Authorization']}" \
    -H "Content-Type:application/json; format=pandas-records" \
    --data @./data/cv-yolo/data.df.json > curl_response.df
# inference request here.

!curl -X POST {pipeline._deployment._url()} \
    -H "Authorization:{wl.auth.auth_header()['Authorization']}" \
    -H "Content-Type:application/json; format=pandas-records" \
    --data @./data/cv-yolo/data.df.json > curl_response.df
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 38.0M  100 22.9M  100 15.0M  5910k  3890k  0:00:03  0:00:03 --:--:-- 9809k

Undeploy the Pipeline

Undeploy the pipeline and return the resources back to the Wallaroo cluster.

Undeploy the Pipeline Exercise

Sample code:

pipeline.undeploy()
# undeploy the pipeline

pipeline.undeploy()
nameyolo8demonstration
created2023-12-11 20:30:22.268052+00:00
last_updated2023-12-11 20:30:23.212091+00:00
deployedFalse
archNone
tags
versions0c72211e-bc09-4026-b0ce-7c31db1a377d, 6069cb3a-ec03-44b4-8845-a4d8cafb2978
stepsyolov8n
publishedFalse

Publish the Pipeline for Edge Deployment

It worked! For a demo, we’ll take working once as “tested”. So now that we’ve tested our pipeline, we are ready to publish it for edge deployment.

Publishing it means assembling all of the configuration files and model assets and pushing them to an Open Container Initiative (OCI) repository set in the Wallaroo instance as the Edge Registry service. DevOps engineers then retrieve that image and deploy it through Docker, Kubernetes, or similar deployments.

See Edge Deployment Registry Guide for details on adding an OCI Registry Service to Wallaroo as the Edge Deployment Registry.

This is done through the SDK command wallaroo.pipeline.publish(deployment_config) which has the following parameters and returns.

Publish a Pipeline Parameters

The publish method takes the following parameters. The containerized pipeline will be pushed to the Edge registry service with the model, pipeline configurations, and other artifacts needed to deploy the pipeline.

ParameterTypeDescription
deployment_configwallaroo.deployment_config.DeploymentConfig (Optional)Sets the pipeline deployment configuration. For example: For more information on pipeline deployment configuration, see the Wallaroo SDK Essentials Guide: Pipeline Deployment Configuration.

Publish a Pipeline Returns

FieldTypeDescription
idintegerNumerical Wallaroo id of the published pipeline.
pipeline version idintegerNumerical Wallaroo id of the pipeline version published.
statusstringThe status of the pipeline publication. Values include:
  • PendingPublish: The pipeline publication is about to be uploaded or is in the process of being uploaded.
  • Published: The pipeline is published and ready for use.
Engine URLstringThe URL of the published pipeline engine in the edge registry.
Pipeline URLstringThe URL of the published pipeline in the edge registry.
Helm Chart URLstringThe URL of the helm chart for the published pipeline in the edge registry.
Helm Chart ReferencestringThe help chart reference.
Helm Chart VersionstringThe version of the Helm Chart of the published pipeline. This is also used as the Docker tag.
Engine Configwallaroo.deployment_config.DeploymentConfigThe pipeline configuration included with the published pipeline.
Created AtDateTimeWhen the published pipeline was created.
Updated AtDateTimeWhen the published pipeline was updated.

Publish Exercise

We will now publish the pipeline to our Edge Deployment Registry with the pipeline.publish(deployment_config) command. deployment_config is an optional field that specifies the pipeline deployment. This can be overridden by the DevOps engineer during deployment.

Save the publish to a variable for later use.

Sample code:

pub = pipeline.publish(deployment_config)
pub
pub = pipeline.publish(deployment_config)
pub
Waiting for pipeline publish... It may take up to 600 sec.
Pipeline is Publishing...Published.
ID2
Pipeline Versiona0d7c479-3696-4a65-96a3-6b247a8b7417
StatusPublished
Engine URLghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/standalone-mini:v2023.4.0-4103
Pipeline URLghcr.io/wallaroolabs/doc-samples/pipelines/yolo8demonstration:a0d7c479-3696-4a65-96a3-6b247a8b7417
Helm Chart URLoci://ghcr.io/wallaroolabs/doc-samples/charts/yolo8demonstration
Helm Chart Referenceghcr.io/wallaroolabs/doc-samples/charts@sha256:37d78ff9caf64605335b6809bd62f8ebd5503f7eea10ea3c900f9c76df22a4dc
Helm Chart Version0.0.1-a0d7c479-3696-4a65-96a3-6b247a8b7417
Engine Config{'engine': {'resources': {'limits': {'cpu': 1.0, 'memory': '512Mi'}, 'requests': {'cpu': 1.0, 'memory': '512Mi'}}}, 'engineAux': {'images': {}}, 'enginelb': {'resources': {'limits': {'cpu': 1.0, 'memory': '512Mi'}, 'requests': {'cpu': 1.0, 'memory': '512Mi'}}}}
User Images[]
Created Byjohn.hansarick@wallaroo.ai
Created At2023-12-11 20:31:33.040354+00:00
Updated At2023-12-11 20:31:33.040354+00:00
Docker Run Variables{}

List Published Pipeline

The method wallaroo.client.list_pipelines() shows a list of all pipelines in the Wallaroo instance, and includes the published field that indicates whether the pipeline was published to the registry (True), or has not yet been published (False).

List Published Pipeline Exercise

List the pipelines and verify which ones are published or not.

Sample code:

wl.list_pipelines()
# list pipelines

wl.list_pipelines()
namecreatedlast_updateddeployedarchtagsversionsstepspublished
yolo8demonstration2023-11-Dec 20:30:222023-11-Dec 20:31:31FalseNonea0d7c479-3696-4a65-96a3-6b247a8b7417, 0c72211e-bc09-4026-b0ce-7c31db1a377d, 6069cb3a-ec03-44b4-8845-a4d8cafb2978yolov8nTrue
api-pipeline-with-models2023-11-Dec 15:28:292023-11-Dec 15:28:29TrueNonedd441dbc-196a-43cd-add9-7b55352a1c23api-sample-modelFalse
cv-retail-edge2023-08-Dec 18:29:582023-08-Dec 18:43:44FalseNone56837872-9c51-4e81-89b8-5df3703a2c72, 9e9a2045-74cb-4451-b151-13fcfddba919, d73a04bf-b725-44d7-98e1-7110e94346a9, 0436d568-ad00-4191-a15c-5810b06fdfcd, 6c42790f-6ba2-42cf-bc97-93f47bfae5c2, 37582867-39e9-4243-b87e-ae9b171103e3, c73d51be-a384-46b0-84c0-9ba6348ec760resnet50True
cv-retail-assay2023-08-Dec 18:18:222023-08-Dec 18:18:38FalseNonea420499e-bb9e-495d-b0f5-67c7102b4949, af4a642f-8186-460b-a24f-2d0fe5a89765, bf0336d3-2bbd-455f-8b2d-907135b90c7c, 0199813b-e7cf-4d7c-8690-3dcf3291fbc2resnet50False
cv-retail2023-08-Dec 18:10:332023-08-Dec 18:16:07FalseNone01ce3a4c-eb4c-4a9d-acea-d4b8fde02d46, e791bb6f-1f0c-49be-8f27-3c987cdee4ab, 43ef6552-12c1-4d4c-a8d0-58a0209a721f, 7ce5dd31-5851-492b-ba36-80e4adf8ef04, 0a4ce878-3f59-4134-b32f-1a6946db9775, e7149fb9-8253-4e87-b9a6-a220c8df5639, c6abcdaa-047b-4eef-8f83-eb2dcc8782d1mobilenetFalse

List Publishes from a Pipeline

All publishes created from a pipeline are displayed with the wallaroo.pipeline.publishes method. The pipeline_version_id is used to know what version of the pipeline was used in that specific publish. This allows for pipelines to be updated over time, and newer versions to be sent and tracked to the Edge Deployment Registry service.

List Publishes Parameters

N/A

List Publishes Returns

A List of the following fields:

FieldTypeDescription
idintegerNumerical Wallaroo id of the published pipeline.
pipeline_version_idintegerNumerical Wallaroo id of the pipeline version published.
engine_urlstringThe URL of the published pipeline engine in the edge registry.
pipeline_urlstringThe URL of the published pipeline in the edge registry.
created_bystringThe email address of the user that published the pipeline.
Created AtDateTimeWhen the published pipeline was created.
Updated AtDateTimeWhen the published pipeline was updated.

List Publishes from a Pipeline Exercise

List the publishes from a pipeline.

Sample code:

pipeline.publishes()
# list the pipeline publishes

pipeline.publishes()
idpipeline_version_nameengine_urlpipeline_urlcreated_bycreated_atupdated_at
2a0d7c479-3696-4a65-96a3-6b247a8b7417ghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/standalone-mini:v2023.4.0-4103ghcr.io/wallaroolabs/doc-samples/pipelines/yolo8demonstration:a0d7c479-3696-4a65-96a3-6b247a8b7417john.hansarick@wallaroo.ai2023-11-Dec 20:31:332023-11-Dec 20:31:33

Add Edge Location

With the pipeline publish created, we can add an Edge Location. This allows the edge deployment to upload its inference results back to the Wallaroo Ops location, which are then added to the pipeline the publish originated from. These are added to the pipeline logs partition metadata.

First we’ll retrieve the pipeline logs for our current pipeline, and show the current pipeline logs metadata.

Then we’ll add the edge location we’ll deploy with the wallaroo.pipeline_publish.add_edge(name: string, tags: List[string]) method.

For the edge name, set it to firstname-lastname-edge-yolo.

Add Edge Location Exercise

Display the log information with the metadata.partition, then add the edge location to the publish. Note that edge names must be unique, so add your first and last name to the list.

Sample code:

logs = pipeline.logs(dataset=['time', 'out.output0', 'metadata'])
display(logs.loc[:, ['time', 'metadata.partition']])

first_last_name = '-Gale-Karlach'

edge_name = f'yolo-edge-demo{first_last_name}'

edge_publish = pub.add_edge(edge_name)
display(edge_publish)
# display log information here with partition

logs = pipeline.logs(dataset=['time', 'out.output0', 'metadata'])
display(logs.loc[:, ['time', 'metadata.partition']])
Warning: The inference log is above the allowable limit and the following columns may have been suppressed for various rows in the logs: ['in.images']. To review the dropped columns for an individual inference’s suppressed data, include dataset=["metadata"] in the log request.
Warning: Pipeline log size limit exceeded. Please request logs using export_logs
timemetadata.partition
02023-12-06 23:18:52.460engine-b4bb5689c-h49ws
# add edge here
edge_name = 'yolo-edge-demo-jch2'

edge_publish = pub.add_edge(edge_name)
display(edge_publish)
ID5
Pipeline Version85bb8216-f4e8-4d29-a3c8-78806738ee4b
StatusPublished
Engine URLghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/standalone-mini:v2023.4.0-4103
Pipeline URLghcr.io/wallaroolabs/doc-samples/pipelines/yolo8demonstration:85bb8216-f4e8-4d29-a3c8-78806738ee4b
Helm Chart URLoci://ghcr.io/wallaroolabs/doc-samples/charts/yolo8demonstration
Helm Chart Referenceghcr.io/wallaroolabs/doc-samples/charts@sha256:138ec8b7365b7b3590871aaf3fea7dc126eb9ecb8a943f9228d5961f18d92735
Helm Chart Version0.0.1-85bb8216-f4e8-4d29-a3c8-78806738ee4b
Engine Config{'engine': {'resources': {'limits': {'cpu': 1.0, 'memory': '512Mi'}, 'requests': {'cpu': 1.0, 'memory': '512Mi'}}}, 'engineAux': {'images': {}}, 'enginelb': {'resources': {'limits': {'cpu': 1.0, 'memory': '512Mi'}, 'requests': {'cpu': 1.0, 'memory': '512Mi'}}}}
User Images[]
Created Byjohn.hummel@wallaroo.ai
Created At2023-12-06 23:20:14.566889+00:00
Updated At2023-12-06 23:20:14.566889+00:00
Docker Run Variables{'EDGE_BUNDLE': 'abcde'}

DevOps - Pipeline Edge Deployment

Once a pipeline is deployed to the Edge Registry service, it can be deployed in environments such as Docker, Kubernetes, or similar container running services by a DevOps engineer.

Docker Deployment

First, the DevOps engineer must authenticate to the same OCI Registry service used for the Wallaroo Edge Deployment registry.

For more details, check with the documentation on your artifact service. The following are provided for the three major cloud services:

For the deployment, the engine URL is specified with the following environmental variables:

  • DEBUG (true|false): Whether to include debug output.
  • OCI_REGISTRY: The URL of the registry service.
  • CONFIG_CPUS: The number of CPUs to use.
  • OCI_USERNAME: The edge registry username.
  • OCI_PASSWORD: The edge registry password or token.
  • PIPELINE_URL: The published pipeline URL.

Docker Deployment Example

Using our sample environment, here’s sample deployment using Docker with a computer vision ML model, the same used in the Wallaroo Use Case Tutorials Computer Vision: Retail tutorials.

Note the use of the -v ./data:/persist option. This will store the one time authentication token stored in the EDGE_BUNDLE

mkdir ./data

docker run -p 8080:8080 \
    -v ./data:/persist \
    -e DEBUG=true -e OCI_REGISTRY={your registry server} \
    -e EDGE_BUNDLE={edge_publish.docker_run_variables['EDGE_BUNDLE']} \
    -e CONFIG_CPUS=4 \
    -e OCI_USERNAME=oauth2accesstoken \
    -e OCI_PASSWORD={registry token here} \
    -e PIPELINE_URL={your registry server}/pipelines/yolo8demonstration:bf70eaf7-8c11-4b46-b751-916a43b1a555 \
    {your registry server}/engine:v2023.3.0-main-3707

Docker Compose Deployment

For users who prefer to use docker compose, the following sample compose.yaml file is used to launch the Wallaroo Edge pipeline. This is the same used in the Wallaroo Use Case Tutorials Computer Vision: Retail tutorials.

The volumes settings allows for persistent volumes to store the session information. Without it, the one-time authentication token included in the EDGE_BUNDLE settings would have to be regenerated.

services:
  engine:
    image: {Your Engine URL}
    volumes:
      - ./data:/persist
    ports:
      - 8080:8080
    environment:
      EDGE_BUNDLE: abcdefg
      PIPELINE_URL: {Your Pipeline URL}
      OCI_REGISTRY: {Your Edge Registry URL}
      OCI_USERNAME:  {Your Registry Username}
      OCI_PASSWORD: {Your Token or Password}
      CONFIG_CPUS: 4

For example:

services:
  engine:
    image: sample-registry.com/engine:v2023.3.0-main-3707
    ports:
      - 8080:8080
    environment:
      PIPELINE_URL: sample-registry.com/pipelines/yolo8demonstration:bf70eaf7-8c11-4b46-b751-916a43b1a555
      OCI_REGISTRY: sample-registry.com
      OCI_USERNAME:  _json_key_base64
      OCI_PASSWORD: abc123
      CONFIG_CPUS: 4

Docker Compose Deployment Example

The deployment and undeployment is then just a simple docker compose up and docker compose down. The following shows an example of deploying the Wallaroo edge pipeline using docker compose.

docker compose up
[+] Running 1/1
 ✔ Container yolo8demonstration-engine-1  Recreated                                                                                                                                                                 0.5s
Attaching to yolo8demonstration-engine-1
yolo8demonstration-engine-1  | Wallaroo Engine - Standalone mode
yolo8demonstration-engine-1  | Login Succeeded
yolo8demonstration-engine-1  | Fetching manifest and config for pipeline: sample-registry.com/pipelines/yolo8demonstration:bf70eaf7-8c11-4b46-b751-916a43b1a555
yolo8demonstration-engine-1  | Fetching model layers
yolo8demonstration-engine-1  | digest: sha256:c6c8869645962e7711132a7e17aced2ac0f60dcdc2c7faa79b2de73847a87984
yolo8demonstration-engine-1  |   filename: c6c8869645962e7711132a7e17aced2ac0f60dcdc2c7faa79b2de73847a87984
yolo8demonstration-engine-1  |   name: yolov8n
yolo8demonstration-engine-1  |   type: model
yolo8demonstration-engine-1  |   runtime: onnx
yolo8demonstration-engine-1  |   version: 693e19b5-0dc7-4afb-9922-e3f7feefe66d
yolo8demonstration-engine-1  |
yolo8demonstration-engine-1  | Fetched
yolo8demonstration-engine-1  | Starting engine
yolo8demonstration-engine-1  | Looking for preexisting `yaml` files in //modelconfigs
yolo8demonstration-engine-1  | Looking for preexisting `yaml` files in //pipelines

Helm Deployment

Published pipelines can be deployed through the use of helm charts.

Helm deployments take up to two steps - the first step is in retrieving the required values.yaml and making updates to override.

Kubernetes provides persistent volume support, so no settings are required.

  1. Pull the helm charts from the published pipeline. The two fields are the Helm Chart URL and the Helm Chart version to specify the OCI . This typically takes the format of:
helm pull oci://{published.helm_chart_url} --version {published.helm_chart_version}
  1. Extract the tgz file and copy the values.yaml and copy the values used to edit engine allocations, etc. The following are required for the deployment to run:
ociRegistry:
  registry: {your registry service}
  username:  {registry username here}
  password: {registry token here}

Store this into another file, suc as local-values.yaml.

  1. Create the namespace to deploy the pipeline to. For example, the namespace wallaroo-edge-pipeline would be:
kubectl create -n wallaroo-edge-pipeline
  1. Deploy the helm installation with helm install through one of the following options:

    1. Specify the tgz file that was downloaded and the local values file. For example:

      helm install --namespace {namespace} --values {local values file} {helm install name} {tgz path}
      
    2. Specify the expended directory from the downloaded tgz file.

      helm install --namespace {namespace} --values {local values file} {helm install name} {helm directory path}
      
    3. Specify the Helm Pipeline Helm Chart and the Pipeline Helm Version.

      helm install --namespace {namespace} --values {local values file} {helm install name} oci://{published.helm_chart_url} --version {published.helm_chart_version}
      
  2. Once deployed, the DevOps engineer will have to forward the appropriate ports to the svc/engine-svc service in the specific pipeline. For example, using kubectl port-forward to the namespace ccfraud that would be:

    kubectl port-forward svc/engine-svc -n ccfraud01 8080 --address 0.0.0.0`
    

Docker Deployment Code Generation Exercise

The following code segment generates a docker run template based on the previously published pipeline. Replace the $REGISTRYURL, $REGISTRYUSERNAME, and $REGISTRYPASSWORD to match the OCI Registry being used.

docker_deploy = f'''
mkdir ./data
docker run -p 8080:8080 \\
    -v ./data:/persist \\
    -e DEBUG=true \\
    -e OCI_REGISTRY=$REGISTRYURL \\
    -e EDGE_BUNDLE={edge_publish.docker_run_variables['EDGE_BUNDLE']} \\
    -e CONFIG_CPUS=1 \\
    -e OCI_USERNAME=$REGISTRYUSERNAME \\
    -e OCI_PASSWORD=$REGISTRYPASSWORD \\
    -e PIPELINE_URL={edge_publish.pipeline_url} \\
    {edge_publish.engine_url}
'''

print(docker_deploy)
mkdir ./data
docker run -p 8080:8080 \
    -v ./data:/persist \
    -e DEBUG=true \
    -e OCI_REGISTRY=$REGISTRYURL \
    -e EDGE_BUNDLE=ZXhwb3J0IEJVTkRMRV9WRVJTSU9OPTEKZXhwb3J0IEVER0VfTkFNRT15b2xvLWVkZ2UtZGVtby1qY2gyCmV4cG9ydCBKT0lOX1RPS0VOPWQ4OGQyZDY0LWQyNmUtNDg3YS04ZGY4LWFhYTdhNzY0OTA0MApleHBvcnQgT1BTQ0VOVEVSX0hPU1Q9ZG9jLXRlc3QuZWRnZS53YWxsYXJvb2NvbW11bml0eS5uaW5qYQpleHBvcnQgUElQRUxJTkVfVVJMPWdoY3IuaW8vd2FsbGFyb29sYWJzL2RvYy1zYW1wbGVzL3BpcGVsaW5lcy95b2xvOGRlbW9uc3RyYXRpb246ODViYjgyMTYtZjRlOC00ZDI5LWEzYzgtNzg4MDY3MzhlZTRiCmV4cG9ydCBXT1JLU1BBQ0VfSUQ9MTA= \
    -e CONFIG_CPUS=1 \
    -e OCI_USERNAME=$REGISTRYUSERNAME \
    -e OCI_PASSWORD=$REGISTRYPASSWORD \
    -e PIPELINE_URL=ghcr.io/wallaroolabs/doc-samples/pipelines/yolo8demonstration:85bb8216-f4e8-4d29-a3c8-78806738ee4b \
    ghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/standalone-mini:v2023.4.0-4103

Edge Deployed Pipeline API Endpoints

Once deployed, we can check the pipelines and models available. We’ll use a curl command, but any HTTP based request will work the same way.

The endpoint /pipelines returns:

  • id (String): The name of the pipeline.
  • status (String): The status as either Running, or Error if there are any issues.
curl localhost:8080/pipelines
{"pipelines":[{"id":"yolo8demonstration","status":"Running"}]}

The following example uses the host workshop-yolo8n-x86-demo.eastus.cloudapp.azure.com. Replace with your own host name of your Edge deployed pipeline.

!curl workshop-yolo8n-x86-demo.eastus.cloudapp.azure.com:8080/pipelines
!curl testboy.local:8080/pipelines
{"pipelines":[{"id":"yolo8demonstration","status":"Running"}]}

The endpoint /models returns a List of models with the following fields:

  • name (String): The model name.
  • sha (String): The sha hash value of the ML model.
  • status (String): The status of either Running or Error if there are any issues.
  • version (String): The model version. This matches the version designation used by Wallaroo to track model versions in UUID format.
{"models":[{"name":"yolov8n","sha":"3ed5cd199e0e6e419bd3d474cf74f2e378aacbf586e40f24d1f8c89c2c476a08","status":"Running","version":"7af40d06-d18f-4b3f-9dd3-0a15248f01c8"}]}

The following example uses the host workshop-yolo8n-x86-demo.eastus.cloudapp.azure.com. Replace with your own host name of your Edge deployed pipeline.

!curl workshop-yolo8n-x86-demo.eastus.cloudapp.azure.com:8080/models
!curl testboy.local:8080/models
{"models":[{"name":"yolov8n","version":"38deda47-daf4-42d3-b4cf-fc78c2b7b35d","sha":"3ed5cd199e0e6e419bd3d474cf74f2e378aacbf586e40f24d1f8c89c2c476a08","status":"Running"}]}

Edge Inference Endpoint

The inference endpoint takes the following pattern:

  • /pipelines/{pipeline-name}: The pipeline-name is the same as returned from the /pipelines endpoint as id.

Wallaroo inference endpoint URLs accept the following data inputs through the Content-Type header:

  • Content-Type: application/vnd.apache.arrow.file: For Apache Arrow tables.
  • Content-Type: application/json; format=pandas-records: For pandas DataFrame in record format.

Once deployed, we can perform an inference through the deployment URL.

The endpoint returns Content-Type: application/json; format=pandas-records by default with the following fields:

  • check_failures (List[Integer]): Whether any validation checks were triggered. For more information, see Wallaroo SDK Essentials Guide: Pipeline Management: Anomaly Testing.
  • elapsed (List[Integer]): A list of time in nanoseconds for:
    • [0] The time to serialize the input.
    • [1…n] How long each step took.
  • model_name (String): The name of the model used.
  • model_version (String): The version of the model in UUID format.
  • original_data: The original input data. Returns null if the input may be too long for a proper return.
  • outputs (List): The outputs of the inference result separated by data type, where each data type includes:
    • data: The returned values.
    • dim (List[Integer]): The dimension shape returned.
    • v (Integer): The vector shape of the data.
  • pipeline_name (String): The name of the pipeline.
  • shadow_data: Any shadow deployed data inferences in the same format as outputs.
  • time (Integer): The time since UNIX epoch.

Once deployed, we can perform an inference through the deployment URL. We’ll assume we’re running the inference request through the localhost and submitting the local file ./data/dogbike.df.json. Note that our inference endpoint is pipelines/yolo8demonstration - the same as our pipeline name.

The following example demonstrates sending an inference request to the edge deployed pipeline and storing the results in a pandas DataFrame in record format. The results can then be exported to other processes to render the detected images or other use cases.

The following example uses the host workshop-yolo8n-x86-demo.eastus.cloudapp.azure.com. Replace with your own host name of your Edge deployed pipeline.

!curl -X POST workshop-yolo8n-x86-demo.eastus.cloudapp.azure.com:8080/pipelines/yolo-v8 \
    -H "Content-Type: application/json; format=pandas-records" \
    --data @./data/cv-yolo/dogbike.df.json
import datetime

start_inference = datetime.datetime.now()
!curl -X POST testboy.local:8080/pipelines/yolo8demonstration \
    -H "Content-Type: application/json; format=pandas-records" \
    --data @./data/cv-yolo/dogbike.df.json
[{"time":1701906109692,"in":{"images":[[[[...]]]]},"check_failures":[],"metadata":{"last_model":"{\"model_name\":\"yolov8n\",\"model_sha\":\"3ed5cd199e0e6e419bd3d474cf74f2e378aacbf586e40f24d1f8c89c2c476a08\"}","pipeline_version":"","elapsed":[315510995,56107409],"dropped":[],"partition":"yolo-edge-demo-jch2"}}]

Display Partition Logs

To view the edge deployed pipeline logs, we can use wallaroo.pipeline.export_logs method to retrieve all of the recent logs from this pipeline, and show the edge inference results were sent with the edge name in the partition metadata.

# display log information here with partition

pipeline.export_logs(directory='./logs/partition-edge-observability-yolo',
                     file_prefix='edge-logs',
                     dataset=['time', 'metadata'])

# display the partition only results

df_logs = pd.read_json('./logs/partition-edge-observability-yolo/edge-logs-1.json', 
                       orient="records", 
                       lines=True)
#filter out just the `metadata.partition='houseprice-edgebaseline-examples'

display(df_logs[df_logs['metadata.partition']==edge_name].loc[:, ['time', 'metadata.partition']])
# display(df_logs.loc[:, ['out.variable', 'metadata.partition']])
Warning: The inference log is above the allowable limit and the following columns may have been suppressed for various rows in the logs: ['in.images']. To review the dropped columns for an individual inference’s suppressed data, include dataset=["metadata"] in the log request.
timemetadata.partition
61701906109692yolo-edge-demo-jch2
81701905914859yolo-edge-demo-jch2

2.2 - Edge Deployment: Large Language Models (LLM) Summarization

Wallaroo Use Case Tutorials focused on Edge Deployments of Large Language Models (LLM) Summarization ML Models.

This workshop is available from the Wallaroo Workshop GitHub Repository.

Large Language Model Summarization Deployment in Wallaroo

The Hugging Face LLM Summarization model is a pre-trained model that condenses text into a shorter format. This tutorial demonstrates how to:

  • Deploy a Hugging Face LLM Summarization pre-trained model into a Wallaroo Ops server and perform inferences on it.
  • Publish the pipeline to the OCI registry configured in the Wallaroo Ops server.
  • Add an edge location to the Wallaroo pipeline publish.
  • Deploy the pipeline as a Wallaroo Server on an edge device through Docker, and display the inference logs submitted to the Wallaroo Ops server.

Wallaroo Ops Center provides the ability to publish Wallaroo pipelines to an Open Continer Initative (OCI) compliant registry, then deploy those pipelines on edge devices as Docker container or Kubernetes pods. See Wallaroo SDK Essentials Guide: Pipeline Edge Publication for full details.

This demonstration will focus on deployment to the edge. The sample model is available at the following URL. This model should be downloaded and placed into the ./models/llm-summarization folder before beginning this demonstration.

model-auto-conversion_hugging-face_complex-pipelines_hf-summarisation-bart-large-samsun.zip (1.4 GB)

References

  • Wallaroo Workspaces: Workspaces are environments were users upload models, create pipelines and other artifacts. The workspace should be considered the fundamental area where work is done. Workspaces are shared with other users to give them access to the same models, pipelines, etc.
  • Wallaroo Model Upload and Registration: ML Models are uploaded to Wallaroo through the SDK or the MLOps API to a workspace. ML models include default runtimes (ONNX, Python Step, and TensorFlow) that are run directly through the Wallaroo engine, and containerized runtimes (Hugging Face, PyTorch, etc) that are run through in a container through the Wallaroo engine.
  • Wallaroo Pipelines: Pipelines are used to deploy models for inferencing. Each model is a pipeline step in a pipelines, where the inputs of the previous step are fed into the next. Pipeline steps can be ML models, Python scripts, or Arbitrary Python (these contain necessary models and artifacts for running a model).
  • Wallaroo SDK Essentials Guide: Pipeline Edge Publication: Details on publishing a Wallaroo pipeline to an OCI Registry and deploying it as a Wallaroo Server instance.

Data Scientist Steps

The following details the steps a Data Scientist performs in uploading and verifying the model in a Wallaroo Ops server.

Load Libraries

The first step is loading the required libraries including the Wallaroo Python module.

# Import Wallaroo Python SDK
import wallaroo
from wallaroo.object import EntityNotFoundError
from wallaroo.framework import Framework

# used to display DataFrame information without truncating
from IPython.display import display
import pandas as pd
pd.set_option('display.max_colwidth', None)

import pyarrow as pa

Connect to the Wallaroo Instance through the User Interface

The next step is to connect to Wallaroo through the Wallaroo client. The Python library is included in the Wallaroo install and available through the Jupyter Hub interface provided with your Wallaroo environment.

This is accomplished using the wallaroo.Client() command, which provides a URL to grant the SDK permission to your specific Wallaroo environment. When displayed, enter the URL into a browser and confirm permissions. Store the connection into a variable that can be referenced later.

If logging into the Wallaroo instance through the internal JupyterHub service, use wl = wallaroo.Client(). For more information on Wallaroo Client settings, see the Client Connection guide.

wl = wallaroo.Client()

Create a New Workspace

We’ll use the SDK below to create our workspace , assign as our current workspace, then display all of the workspaces we have at the moment. We’ll also set up variables for our models and pipelines down the road, so we have one spot to change names to whatever fits your organization’s standards best.

To allow this tutorial to be run by multiple users in the same Wallaroo instance, update suffix with your first and last name. For example:

suffix = 'lazel-geth'

Create a New Workspace Exercise

Set the model name, file name, pipeline name, and workspace name.

Sample code:

suffix = ''

model_name = 'llm-summarization'
model_filename = './models/llm-summarization/model-auto-conversion_hugging-face_complex-pipelines_hf-summarisation-bart-large-samsun.zip'
pipeline_name = 'llm-edge-summarization'
workspace_name = f'llm-edge-summarization{suffix}'
suffix = ''

model_name = 'llm-summarization'
model_filename = './models/llm-summarization/model-auto-conversion_hugging-face_complex-pipelines_hf-summarisation-bart-large-samsun.zip'
pipeline_name = 'llm-edge-summarization'
workspace_name = f'llm-edge-summarization{suffix}'

Set the Current Workspace

Set the current workspace where the models are uploaded to and pipelines created.

Set the Current Workspace Exercise

Setting the workspace is performed with the wallaroo.client.set_current_workspace(workspace) method.

Sample code:

workspace = get_workspace(workspace_name, client)
wl.set_current_workspace(workspace)
def get_workspace(name):
    workspace = None
    for ws in wl.list_workspaces():
        if ws.name() == name:
            workspace= ws
    if(workspace == None):
        workspace = wl.create_workspace(name)
    return workspace

workspace = get_workspace(workspace_name)
wl.set_current_workspace(workspace)
{'name': 'llm-edge-summarization', 'id': 8, 'archived': False, 'created_by': 'a6e82da8-817d-4cca-bb62-5dbacd38ca22', 'created_at': '2023-12-05T20:43:07.604805+00:00', 'models': [{'name': 'llm-summarization', 'versions': 1, 'owner_id': '""', 'last_update_time': datetime.datetime(2023, 12, 5, 20, 44, 13, 14974, tzinfo=tzutc()), 'created_at': datetime.datetime(2023, 12, 5, 20, 44, 13, 14974, tzinfo=tzutc())}], 'pipelines': [{'name': 'llm-edge-summarization', 'create_time': datetime.datetime(2023, 12, 5, 20, 48, 36, 535526, tzinfo=tzutc()), 'definition': '[]'}]}

Upload the Model

When a model is uploaded to a Wallaroo cluster, it is optimized and packaged to make it ready to run as part of a pipeline. In many times, the Wallaroo Server can natively run a model without any Python overhead. In other cases, such as a Python script, a custom Python environment will be automatically generated. This is comparable to the process of “containerizing” a model by adding a small HTTP server and other wrapping around it.

Our pretrained model is in the Hugging Face framework, which is specified in the framework parameter. Hugging Face is a non-native runtime, which requires that the input and output schemas as specified during the model upload process.

Upload the Model Exercise

The model name and file name were set in the variables above. Use them to upload the model.

Sample code:

input_schema = pa.schema([
    pa.field('inputs', pa.string()),
    pa.field('return_text', pa.bool_()),
    pa.field('return_tensors', pa.bool_()),
    pa.field('clean_up_tokenization_spaces', pa.bool_()),
    # pa.field('generate_kwargs', pa.map_(pa.string(), pa.null())), # dictionaries are not currently supported by the engine
])

output_schema = pa.schema([
    pa.field('summary_text', pa.string()),
])

model = wl.upload_model(model_name, 
                        model_filename, 
                        framework=wallaroo.framework.Framework.HUGGING_FACE_SUMMARIZATION, 
                        input_schema=input_schema, 
                        output_schema=output_schema
                        )
# Upload Retrained LLM Summarization Model 

input_schema = pa.schema([
    pa.field('inputs', pa.string()),
    pa.field('return_text', pa.bool_()),
    pa.field('return_tensors', pa.bool_()),
    pa.field('clean_up_tokenization_spaces', pa.bool_()),
    # pa.field('generate_kwargs', pa.map_(pa.string(), pa.null())), # dictionaries are not currently supported by the engine
])

output_schema = pa.schema([
    pa.field('summary_text', pa.string()),
])

model = wl.upload_model(model_name, 
                        model_filename, 
                        framework=wallaroo.framework.Framework.HUGGING_FACE_SUMMARIZATION, 
                        input_schema=input_schema, 
                        output_schema=output_schema
                        )
Waiting for model loading - this will take up to 10.0min.
Model is pending loading to a container runtime..
Model is attempting loading to a container runtime.............................................successful

Ready

Pipeline Deployment Configuration

For our pipeline we set the deployment configuration to set the resources the pipeline will be allocated from the Kubernetes cluster hosting the Wallaroo Ops instance. The Hugging Face model is deployed as a Containerized Runtime in Wallaroo, so the configuration specified the sidekick cpu and memory options.

Pipeline Deployment Configuration Exercise

Use the deployment configuration below.

deployment_config = wallaroo.DeploymentConfigBuilder() \
    .cpus(0.25).memory('1Gi') \
    .sidekick_cpus(model, 4) \
    .sidekick_memory(model, "8Gi") \
    .build()

Build and Deploy the Pipeline

Now we build our pipeline and set our Yolo8 model as a pipeline step, then deploy the pipeline using the deployment configuration above.

Build and Deploy the Pipeline Exercise

We’ll do both commands in one step:

  • Build the pipeline with wallaroo.client.build_pipeline.
  • Set the model as a pipeline step with wallaroo.pipeline.add_model_step(model) method.

Sample code:

pipeline = wl.build_pipeline(pipeline_name) \
            .add_model_step(model)    
pipeline = wl.build_pipeline(pipeline_name) \
            .add_model_step(model)        

Deploy the Pipeline

We deploy the pipeline with the wallaroo.pipeline.deploy(deployment_config) command, using the deployment configuration set up in previous steps.

Deploy the Pipeline Exercise

Deploy the pipeline.

Sample code:

pipeline.deploy(deployment_config=deployment_config)
pipeline.deploy(deployment_config=deployment_config)
namellm-edge-summarization
created2023-12-05 20:48:36.535526+00:00
last_updated2023-12-05 20:48:37.172602+00:00
deployedTrue
archNone
tags
versions040afe0c-9a17-4f0b-8cb2-68e8dc85d9a4, 6f1db20f-e1b5-47a5-a6b4-c6a31f9c1023
stepsllm-summarization
publishedFalse

Inference Request

We submit the DataFrame to the pipeline using wallaroo.pipeline.infer and display the results. We’ll use both the Wallaroo SDK and the MLOps API.

Inference Request Exercise

Perform an inference request. We’ll generate our sample dataframe, then use it for the inference.

Sample Code:

# sample dataframe input

input_data = {
        "inputs": ["LinkedIn (/lɪŋktˈɪn/) is a business and employment-focused social media platform that works through websites and mobile apps. It launched on May 5, 2003. It is now owned by Microsoft. The platform is primarily used for professional networking and career development, and allows jobseekers to post their CVs and employers to post jobs. From 2015 most of the company's revenue came from selling access to information about its members to recruiters and sales professionals. Since December 2016, it has been a wholly owned subsidiary of Microsoft. As of March 2023, LinkedIn has more than 900 million registered members from over 200 countries and territories. LinkedIn allows members (both workers and employers) to create profiles and connect with each other in an online social network which may represent real-world professional relationships. Members can invite anyone (whether an existing member or not) to become a connection. LinkedIn can also be used to organize offline events, join groups, write articles, publish job postings, post photos and videos, and more"], # required
        "return_text": [True], # optional: using the defaults, similar to not passing this parameter
        "return_tensors": [False], # optional: using the defaults, similar to not passing this parameter
        "clean_up_tokenization_spaces": [False], # optional: using the defaults, similar to not passing this parameter
}
dataframe = pd.DataFrame(input_data)
dataframe
# sample dataframe input

input_data = {
        "inputs": ["LinkedIn (/lɪŋktˈɪn/) is a business and employment-focused social media platform that works through websites and mobile apps. It launched on May 5, 2003. It is now owned by Microsoft. The platform is primarily used for professional networking and career development, and allows jobseekers to post their CVs and employers to post jobs. From 2015 most of the company's revenue came from selling access to information about its members to recruiters and sales professionals. Since December 2016, it has been a wholly owned subsidiary of Microsoft. As of March 2023, LinkedIn has more than 900 million registered members from over 200 countries and territories. LinkedIn allows members (both workers and employers) to create profiles and connect with each other in an online social network which may represent real-world professional relationships. Members can invite anyone (whether an existing member or not) to become a connection. LinkedIn can also be used to organize offline events, join groups, write articles, publish job postings, post photos and videos, and more"], # required
        "return_text": [True], # optional: using the defaults, similar to not passing this parameter
        "return_tensors": [False], # optional: using the defaults, similar to not passing this parameter
        "clean_up_tokenization_spaces": [False], # optional: using the defaults, similar to not passing this parameter
}
dataframe = pd.DataFrame(input_data)
dataframe
inputsreturn_textreturn_tensorsclean_up_tokenization_spaces
0LinkedIn (/lɪŋktˈɪn/) is a business and employment-focused social media platform that works through websites and mobile apps. It launched on May 5, 2003. It is now owned by Microsoft. The platform is primarily used for professional networking and career development, and allows jobseekers to post their CVs and employers to post jobs. From 2015 most of the company's revenue came from selling access to information about its members to recruiters and sales professionals. Since December 2016, it has been a wholly owned subsidiary of Microsoft. As of March 2023, LinkedIn has more than 900 million registered members from over 200 countries and territories. LinkedIn allows members (both workers and employers) to create profiles and connect with each other in an online social network which may represent real-world professional relationships. Members can invite anyone (whether an existing member or not) to become a connection. LinkedIn can also be used to organize offline events, join groups, write articles, publish job postings, post photos and videos, and moreTrueFalseFalse

Use the Pipeline Deployment URL for the inference request, and submit the sample DataFrame as our input.

Sample code:

!curl {pipeline._deployment._url()} \
    -H "Content-Type: application/json; format=pandas-records" \
    -H "Authorization: {wl.auth.auth_header()['Authorization']}" \
    -H "Accept:{headers['Accept']}" \
     --data @./data/llm-summarization/test_summarization.df.json
# inference request here

!curl {pipeline._deployment._url()} \
    -H "Content-Type: application/json; format=pandas-records" \
    -H "Authorization: {wl.auth.auth_header()['Authorization']}" \
    -H "Accept:{headers['Accept']}" \
     --data @./data/llm-summarization/test_summarization.df.json

Undeploy the Pipeline

With the testing complete, we undeploy the pipeline and return the resources back to the cluster.

Undeploy the Pipeline Exercise

Sample code:

pipeline.undeploy()
# undeploy the pipeline

pipeline.undeploy()
namellm-edge-summarization
created2023-12-05 20:48:36.535526+00:00
last_updated2023-12-05 20:48:37.172602+00:00
deployedFalse
archNone
tags
versions040afe0c-9a17-4f0b-8cb2-68e8dc85d9a4, 6f1db20f-e1b5-47a5-a6b4-c6a31f9c1023
stepsllm-summarization
publishedFalse

Publish the Pipeline for Edge Deployment

It worked! For a demo, we’ll take working once as “tested”. So now that we’ve tested our pipeline, we are ready to publish it for edge deployment.

Publishing it means assembling all of the configuration files and model assets and pushing them to an Open Container Initiative (OCI) repository set in the Wallaroo instance as the Edge Registry service. DevOps engineers then retrieve that image and deploy it through Docker, Kubernetes, or similar deployments.

See Edge Deployment Registry Guide for details on adding an OCI Registry Service to Wallaroo as the Edge Deployment Registry.

This is done through the SDK command wallaroo.pipeline.publish(deployment_config) which has the following parameters and returns.

Publish a Pipeline Parameters

The publish method takes the following parameters. The containerized pipeline will be pushed to the Edge registry service with the model, pipeline configurations, and other artifacts needed to deploy the pipeline.

ParameterTypeDescription
deployment_configwallaroo.deployment_config.DeploymentConfig (Optional)Sets the pipeline deployment configuration. For example: For more information on pipeline deployment configuration, see the Wallaroo SDK Essentials Guide: Pipeline Deployment Configuration.

Publish a Pipeline Returns

FieldTypeDescription
idintegerNumerical Wallaroo id of the published pipeline.
pipeline version idintegerNumerical Wallaroo id of the pipeline version published.
statusstringThe status of the pipeline publication. Values include:
  • PendingPublish: The pipeline publication is about to be uploaded or is in the process of being uploaded.
  • Published: The pipeline is published and ready for use.
Engine URLstringThe URL of the published pipeline engine in the edge registry.
Pipeline URLstringThe URL of the published pipeline in the edge registry.
Helm Chart URLstringThe URL of the helm chart for the published pipeline in the edge registry.
Helm Chart ReferencestringThe help chart reference.
Helm Chart VersionstringThe version of the Helm Chart of the published pipeline. This is also used as the Docker tag.
Engine Configwallaroo.deployment_config.DeploymentConfigThe pipeline configuration included with the published pipeline.
Created AtDateTimeWhen the published pipeline was created.
Updated AtDateTimeWhen the published pipeline was updated.

Publish Exercise

We will now publish the pipeline to our Edge Deployment Registry with the pipeline.publish(deployment_config) command. deployment_config is an optional field that specifies the pipeline deployment. This can be overridden by the DevOps engineer during deployment.

Save the publish to a variable for later use.

Sample code:

pub = pipeline.publish(deployment_config)
pub
pub = pipeline.publish(deployment_config)
pub
Waiting for pipeline publish... It may take up to 600 sec.
Pipeline is Publishing................Published.
ID1
Pipeline Version1c8d0586-f3ff-453c-82a9-14602830f97f
StatusPublished
Engine URLghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/standalone-mini:v2023.4.0-4103
Pipeline URLghcr.io/wallaroolabs/doc-samples/pipelines/llm-edge-summarization:1c8d0586-f3ff-453c-82a9-14602830f97f
Helm Chart URLoci://ghcr.io/wallaroolabs/doc-samples/charts/llm-edge-summarization
Helm Chart Referenceghcr.io/wallaroolabs/doc-samples/charts@sha256:0a0c5b2df650b33fe4457a6eb7b2701caea19d0c81f97c0fe6345b088baaac41
Helm Chart Version0.0.1-1c8d0586-f3ff-453c-82a9-14602830f97f
Engine Config{'engine': {'resources': {'limits': {'cpu': 1.0, 'memory': '512Mi'}, 'requests': {'cpu': 1.0, 'memory': '512Mi'}}}, 'engineAux': {'images': {}}, 'enginelb': {'resources': {'limits': {'cpu': 1.0, 'memory': '512Mi'}, 'requests': {'cpu': 1.0, 'memory': '512Mi'}}}}
User Images[]
Created Byjohn.hummel@wallaroo.ai
Created At2023-12-05 20:50:16.153199+00:00
Updated At2023-12-05 20:50:16.153199+00:00
Docker Run Variables{}

List Published Pipeline

The method wallaroo.client.list_pipelines() shows a list of all pipelines in the Wallaroo instance, and includes the published field that indicates whether the pipeline was published to the registry (True), or has not yet been published (False).

List Published Pipeline Exercise

List the pipelines and verify which ones are published or not.

Sample code:

wl.list_pipelines()
wl.list_pipelines()
namecreatedlast_updateddeployedarchtagsversionsstepspublished
llm-edge-summarization2023-05-Dec 20:48:362023-05-Dec 20:50:14FalseNone1c8d0586-f3ff-453c-82a9-14602830f97f, 040afe0c-9a17-4f0b-8cb2-68e8dc85d9a4, 6f1db20f-e1b5-47a5-a6b4-c6a31f9c1023llm-summarizationTrue

List Publishes from a Pipeline

All publishes created from a pipeline are displayed with the wallaroo.pipeline.publishes method. The pipeline_version_id is used to know what version of the pipeline was used in that specific publish. This allows for pipelines to be updated over time, and newer versions to be sent and tracked to the Edge Deployment Registry service.

List Publishes Parameters

N/A

List Publishes Returns

A List of the following fields:

FieldTypeDescription
idintegerNumerical Wallaroo id of the published pipeline.
pipeline_version_idintegerNumerical Wallaroo id of the pipeline version published.
engine_urlstringThe URL of the published pipeline engine in the edge registry.
pipeline_urlstringThe URL of the published pipeline in the edge registry.
created_bystringThe email address of the user that published the pipeline.
Created AtDateTimeWhen the published pipeline was created.
Updated AtDateTimeWhen the published pipeline was updated.

List Publishes from a Pipeline Exercise

List the publishes from a pipeline.

Sample code:

pipeline.publishes()
pipeline.publishes()
idpipeline_version_nameengine_urlpipeline_urlcreated_bycreated_atupdated_at
11c8d0586-f3ff-453c-82a9-14602830f97fghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/standalone-mini:v2023.4.0-4103ghcr.io/wallaroolabs/doc-samples/pipelines/llm-edge-summarization:1c8d0586-f3ff-453c-82a9-14602830f97fjohn.hummel@wallaroo.ai2023-05-Dec 20:50:162023-05-Dec 20:50:16

Add Edge Location

With the pipeline publish created, we can add an Edge Location. This allows the edge deployment to upload its inference results back to the Wallaroo Ops location, which are then added to the pipeline the publish originated from. These are added to the pipeline logs partition metadata.

First we’ll retrieve the pipeline logs for our current pipeline, and show the current pipeline logs metadata.

For the edge name, set it to firstname-lastname-edge-yolo.

Add Edge Location Exercise

Display the log information with the metadata.partition, then add the edge location to the publish. Note that edge names must be unique, so add your first and last name to the list.

Sample code:

logs = pipeline.logs(dataset=['time', 'out.output0', 'metadata'])
display(logs.loc[:, ['time', 'metadata.partition']])

first_last_name = '-Gale-Karlach'

edge_name = f'hf-summarizer-edge-demo{first_last_name}'

edge_publish = pub.add_edge(edge_name)
display(edge_publish)
logs = pipeline.logs(dataset=['time', 'out.summary_text', 'metadata'])
display(logs.loc[:, ['time', 'out.summary_text', 'metadata.partition']])
timeout.summary_textmetadata.partition
02023-12-05 20:49:24.105LinkedIn is a business and employment-focused social media platform that works through websites and mobile apps. It launched on May 5, 2003. LinkedIn allows members (both workers and employers) to create profiles and connect with each other in an online social network which may represent real-world professional relationships.engine-66f69d685-4d4s2
edge_name = 'edge-summarization-demo-arm'

edge_publish = pub.add_edge(edge_name)
display(edge_publish)
ID1
Pipeline Version1c8d0586-f3ff-453c-82a9-14602830f97f
StatusPublished
Engine URLghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/standalone-mini:v2023.4.0-4103
Pipeline URLghcr.io/wallaroolabs/doc-samples/pipelines/llm-edge-summarization:1c8d0586-f3ff-453c-82a9-14602830f97f
Helm Chart URLoci://ghcr.io/wallaroolabs/doc-samples/charts/llm-edge-summarization
Helm Chart Referenceghcr.io/wallaroolabs/doc-samples/charts@sha256:0a0c5b2df650b33fe4457a6eb7b2701caea19d0c81f97c0fe6345b088baaac41
Helm Chart Version0.0.1-1c8d0586-f3ff-453c-82a9-14602830f97f
Engine Config{'engine': {'resources': {'limits': {'cpu': 1.0, 'memory': '512Mi'}, 'requests': {'cpu': 1.0, 'memory': '512Mi'}}}, 'engineAux': {'images': {}}, 'enginelb': {'resources': {'limits': {'cpu': 1.0, 'memory': '512Mi'}, 'requests': {'cpu': 1.0, 'memory': '512Mi'}}}}
User Images[]
Created Byjohn.hummel@wallaroo.ai
Created At2023-12-05 20:50:16.153199+00:00
Updated At2023-12-05 20:50:16.153199+00:00
Docker Run Variables{'EDGE_BUNDLE': 'abcde'}

DevOps - Pipeline Edge Deployment

Once a pipeline is deployed to the Edge Registry service, it can be deployed in environments such as Docker, Kubernetes, or similar container running services by a DevOps engineer.

Docker Deployment

First, the DevOps engineer must authenticate to the same OCI Registry service used for the Wallaroo Edge Deployment registry.

For more details, check with the documentation on your artifact service. The following are provided for the three major cloud services:

For the deployment, the engine URL is specified with the following environmental variables:

  • DEBUG (true|false): Whether to include debug output.
  • OCI_REGISTRY: The URL of the registry service.
  • CONFIG_CPUS: The number of CPUs to use.
  • OCI_USERNAME: The edge registry username.
  • OCI_PASSWORD: The edge registry password or token.
  • PIPELINE_URL: The published pipeline URL.

Docker Deployment Example

Using our sample environment, here’s sample deployment using Docker with a computer vision ML model, the same used in the Wallaroo Use Case Tutorials Computer Vision: Retail tutorials.

Note the use of the -v ./data:/persist option. This will store the one time authentication token stored in the EDGE_BUNDLE

mkdir ./data

docker run -p 8080:8080 \
    -v ./data:/persist \
    -e DEBUG=true -e OCI_REGISTRY={your registry server} \
    -e EDGE_BUNDLE={edge_publish.docker_run_variables['EDGE_BUNDLE']} \
    -e CONFIG_CPUS=4 \
    -e OCI_USERNAME=oauth2accesstoken \
    -e OCI_PASSWORD={registry token here} \
    -e PIPELINE_URL={your registry server}/pipelines/yolo8demonstration:bf70eaf7-8c11-4b46-b751-916a43b1a555 \
    {your registry server}/engine:v2023.3.0-main-3707

Docker Compose Deployment

For users who prefer to use docker compose, the following sample compose.yaml file is used to launch the Wallaroo Edge pipeline. This is the same used in the Wallaroo Use Case Tutorials Computer Vision: Retail tutorials.

The volumes settings allows for persistent volumes to store the session information. Without it, the one-time authentication token included in the EDGE_BUNDLE settings would have to be regenerated.

services:
  engine:
    image: {Your Engine URL}
    volumes:
      - ./data:/persist
    ports:
      - 8080:8080
    environment:
      EDGE_BUNDLE: abcdefg
      PIPELINE_URL: {Your Pipeline URL}
      OCI_REGISTRY: {Your Edge Registry URL}
      OCI_USERNAME:  {Your Registry Username}
      OCI_PASSWORD: {Your Token or Password}
      CONFIG_CPUS: 4

For example:

services:
  engine:
    image: sample-registry.com/engine:v2023.3.0-main-3707
    ports:
      - 8080:8080
    environment:
      PIPELINE_URL: sample-registry.com/pipelines/yolo8demonstration:bf70eaf7-8c11-4b46-b751-916a43b1a555
      OCI_REGISTRY: sample-registry.com
      OCI_USERNAME:  _json_key_base64
      OCI_PASSWORD: abc123
      CONFIG_CPUS: 4

Docker Compose Deployment Example

The deployment and undeployment is then just a simple docker compose up and docker compose down. The following shows an example of deploying the Wallaroo edge pipeline using docker compose.

docker compose up
[+] Running 1/1
 ✔ Container yolo8demonstration-engine-1  Recreated                                                                                                                                                                 0.5s
Attaching to yolo8demonstration-engine-1
yolo8demonstration-engine-1  | Wallaroo Engine - Standalone mode
yolo8demonstration-engine-1  | Login Succeeded
yolo8demonstration-engine-1  | Fetching manifest and config for pipeline: sample-registry.com/pipelines/yolo8demonstration:bf70eaf7-8c11-4b46-b751-916a43b1a555
yolo8demonstration-engine-1  | Fetching model layers
yolo8demonstration-engine-1  | digest: sha256:c6c8869645962e7711132a7e17aced2ac0f60dcdc2c7faa79b2de73847a87984
yolo8demonstration-engine-1  |   filename: c6c8869645962e7711132a7e17aced2ac0f60dcdc2c7faa79b2de73847a87984
yolo8demonstration-engine-1  |   name: yolov8n
yolo8demonstration-engine-1  |   type: model
yolo8demonstration-engine-1  |   runtime: onnx
yolo8demonstration-engine-1  |   version: 693e19b5-0dc7-4afb-9922-e3f7feefe66d
yolo8demonstration-engine-1  |
yolo8demonstration-engine-1  | Fetched
yolo8demonstration-engine-1  | Starting engine
yolo8demonstration-engine-1  | Looking for preexisting `yaml` files in //modelconfigs
yolo8demonstration-engine-1  | Looking for preexisting `yaml` files in //pipelines

Helm Deployment

Published pipelines can be deployed through the use of helm charts.

Helm deployments take up to two steps - the first step is in retrieving the required values.yaml and making updates to override.

Kubernetes provides persistent volume support, so no settings are required.

  1. Pull the helm charts from the published pipeline. The two fields are the Helm Chart URL and the Helm Chart version to specify the OCI . This typically takes the format of:
helm pull oci://{published.helm_chart_url} --version {published.helm_chart_version}
  1. Extract the tgz file and copy the values.yaml and copy the values used to edit engine allocations, etc. The following are required for the deployment to run:
ociRegistry:
  registry: {your registry service}
  username:  {registry username here}
  password: {registry token here}

Store this into another file, suc as local-values.yaml.

  1. Create the namespace to deploy the pipeline to. For example, the namespace wallaroo-edge-pipeline would be:
kubectl create -n wallaroo-edge-pipeline
  1. Deploy the helm installation with helm install through one of the following options:

    1. Specify the tgz file that was downloaded and the local values file. For example:

      helm install --namespace {namespace} --values {local values file} {helm install name} {tgz path}
      
    2. Specify the expended directory from the downloaded tgz file.

      helm install --namespace {namespace} --values {local values file} {helm install name} {helm directory path}
      
    3. Specify the Helm Pipeline Helm Chart and the Pipeline Helm Version.

      helm install --namespace {namespace} --values {local values file} {helm install name} oci://{published.helm_chart_url} --version {published.helm_chart_version}
      
  2. Once deployed, the DevOps engineer will have to forward the appropriate ports to the svc/engine-svc service in the specific pipeline. For example, using kubectl port-forward to the namespace ccfraud that would be:

    kubectl port-forward svc/engine-svc -n ccfraud01 8080 --address 0.0.0.0`
    

Docker Deployment Code Generation Exercise

The following code segment generates a docker run template based on the previously published pipeline. Replace the $REGISTRYURL, $REGISTRYUSERNAME, and $REGISTRYPASSWORD to match the OCI Registry being used.

docker_deploy = f'''
docker run -p 8080:8080 \\
    -v ./data:/persist \\
    -e DEBUG=true \\
    -e OCI_REGISTRY=$REGISTRYURL \\
    -e EDGE_BUNDLE={edge_publish.docker_run_variables['EDGE_BUNDLE']} \\
    -e CONFIG_CPUS=1 \\
    -e OCI_USERNAME=$REGISTRYUSERNAME \\
    -e OCI_PASSWORD=$REGISTRYPASSWORD \\
    -e PIPELINE_URL={edge_publish.pipeline_url} \\
    {edge_publish.engine_url}
'''

print(docker_deploy)
docker run -p 8080:8080 \
    -v ./data:/persist \
    -e DEBUG=true \
    -e OCI_REGISTRY=$REGISTRYURL \
    -e EDGE_BUNDLE=ZXhwb3J0IEJVTkRMRV9WRVJTSU9OPTEKZXhwb3J0IEVER0VfTkFNRT1lZGdlLXN1bW1hcml6YXRpb24tZGVtby1hcm0KZXhwb3J0IEpPSU5fVE9LRU49MTlkMGE4NzAtNTgyYy00YWExLWE5YjAtMWQxMTQ4MWI4MWRjCmV4cG9ydCBPUFNDRU5URVJfSE9TVD1kb2MtdGVzdC5lZGdlLndhbGxhcm9vY29tbXVuaXR5Lm5pbmphCmV4cG9ydCBQSVBFTElORV9VUkw9Z2hjci5pby93YWxsYXJvb2xhYnMvZG9jLXNhbXBsZXMvcGlwZWxpbmVzL2xsbS1lZGdlLXN1bW1hcml6YXRpb246MWM4ZDA1ODYtZjNmZi00NTNjLTgyYTktMTQ2MDI4MzBmOTdmCmV4cG9ydCBXT1JLU1BBQ0VfSUQ9OA== \
    -e CONFIG_CPUS=1 \
    -e OCI_USERNAME=$REGISTRYUSERNAME \
    -e OCI_PASSWORD=$REGISTRYPASSWORD \
    -e PIPELINE_URL=ghcr.io/wallaroolabs/doc-samples/pipelines/llm-edge-summarization:1c8d0586-f3ff-453c-82a9-14602830f97f \
    ghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/standalone-mini:v2023.4.0-4103

Edge Deployed Pipeline API Endpoints

Once deployed, we can check the pipelines and models available. We’ll use a curl command, but any HTTP based request will work the same way.

The endpoint /pipelines returns:

  • id (String): The name of the pipeline.
  • status (String): The status as either Running, or Error if there are any issues.
curl localhost:8080/pipelines
{"pipelines":[{"id":"yolo8demonstration","status":"Running"}]}

The following example uses the host localhost. Replace with your own host name of your Edge deployed pipeline.

!curl workshop-hf-summarizer-demo.eastus.cloudapp.azure.com:8080/pipelines
{"pipelines":[{"id":"hf-summarizer-standard","status":"Running"}]}

The endpoint /models returns a List of models with the following fields:

  • name (String): The model name.
  • sha (String): The sha hash value of the ML model.
  • status (String): The status of either Running or Error if there are any issues.
  • version (String): The model version. This matches the version designation used by Wallaroo to track model versions in UUID format.
{"models":[{"name":"yolov8n","sha":"3ed5cd199e0e6e419bd3d474cf74f2e378aacbf586e40f24d1f8c89c2c476a08","status":"Running","version":"7af40d06-d18f-4b3f-9dd3-0a15248f01c8"}]}

The following example uses the host localhost. Replace with your own host name of your Edge deployed pipeline.

!curl workshop-hf-summarizer-demo.eastus.cloudapp.azure.com:8080/models
{"models":[{"name":"hf-summarizer-standard","sha":"ee71d066a83708e7ca4a3c07caf33fdc528bb000039b6ca2ef77fa2428dc6268","status":"Running","version":"7dbae7b4-20d0-40f7-a3f5-eeabdd77f418"}]}

Edge Inference Endpoint

The inference endpoint takes the following pattern:

  • /pipelines/{pipeline-name}: The pipeline-name is the same as returned from the /pipelines endpoint as id.

Wallaroo inference endpoint URLs accept the following data inputs through the Content-Type header:

  • Content-Type: application/vnd.apache.arrow.file: For Apache Arrow tables.
  • Content-Type: application/json; format=pandas-records: For pandas DataFrame in record format.

Once deployed, we can perform an inference through the deployment URL.

The endpoint returns Content-Type: application/json; format=pandas-records by default with the following fields:

  • check_failures (List[Integer]): Whether any validation checks were triggered. For more information, see Wallaroo SDK Essentials Guide: Pipeline Management: Anomaly Testing.
  • elapsed (List[Integer]): A list of time in nanoseconds for:
    • [0] The time to serialize the input.
    • [1…n] How long each step took.
  • model_name (String): The name of the model used.
  • model_version (String): The version of the model in UUID format.
  • original_data: The original input data. Returns null if the input may be too long for a proper return.
  • outputs (List): The outputs of the inference result separated by data type, where each data type includes:
    • data: The returned values.
    • dim (List[Integer]): The dimension shape returned.
    • v (Integer): The vector shape of the data.
  • pipeline_name (String): The name of the pipeline.
  • shadow_data: Any shadow deployed data inferences in the same format as outputs.
  • time (Integer): The time since UNIX epoch.

Once deployed, we can perform an inference through the deployment URL. We’ll assume we’re running the inference request through the localhost and submitting the local file ./data/dogbike.df.json. Note that our inference endpoint is pipelines/yolo8demonstration - the same as our pipeline name.

The following example demonstrates sending an inference request to the edge deployed pipeline and storing the results in a pandas DataFrame in record format. The results can then be exported to other processes to render the detected images or other use cases.

!curl workshop-hf-summarizer-demo.eastus.cloudapp.azure.com:8080/pipelines/hf-summarizer-standard \
    -H "Content-Type: application/json; format=pandas-records" \
    --data @./data/llm-summarization/test_summarization.df.json
[{"check_failures":[],"elapsed":[310666674,4294967295],"model_name":"hf-summarizer-standard","model_version":"7dbae7b4-20d0-40f7-a3f5-eeabdd77f418","original_data":null,"outputs":[{"String":{"data":["LinkedIn is a business and employment-focused social media platform that works through websites and mobile apps. It launched on May 5, 2003. LinkedIn allows members (both workers and employers) to create profiles and connect with each other in an online social network which may represent real-world professional relationships."],"dim":[1,1],"v":1}}],"pipeline_name":"hf-summarizer-standard","shadow_data":{},"time":1701815846451}]

2.3 - Edge Deployment: Forecast

Wallaroo Use Case Tutorials focused on Edge Deployments of Forecast ML Models.

Forecast Retail Deployment in Wallaroo

This tutorial demonstrates how to:

  • Deploy a Forecast Python trained model into a Wallaroo Ops server and perform inferences on it.
  • Publish the pipeline to the OCI registry configured in the Wallaroo Ops server.
  • Add an edge location to the Wallaroo pipeline publish.
  • Deploy the pipeline as a Wallaroo Server on an edge device through Docker, and display the inference logs submitted to the Wallaroo Ops server.

Wallaroo Ops Center provides the ability to publish Wallaroo pipelines to an Open Continer Initative (OCI) compliant registry, then deploy those pipelines on edge devices as Docker container or Kubernetes pods. See Wallaroo SDK Essentials Guide: Pipeline Edge Publication for full details.

This demonstration will focus on deployment to the edge.

References

  • Wallaroo Workspaces: Workspaces are environments were users upload models, create pipelines and other artifacts. The workspace should be considered the fundamental area where work is done. Workspaces are shared with other users to give them access to the same models, pipelines, etc.
  • Wallaroo Model Upload and Registration: ML Models are uploaded to Wallaroo through the SDK or the MLOps API to a workspace. ML models include default runtimes (ONNX, Python Step, and TensorFlow) that are run directly through the Wallaroo engine, and containerized runtimes (Hugging Face, PyTorch, etc) that are run through in a container through the Wallaroo engine.
  • Wallaroo Pipelines: Pipelines are used to deploy models for inferencing. Each model is a pipeline step in a pipelines, where the inputs of the previous step are fed into the next. Pipeline steps can be ML models, Python scripts, or Arbitrary Python (these contain necessary models and artifacts for running a model).
  • Wallaroo SDK Essentials Guide: Pipeline Edge Publication: Details on publishing a Wallaroo pipeline to an OCI Registry and deploying it as a Wallaroo Server instance.

Data Scientist Steps

The following details the steps a Data Scientist performs in uploading and verifying the model in a Wallaroo Ops server.

Load Libraries

The first step is loading the required libraries including the Wallaroo Python module.

# Import Wallaroo Python SDK
import wallaroo
from wallaroo.object import EntityNotFoundError
from wallaroo.framework import Framework

# used to display DataFrame information without truncating
from IPython.display import display
import pandas as pd
pd.set_option('display.max_colwidth', None)

import pyarrow as pa

Connect to the Wallaroo Instance through the User Interface

The next step is to connect to Wallaroo through the Wallaroo client. The Python library is included in the Wallaroo install and available through the Jupyter Hub interface provided with your Wallaroo environment.

This is accomplished using the wallaroo.Client() command, which provides a URL to grant the SDK permission to your specific Wallaroo environment. When displayed, enter the URL into a browser and confirm permissions. Store the connection into a variable that can be referenced later.

If logging into the Wallaroo instance through the internal JupyterHub service, use wl = wallaroo.Client(). For more information on Wallaroo Client settings, see the Client Connection guide.

Connect to the Wallaroo Instance Exercise

Connect to the Wallaroo instance. If connecting through the JupyterHub service, then only the wallaroo.Client() is required. If connecting externally through the Wallaroo SDK, use the wallaroo.client(api_endpoint, auth_endpoint) method.

Sample code:

wl = wallaroo.Client()
# connect to Wallaroo here

wl = wallaroo.Client()

Create a New Workspace

We’ll use the SDK below to create our workspace , assign as our current workspace, then display all of the workspaces we have at the moment. We’ll also set up variables for our models and pipelines down the road, so we have one spot to change names to whatever fits your organization’s standards best.

To allow this tutorial to be run by multiple users in the same Wallaroo instance, update suffix with your first and last name. For example:

suffix = 'lazel-geth'

Create a New Workspace Exercise

Set the model name, file name, pipeline name, and workspace name.

Sample code:

suffix = ''

model_name = 'retail-forecast'
model_filename = './models/forecast/forecast_standard.py'
pipeline_name = 'retail-forecast'
workspace_name = f'retail-forecast-edge-demo{suffix}'
# set variables

suffix = ''

model_name = 'retail-forecast'
model_filename = './models/forecast/forecast_standard.py'
pipeline_name = 'retail-forecast'
workspace_name = f'retail-forecast-edge-demo{suffix}'

Set the Current Workspace

Set the current workspace where the models are uploaded to and pipelines created.

Set the Current Workspace Exercise

Setting the workspace is performed with the wallaroo.client.set_current_workspace(workspace) method.

Sample code:

workspace = get_workspace(workspace_name, client)
wl.set_current_workspace(workspace)
def get_workspace(name):
    workspace = None
    for ws in wl.list_workspaces():
        if ws.name() == name:
            workspace= ws
    if(workspace == None):
        workspace = wl.create_workspace(name)
    return workspace

workspace = get_workspace(workspace_name)
wl.set_current_workspace(workspace)
{'name': 'retail-forecast-edge-demo', 'id': 9, 'archived': False, 'created_by': 'a6e82da8-817d-4cca-bb62-5dbacd38ca22', 'created_at': '2023-12-05T23:12:54.354351+00:00', 'models': [{'name': 'forecast-control-model', 'versions': 1, 'owner_id': '""', 'last_update_time': datetime.datetime(2023, 12, 5, 23, 12, 57, 994250, tzinfo=tzutc()), 'created_at': datetime.datetime(2023, 12, 5, 23, 12, 57, 994250, tzinfo=tzutc())}], 'pipelines': [{'name': 'retail-forecast', 'create_time': datetime.datetime(2023, 12, 5, 23, 13, 1, 779624, tzinfo=tzutc()), 'definition': '[]'}]}

Upload the Model

When a model is uploaded to a Wallaroo cluster, it is optimized and packaged to make it ready to run as part of a pipeline. In many times, the Wallaroo Server can natively run a model without any Python overhead. In other cases, such as a Python script, a custom Python environment will be automatically generated. This is comparable to the process of “containerizing” a model by adding a small HTTP server and other wrapping around it.

Our pretrained model is a Python script, which is specified in the framework parameter. To properly receive and return inference results, we specify the input and output schemas in Apache Arrow format.

Upload the Model Exercise

The model name and file name were set in the variables above. Use them to upload the model.

Sample code:

# set the input and output schemas

input_schema = pa.schema([
    pa.field('count', pa.list_(pa.int64()))
])

output_schema = pa.schema([
    pa.field('forecast', pa.list_(pa.int64())),
    pa.field('weekly_average', pa.list_(pa.float64()))
])

# upload the models

model_version = wl.upload_model('forecast-control-model', 
                './models/forecast/forecast_standard.py', 
                framework=Framework.PYTHON).configure(
                "python", 
                input_schema=input_schema, 
                output_schema=output_schema
                )
# Upload forecasting model

# set the input and output schemas

input_schema = pa.schema([
    pa.field('count', pa.list_(pa.int64()))
])

output_schema = pa.schema([
    pa.field('forecast', pa.list_(pa.int64())),
    pa.field('weekly_average', pa.list_(pa.float64()))
])

# upload the models

model_version = wl.upload_model('forecast-control-model', 
                './models/forecast/forecast_standard.py', 
                framework=Framework.PYTHON).configure(
                "python", 
                input_schema=input_schema, 
                output_schema=output_schema
                )

Pipeline Deployment Configuration

For our pipeline we set the deployment configuration to set the resources the pipeline will be allocated from the Kubernetes cluster hosting the Wallaroo Ops instance. The Hugging Face model is deployed as a Containerized Runtime in Wallaroo, so the configuration specified the sidekick cpu and memory options.

Pipeline Deployment Configuration Exercise

Use the deployment configuration below.

deploy_config = wallaroo.DeploymentConfigBuilder().replica_count(1).cpus(0.5).memory("1Gi").build()

Build and Deploy the Pipeline

Now we build our pipeline and set our Yolo8 model as a pipeline step, then deploy the pipeline using the deployment configuration above.

Build and Deploy the Pipeline

We’ll do both commands in one step:

  • Build the pipeline with wallaroo.client.build_pipeline.
  • Set the model as a pipeline step with wallaroo.pipeline.add_model_step(model) method.

Sample code:

pipeline = wl.build_pipeline(pipeline_name) \
            .add_model_step(model_version)        
# build pipeline and set pipeline step

pipeline = wl.build_pipeline(pipeline_name) \
            .add_model_step(model_version)        

Deploy the Pipeline

We deploy the pipeline with the wallaroo.pipeline.deploy(deployment_config) command, using the deployment configuration set up in previous steps.

Deploy the Pipeline Exercise

Deploy the pipeline.

Sample code:

pipeline.deploy(deployment_config=deployment_config)
pipeline.deploy(deployment_config=deploy_config)
nameretail-forecast
created2023-12-05 23:13:01.779624+00:00
last_updated2023-12-05 23:13:03.332096+00:00
deployedTrue
archNone
tags
versions5d051000-de45-4167-b992-c6d092d2cb2e, 782b178a-ad0f-43a8-8ebc-d00e059a5f2b
stepsforecast-control-model
publishedFalse

Inference Request

We submit the DataFrame to the pipeline using wallaroo.pipeline.infer_from_file and display the results. We’ll use both the Wallaroo SDK and the MLOps API.

Inference Request Exercise

Perform an inference request. We’ll generate our sample dataframe, then use it for the inference.

Sample Code:

single_result = pipeline.infer_from_file('./data/forecast/testdata-standard.df.json')
display(single_result)

We’ll then do the same through the Pipeline Inference URL through an API call.

Sample Code:

!curl {deploy_url} \
    -H "Content-Type: application/json; format=pandas-records" \
    -H "Authorization: {wl.auth.auth_header()['Authorization']}" \
    -H "Accept:{headers['Accept']}" \
     --data @./data/forecast/testdata-standard.df.json
single_result = pipeline.infer_from_file('./data/forecast/testdata-standard.df.json')
display(single_result)
timein.countout.forecastout.weekly_averagecheck_failures
02023-12-05 23:13:21.444[1526, 1550, 1708, 1005, 1623, 1712, 1530, 1605, 1538, 1746, 1472, 1589, 1913, 1815, 2115, 2475, 2927, 1635, 1812, 1107, 1450, 1917, 1807, 1461, 1969, 2402, 1446, 1851][1764, 1749, 1743, 1741, 1740, 1740, 1740][1745.2857142857142]0
# API inference here

!curl {deploy_url} \
    -H "Content-Type: application/json; format=pandas-records" \
    -H "Authorization: {wl.auth.auth_header()['Authorization']}" \
    -H "Accept:{headers['Accept']}" \
     --data @./data/forecast/testdata-standard.df.json
[{"time":1701818005459,"in":{"count":[1526,1550,1708,1005,1623,1712,1530,1605,1538,1746,1472,1589,1913,1815,2115,2475,2927,1635,1812,1107,1450,1917,1807,1461,1969,2402,1446,1851]},"out":{"forecast":[1764,1749,1743,1741,1740,1740,1740],"weekly_average":[1745.2857142857142]},"check_failures":[],"metadata":{"last_model":"{\"model_name\":\"forecast-control-model\",\"model_sha\":\"3cd2acdd1f513f46615be7aa5beac16f09903be851e91f20f6dcdead4a48faa0\"}","pipeline_version":"","elapsed":[52701,33466756],"dropped":[],"partition":"engine-6464f7f889-tzwvp"}}]

Undeploy the Pipeline

With the testing complete, we undeploy the pipeline and return the resources back to the cluster.

# undeploy the pipeline

pipeline.undeploy()
nameretail-forecast
created2023-12-05 23:13:01.779624+00:00
last_updated2023-12-05 23:13:03.332096+00:00
deployedFalse
archNone
tags
versions5d051000-de45-4167-b992-c6d092d2cb2e, 782b178a-ad0f-43a8-8ebc-d00e059a5f2b
stepsforecast-control-model
publishedFalse

Publish the Pipeline for Edge Deployment

It worked! For a demo, we’ll take working once as “tested”. So now that we’ve tested our pipeline, we are ready to publish it for edge deployment.

Publishing it means assembling all of the configuration files and model assets and pushing them to an Open Container Initiative (OCI) repository set in the Wallaroo instance as the Edge Registry service. DevOps engineers then retrieve that image and deploy it through Docker, Kubernetes, or similar deployments.

See Edge Deployment Registry Guide for details on adding an OCI Registry Service to Wallaroo as the Edge Deployment Registry.

This is done through the SDK command wallaroo.pipeline.publish(deployment_config) which has the following parameters and returns.

Publish a Pipeline Parameters

The publish method takes the following parameters. The containerized pipeline will be pushed to the Edge registry service with the model, pipeline configurations, and other artifacts needed to deploy the pipeline.

ParameterTypeDescription
deployment_configwallaroo.deployment_config.DeploymentConfig (Optional)Sets the pipeline deployment configuration. For example: For more information on pipeline deployment configuration, see the Wallaroo SDK Essentials Guide: Pipeline Deployment Configuration.

Publish a Pipeline Returns

FieldTypeDescription
idintegerNumerical Wallaroo id of the published pipeline.
pipeline version idintegerNumerical Wallaroo id of the pipeline version published.
statusstringThe status of the pipeline publication. Values include:
  • PendingPublish: The pipeline publication is about to be uploaded or is in the process of being uploaded.
  • Published: The pipeline is published and ready for use.
Engine URLstringThe URL of the published pipeline engine in the edge registry.
Pipeline URLstringThe URL of the published pipeline in the edge registry.
Helm Chart URLstringThe URL of the helm chart for the published pipeline in the edge registry.
Helm Chart ReferencestringThe help chart reference.
Helm Chart VersionstringThe version of the Helm Chart of the published pipeline. This is also used as the Docker tag.
Engine Configwallaroo.deployment_config.DeploymentConfigThe pipeline configuration included with the published pipeline.
Created AtDateTimeWhen the published pipeline was created.
Updated AtDateTimeWhen the published pipeline was updated.

Publish Exercise

We will now publish the pipeline to our Edge Deployment Registry with the pipeline.publish(deployment_config) command. deployment_config is an optional field that specifies the pipeline deployment. This can be overridden by the DevOps engineer during deployment.

Save the publish to a variable for later use.

Sample code:

pub = pipeline.publish(deployment_config)
pub
# create publish here

pub = pipeline.publish(deploy_config)
pub
Waiting for pipeline publish... It may take up to 600 sec.
Pipeline is Publishing...Published.
ID2
Pipeline Version4dfc8337-a4f2-42ce-b5ca-c401f29dddeb
StatusPublished
Engine URLghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/standalone-mini:v2023.4.0-4103
Pipeline URLghcr.io/wallaroolabs/doc-samples/pipelines/retail-forecast:4dfc8337-a4f2-42ce-b5ca-c401f29dddeb
Helm Chart URLoci://ghcr.io/wallaroolabs/doc-samples/charts/retail-forecast
Helm Chart Referenceghcr.io/wallaroolabs/doc-samples/charts@sha256:6ec77447f5a74eae5add8cd5091b75dcf59aee60075490e54e9e191effdc1436
Helm Chart Version0.0.1-4dfc8337-a4f2-42ce-b5ca-c401f29dddeb
Engine Config{'engine': {'resources': {'limits': {'cpu': 1.0, 'memory': '512Mi'}, 'requests': {'cpu': 1.0, 'memory': '512Mi'}}}, 'engineAux': {'images': {}}, 'enginelb': {'resources': {'limits': {'cpu': 1.0, 'memory': '512Mi'}, 'requests': {'cpu': 1.0, 'memory': '512Mi'}}}}
User Images[]
Created Byjohn.hummel@wallaroo.ai
Created At2023-12-05 23:14:14.452826+00:00
Updated At2023-12-05 23:14:14.452826+00:00
Docker Run Variables{}

List Published Pipeline

The method wallaroo.client.list_pipelines() shows a list of all pipelines in the Wallaroo instance, and includes the published field that indicates whether the pipeline was published to the registry (True), or has not yet been published (False).

List Published Pipeline Exercise

List the pipelines and verify which ones are published or not.

Sample code:

wl.list_pipelines()
# list pipelines

wl.list_pipelines()
namecreatedlast_updateddeployedarchtagsversionsstepspublished
retail-forecast2023-05-Dec 23:13:012023-05-Dec 23:14:13FalseNone4dfc8337-a4f2-42ce-b5ca-c401f29dddeb, 5d051000-de45-4167-b992-c6d092d2cb2e, 782b178a-ad0f-43a8-8ebc-d00e059a5f2bforecast-control-modelTrue
llm-edge-summarization2023-05-Dec 20:48:362023-05-Dec 20:50:14FalseNone1c8d0586-f3ff-453c-82a9-14602830f97f, 040afe0c-9a17-4f0b-8cb2-68e8dc85d9a4, 6f1db20f-e1b5-47a5-a6b4-c6a31f9c1023llm-summarizationTrue

List Publishes from a Pipeline

All publishes created from a pipeline are displayed with the wallaroo.pipeline.publishes method. The pipeline_version_id is used to know what version of the pipeline was used in that specific publish. This allows for pipelines to be updated over time, and newer versions to be sent and tracked to the Edge Deployment Registry service.

List Publishes Parameters

N/A

List Publishes Returns

A List of the following fields:

FieldTypeDescription
idintegerNumerical Wallaroo id of the published pipeline.
pipeline_version_idintegerNumerical Wallaroo id of the pipeline version published.
engine_urlstringThe URL of the published pipeline engine in the edge registry.
pipeline_urlstringThe URL of the published pipeline in the edge registry.
created_bystringThe email address of the user that published the pipeline.
Created AtDateTimeWhen the published pipeline was created.
Updated AtDateTimeWhen the published pipeline was updated.

List Publishes from a Pipeline Exercise

List the publishes from a pipeline.

Sample code:

pipeline.publishes()
pipeline.publishes()
idpipeline_version_nameengine_urlpipeline_urlcreated_bycreated_atupdated_at
24dfc8337-a4f2-42ce-b5ca-c401f29dddebghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/standalone-mini:v2023.4.0-4103ghcr.io/wallaroolabs/doc-samples/pipelines/retail-forecast:4dfc8337-a4f2-42ce-b5ca-c401f29dddebjohn.hummel@wallaroo.ai2023-05-Dec 23:14:142023-05-Dec 23:14:14

Add Edge Location

With the pipeline publish created, we can add an Edge Location. This allows the edge deployment to upload its inference results back to the Wallaroo Ops location, which are then added to the pipeline the publish originated from. These are added to the pipeline logs partition metadata.

First we’ll retrieve the pipeline logs for our current pipeline, and show the current pipeline logs metadata.

Add Edge Location Exercise

Display the log information with the metadata.partition, then add the edge location to the publish. Note that edge names must be unique, so add your first and last name to the list.

Sample code:

logs = pipeline.logs(dataset=['time', 'out.output0', 'metadata'])
display(logs.loc[:, ['time', 'metadata.partition']])

first_last_name = '-Gale-Karlach'

edge_name = f'edge-forecast-retail-demo{first_last_name}'

edge_publish = pub.add_edge(edge_name)
display(edge_publish)
# get the log metadata

logs = pipeline.logs(dataset=['time', 'out.weekly_average', 'metadata'])
display(logs.loc[:, ['time', 'out.weekly_average', 'metadata.partition']])
timeout.weekly_averagemetadata.partition
02023-12-05 23:13:25.459[1745.2857142857142]engine-6464f7f889-tzwvp
12023-12-05 23:13:21.444[1745.2857142857142]engine-6464f7f889-tzwvp

Now we’ll add the edge location.

For the edge name, set it to firstname-lastname-edge-llm-summarization.

pub = pipeline.publishes()[0]
pub
ID2
Pipeline Version4dfc8337-a4f2-42ce-b5ca-c401f29dddeb
StatusPublished
Engine URLghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/standalone-mini:v2023.4.0-4103
Pipeline URLghcr.io/wallaroolabs/doc-samples/pipelines/retail-forecast:4dfc8337-a4f2-42ce-b5ca-c401f29dddeb
Helm Chart URLoci://ghcr.io/wallaroolabs/doc-samples/charts/retail-forecast
Helm Chart Referenceghcr.io/wallaroolabs/doc-samples/charts@sha256:6ec77447f5a74eae5add8cd5091b75dcf59aee60075490e54e9e191effdc1436
Helm Chart Version0.0.1-4dfc8337-a4f2-42ce-b5ca-c401f29dddeb
Engine Config{'engine': {'resources': {'limits': {'cpu': 1.0, 'memory': '512Mi'}, 'requests': {'cpu': 1.0, 'memory': '512Mi'}}}, 'enginelb': {'resources': {'limits': {'cpu': 1.0, 'memory': '512Mi'}, 'requests': {'cpu': 1.0, 'memory': '512Mi'}}}, 'engineAux': {'images': {}}}
User Images[]
Created Byjohn.hummel@wallaroo.ai
Created At2023-12-05 23:14:14.452826+00:00
Updated At2023-12-05 23:14:14.452826+00:00
Docker Run Variables{}
# create the location

edge_name = 'edge-forecast-retail-demo'

edge_publish = pub.add_edge(edge_name)
display(edge_publish)
ID2
Pipeline Version4dfc8337-a4f2-42ce-b5ca-c401f29dddeb
StatusPublished
Engine URLghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/standalone-mini:v2023.4.0-4103
Pipeline URLghcr.io/wallaroolabs/doc-samples/pipelines/retail-forecast:4dfc8337-a4f2-42ce-b5ca-c401f29dddeb
Helm Chart URLoci://ghcr.io/wallaroolabs/doc-samples/charts/retail-forecast
Helm Chart Referenceghcr.io/wallaroolabs/doc-samples/charts@sha256:6ec77447f5a74eae5add8cd5091b75dcf59aee60075490e54e9e191effdc1436
Helm Chart Version0.0.1-4dfc8337-a4f2-42ce-b5ca-c401f29dddeb
Engine Config{'engine': {'resources': {'limits': {'cpu': 1.0, 'memory': '512Mi'}, 'requests': {'cpu': 1.0, 'memory': '512Mi'}}}, 'engineAux': {'images': {}}, 'enginelb': {'resources': {'limits': {'cpu': 1.0, 'memory': '512Mi'}, 'requests': {'cpu': 1.0, 'memory': '512Mi'}}}}
User Images[]
Created Byjohn.hummel@wallaroo.ai
Created At2023-12-05 23:14:14.452826+00:00
Updated At2023-12-05 23:14:14.452826+00:00
Docker Run Variables{'EDGE_BUNDLE': 'abcde'}

DevOps - Pipeline Edge Deployment

Once a pipeline is deployed to the Edge Registry service, it can be deployed in environments such as Docker, Kubernetes, or similar container running services by a DevOps engineer.

Docker Deployment

First, the DevOps engineer must authenticate to the same OCI Registry service used for the Wallaroo Edge Deployment registry.

For more details, check with the documentation on your artifact service. The following are provided for the three major cloud services:

For the deployment, the engine URL is specified with the following environmental variables:

  • DEBUG (true|false): Whether to include debug output.
  • OCI_REGISTRY: The URL of the registry service.
  • CONFIG_CPUS: The number of CPUs to use.
  • OCI_USERNAME: The edge registry username.
  • OCI_PASSWORD: The edge registry password or token.
  • PIPELINE_URL: The published pipeline URL.

Docker Deployment Example

Using our sample environment, here’s sample deployment using Docker with a computer vision ML model, the same used in the Wallaroo Use Case Tutorials Computer Vision: Retail tutorials.

Note the use of the -v ./data:/persist option. This will store the one time authentication token stored in the EDGE_BUNDLE

mkdir ./data

docker run -p 8080:8080 \
    -v ./data:/persist \
    -e DEBUG=true -e OCI_REGISTRY={your registry server} \
    -e EDGE_BUNDLE={edge_publish.docker_run_variables['EDGE_BUNDLE']} \
    -e CONFIG_CPUS=4 \
    -e OCI_USERNAME=oauth2accesstoken \
    -e OCI_PASSWORD={registry token here} \
    -e PIPELINE_URL={your registry server}/pipelines/yolo8demonstration:bf70eaf7-8c11-4b46-b751-916a43b1a555 \
    {your registry server}/engine:v2023.3.0-main-3707

Docker Compose Deployment

For users who prefer to use docker compose, the following sample compose.yaml file is used to launch the Wallaroo Edge pipeline. This is the same used in the Wallaroo Use Case Tutorials Computer Vision: Retail tutorials.

The volumes settings allows for persistent volumes to store the session information. Without it, the one-time authentication token included in the EDGE_BUNDLE settings would have to be regenerated.

services:
  engine:
    image: {Your Engine URL}
    volumes:
      - ./data:/persist
    ports:
      - 8080:8080
    environment:
      EDGE_BUNDLE: abcdefg
      PIPELINE_URL: {Your Pipeline URL}
      OCI_REGISTRY: {Your Edge Registry URL}
      OCI_USERNAME:  {Your Registry Username}
      OCI_PASSWORD: {Your Token or Password}
      CONFIG_CPUS: 4

For example:

services:
  engine:
    image: sample-registry.com/engine:v2023.3.0-main-3707
    ports:
      - 8080:8080
    environment:
      PIPELINE_URL: sample-registry.com/pipelines/yolo8demonstration:bf70eaf7-8c11-4b46-b751-916a43b1a555
      OCI_REGISTRY: sample-registry.com
      OCI_USERNAME:  _json_key_base64
      OCI_PASSWORD: abc123
      CONFIG_CPUS: 4

Docker Compose Deployment Example

The deployment and undeployment is then just a simple docker compose up and docker compose down. The following shows an example of deploying the Wallaroo edge pipeline using docker compose.

docker compose up
[+] Running 1/1
 ✔ Container yolo8demonstration-engine-1  Recreated                                                                                                                                                                 0.5s
Attaching to yolo8demonstration-engine-1
yolo8demonstration-engine-1  | Wallaroo Engine - Standalone mode
yolo8demonstration-engine-1  | Login Succeeded
yolo8demonstration-engine-1  | Fetching manifest and config for pipeline: sample-registry.com/pipelines/yolo8demonstration:bf70eaf7-8c11-4b46-b751-916a43b1a555
yolo8demonstration-engine-1  | Fetching model layers
yolo8demonstration-engine-1  | digest: sha256:c6c8869645962e7711132a7e17aced2ac0f60dcdc2c7faa79b2de73847a87984
yolo8demonstration-engine-1  |   filename: c6c8869645962e7711132a7e17aced2ac0f60dcdc2c7faa79b2de73847a87984
yolo8demonstration-engine-1  |   name: yolov8n
yolo8demonstration-engine-1  |   type: model
yolo8demonstration-engine-1  |   runtime: onnx
yolo8demonstration-engine-1  |   version: 693e19b5-0dc7-4afb-9922-e3f7feefe66d
yolo8demonstration-engine-1  |
yolo8demonstration-engine-1  | Fetched
yolo8demonstration-engine-1  | Starting engine
yolo8demonstration-engine-1  | Looking for preexisting `yaml` files in //modelconfigs
yolo8demonstration-engine-1  | Looking for preexisting `yaml` files in //pipelines

Helm Deployment

Published pipelines can be deployed through the use of helm charts.

Helm deployments take up to two steps - the first step is in retrieving the required values.yaml and making updates to override.

Kubernetes provides persistent volume support, so no settings are required.

  1. Pull the helm charts from the published pipeline. The two fields are the Helm Chart URL and the Helm Chart version to specify the OCI . This typically takes the format of:
helm pull oci://{published.helm_chart_url} --version {published.helm_chart_version}
  1. Extract the tgz file and copy the values.yaml and copy the values used to edit engine allocations, etc. The following are required for the deployment to run:
ociRegistry:
  registry: {your registry service}
  username:  {registry username here}
  password: {registry token here}

Store this into another file, suc as local-values.yaml.

  1. Create the namespace to deploy the pipeline to. For example, the namespace wallaroo-edge-pipeline would be:
kubectl create -n wallaroo-edge-pipeline
  1. Deploy the helm installation with helm install through one of the following options:

    1. Specify the tgz file that was downloaded and the local values file. For example:

      helm install --namespace {namespace} --values {local values file} {helm install name} {tgz path}
      
    2. Specify the expended directory from the downloaded tgz file.

      helm install --namespace {namespace} --values {local values file} {helm install name} {helm directory path}
      
    3. Specify the Helm Pipeline Helm Chart and the Pipeline Helm Version.

      helm install --namespace {namespace} --values {local values file} {helm install name} oci://{published.helm_chart_url} --version {published.helm_chart_version}
      
  2. Once deployed, the DevOps engineer will have to forward the appropriate ports to the svc/engine-svc service in the specific pipeline. For example, using kubectl port-forward to the namespace ccfraud that would be:

    kubectl port-forward svc/engine-svc -n ccfraud01 8080 --address 0.0.0.0`
    

Docker Deployment Code Generation Exercise

The following code segment generates a docker run template based on the previously published pipeline. Replace the $REGISTRYURL, $REGISTRYUSERNAME, and $REGISTRYPASSWORD to match the OCI Registry being used.

docker_deploy = f'''
mkdir data
docker run -p 8080:8080 \\
    -v ./data:/persist \\
    -e DEBUG=true \\
    -e OCI_REGISTRY=$REGISTRYURL \\
    -e EDGE_BUNDLE={edge_publish.docker_run_variables['EDGE_BUNDLE']} \\
    -e CONFIG_CPUS=1 \\
    -e OCI_USERNAME=$REGISTRYUSERNAME \\
    -e OCI_PASSWORD=$REGISTRYPASSWORD \\
    -e PIPELINE_URL={edge_publish.pipeline_url} \\
    {edge_publish.engine_url}
'''

print(docker_deploy)
mkdir data
docker run -p 8080:8080 \
    -v ./data:/persist \
    -e DEBUG=true \
    -e OCI_REGISTRY=$REGISTRYURL \
    -e EDGE_BUNDLE=ZXhwb3J0IEJVTkRMRV9WRVJTSU9OPTEKZXhwb3J0IEVER0VfTkFNRT1lZGdlLWZvcmVjYXN0LXJldGFpbC1kZW1vCmV4cG9ydCBKT0lOX1RPS0VOPTFjNTVjZWJiLTMxNzUtNDk1MC04NDBmLTc5NjIxMzJmYjM5MgpleHBvcnQgT1BTQ0VOVEVSX0hPU1Q9ZG9jLXRlc3QuZWRnZS53YWxsYXJvb2NvbW11bml0eS5uaW5qYQpleHBvcnQgUElQRUxJTkVfVVJMPWdoY3IuaW8vd2FsbGFyb29sYWJzL2RvYy1zYW1wbGVzL3BpcGVsaW5lcy9yZXRhaWwtZm9yZWNhc3Q6NGRmYzgzMzctYTRmMi00MmNlLWI1Y2EtYzQwMWYyOWRkZGViCmV4cG9ydCBXT1JLU1BBQ0VfSUQ9OQ== \
    -e CONFIG_CPUS=1 \
    -e OCI_USERNAME=$REGISTRYUSERNAME \
    -e OCI_PASSWORD=$REGISTRYPASSWORD \
    -e PIPELINE_URL=ghcr.io/wallaroolabs/doc-samples/pipelines/retail-forecast:4dfc8337-a4f2-42ce-b5ca-c401f29dddeb \
    ghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/standalone-mini:v2023.4.0-4103

Edge Deployed Pipeline API Endpoints

Once deployed, we can check the pipelines and models available. We’ll use a curl command, but any HTTP based request will work the same way.

The endpoint /pipelines returns:

  • id (String): The name of the pipeline.
  • status (String): The status as either Running, or Error if there are any issues.
curl localhost:8080/pipelines
{"pipelines":[{"id":"yolo8demonstration","status":"Running"}]}

The following example uses the host localhost. Replace with your own host name of your Edge deployed pipeline.

!curl localhost:8080/pipelines
{"pipelines":[{"id":"retail-forecast","status":"Running"}]}

The endpoint /models returns a List of models with the following fields:

  • name (String): The model name.
  • sha (String): The sha hash value of the ML model.
  • status (String): The status of either Running or Error if there are any issues.
  • version (String): The model version. This matches the version designation used by Wallaroo to track model versions in UUID format.
{"models":[{"name":"yolov8n","sha":"3ed5cd199e0e6e419bd3d474cf74f2e378aacbf586e40f24d1f8c89c2c476a08","status":"Running","version":"7af40d06-d18f-4b3f-9dd3-0a15248f01c8"}]}

The following example uses the host localhost. Replace with your own host name of your Edge deployed pipeline.

!curl localhost:8080/models
{"models":[{"name":"forecast-control-model","version":"3baf8cf9-f638-4b94-b3cb-163a82da959e","sha":"3cd2acdd1f513f46615be7aa5beac16f09903be851e91f20f6dcdead4a48faa0","status":"Running"}]}

Edge Inference Endpoint

The inference endpoint takes the following pattern:

  • /pipelines/{pipeline-name}: The pipeline-name is the same as returned from the /pipelines endpoint as id.

Wallaroo inference endpoint URLs accept the following data inputs through the Content-Type header:

  • Content-Type: application/vnd.apache.arrow.file: For Apache Arrow tables.
  • Content-Type: application/json; format=pandas-records: For pandas DataFrame in record format.

Once deployed, we can perform an inference through the deployment URL.

The endpoint returns Content-Type: application/json; format=pandas-records by default with the following fields:

  • check_failures (List[Integer]): Whether any validation checks were triggered. For more information, see Wallaroo SDK Essentials Guide: Pipeline Management: Anomaly Testing.
  • elapsed (List[Integer]): A list of time in nanoseconds for:
    • [0] The time to serialize the input.
    • [1…n] How long each step took.
  • model_name (String): The name of the model used.
  • model_version (String): The version of the model in UUID format.
  • original_data: The original input data. Returns null if the input may be too long for a proper return.
  • outputs (List): The outputs of the inference result separated by data type, where each data type includes:
    • data: The returned values.
    • dim (List[Integer]): The dimension shape returned.
    • v (Integer): The vector shape of the data.
  • pipeline_name (String): The name of the pipeline.
  • shadow_data: Any shadow deployed data inferences in the same format as outputs.
  • time (Integer): The time since UNIX epoch.

Once deployed, we can perform an inference through the deployment URL. We’ll assume we’re running the inference request through the localhost and submitting the local file ./data/dogbike.df.json. Note that our inference endpoint is pipelines/yolo8demonstration - the same as our pipeline name.

The following example demonstrates sending an inference request to the edge deployed pipeline and storing the results in a pandas DataFrame in record format. The results can then be exported to other processes to render the detected images or other use cases.

!curl testboy.local:8080/pipelines/retail-forecast \
    -H "Content-Type: application/json; format=pandas-records" \
    --data @./data/forecast/testdata-standard.df.json
[{"time":1701962296374,"in":{"count":[1526,1550,1708,1005,1623,1712,1530,1605,1538,1746,1472,1589,1913,1815,2115,2475,2927,1635,1812,1107,1450,1917,1807,1461,1969,2402,1446,1851]},"out":{"forecast":[1764,1749,1743,1741,1740,1740,1740],"weekly_average":[1745.2857142857142]},"check_failures":[],"metadata":{"last_model":"{\"model_name\":\"forecast-control-model\",\"model_sha\":\"3cd2acdd1f513f46615be7aa5beac16f09903be851e91f20f6dcdead4a48faa0\"}","pipeline_version":"","elapsed":[251572,1052979425],"dropped":[],"partition":"edge-forecast-retail-demo"}}]

Display Partition Logs

To view the edge deployed pipeline logs, we can use wallaroo.pipeline.export_logs method to retrieve all of the recent logs from this pipeline, and show the edge inference results were sent with the edge name in the partition metadata.

Sample code:

# display log information here with partition

pipeline.export_logs(directory='./logs/partition-edge-observability-forecasting',
                     file_prefix='edge-logs',
                     dataset=['time', 'metadata'])

# display the partition only results

df_logs = pd.read_json('./logs/partition-edge-observability-forecasting/edge-logs-1.json', 
                       orient="records", 
                       lines=True)

# display just the entries with out edge location
display(df_logs[df_logs['metadata.partition']==edge_name].loc[:, ['time', 'metadata.partition']])
# display log information here with partition

pipeline.export_logs(directory='./logs/partition-edge-observability-forecasting',
                     file_prefix='edge-logs',
                     dataset=['time', 'metadata'])

# display the partition only results

df_logs = pd.read_json('./logs/partition-edge-observability-forecasting/edge-logs-1.json', 
                       orient="records", 
                       lines=True)

# display just the entries with out edge location
display(df_logs[df_logs['metadata.partition']==edge_name].loc[:, ['time', 'metadata.partition']])
timemetadata.partition
21701819095189edge-forecast-retail-demo-arm

3 - Computer Vision

Wallaroo Use Case Tutorials focused on Computer Vision models

3.1 - Computer Vision: Healthcare

Wallaroo Use Case Tutorials focused on solving Healthcare problems with Computer Vision models

This tutorial and the assets can be downloaded as part of the Wallaroo Workshops repository.

Step 00: Introduction and Setup

This tutorial series will demonstrate the following:

  • How to deploy a Wallaroo pipeline with trained rnn mobilenet model and perform sample inferences to detect objects in pictures, then display those objects.
  • How to deploy a Wallaroo pipeline with a trained resnet50 model and perform sample inferences to detect objects in pictures, then display those objects.
  • Use the Wallaroo feature shadow deploy to have both models perform inferences, then select the inference result with the higher confidence and show the objects detected.

This tutorial assumes that users have installed the Wallaroo SDK or are running these tutorials from within their Wallaroo instance’s JupyterHub service.

This demonstration should be run within a Wallaroo JupyterHub instance for best results.

Prerequisites

The included OpenCV class is included in this demonstration as CVDemoUtils.py, and requires the following dependencies:

  • ffmpeg
  • libsm
  • libxext

Internal JupyterHub Service

To install these dependencies in the Wallaroo JupyterHub service, use the following commands from a terminal shell via the following procedure:

  1. Launch the JupyterHub Service within the Wallaroo install.

  2. Select File->New->Terminal.

  3. Enter the following:

    sudo apt-get update
    
    sudo apt-get install ffmpeg libsm6 libxext6  -y
    

External SDK Users

For users using the Wallaroo SDK to connect with a remote Wallaroo instance, the following commands will install the required dependancies:

For Linux users, this can be installed with:

sudo apt-get update
sudo apt-get install ffmpeg libsm6 libxext6  -y

MacOS users can prepare their environments using a package manager such as Brew with the following:

brew install ffmpeg libsm libxext

Libraries and Dependencies

  1. This repository may use large file sizes for the models. If necessary, install Git Large File Storage (LFS) or use the Wallaroo Tutorials Releases to download a .zip file of the most recent computer vision tutorial that includes the models.
  2. Import the following Python libraries into your environment:
    1. torch
    2. wallaroo
    3. torchvision
    4. opencv-python
    5. onnx
    6. onnxruntime
    7. imutils
    8. pytz
    9. ipywidgets

These can be installed by running the command below in the Wallaroo JupyterHub service. Note the use of pip install torch --no-cache-dir for low memory environments.

!pip install torchvision==0.15.2
!pip install torch==2.0.1 --no-cache-dir
!pip install opencv-python==4.7.0.72
!pip install onnx==1.12.0
!pip install onnxruntime==1.15.0
!pip install imutils==0.5.4
!pip install pytz
!pip install ipywidgets==8.0.6
!pip install patchify==0.2.3
!pip install tifffile==2023.4.12
!pip install piexif==1.1.3

The rest of the tutorials will rely on these libraries and applications, so finish their installation before running the tutorials in this series.

Models for Wallaroo Computer Vision Tutorials

In order for the wallaroo tutorial notebooks to run properly, the videos directory must contain these models in the models directory.

To download the Wallaroo Computer Vision models, use the following link:

https://storage.googleapis.com/wallaroo-public-data/cv-demo-models/cv-retail-models.zip

Unzip the contents into the directory models.

Directory contents

  • coco_classes.pickle - contain the 80 COCO classifications used by resnet50 and mobilenet object detectors.
  • frcnn-resent.pt - PyTorch resnet50 model
  • frcnn-resnet.pt.onnx - PyTorch resnet50 model converted to onnx
  • mobilenet.pt - PyTorch mobilenet model
  • mobilenet.pt.onnx - PyTorch mobilenet model converted to onnx

3.1.1 - Healthcare: Upload and Deploy

How to upload and deploy a computer vision model focused on computer vision healthcare to Wallaroo.

This tutorial and the assets can be downloaded as part of the Wallaroo Workshops repository.

For this workshop, let’s pretend that you work for a health care company wants detect mitochondria from microscope images. You have developed a model to predict the sale price of properties that the company has listed, based on data collected in the company’s listings database.

In this set of exercises, you will used a pre-trained model and deploy it to Wallaroo. This will require understanding the following concepts:

  • Wallaroo Workspaces: Workspaces are environments were users upload models, create pipelines and other artifacts. The workspace should be considered the fundamental area where work is done. Workspaces are shared with other users to give them access to the same models, pipelines, etc.
  • Wallaroo Model Upload and Registration: ML Models are uploaded to Wallaroo through the SDK or the MLOps API to a workspace. ML models include default runtimes (ONNX, Python Step, and TensorFlow) that are run directly through the Wallaroo engine, and containerized runtimes (Hugging Face, PyTorch, etc) that are run through in a container through the Wallaroo engine.
  • Wallaroo Pipelines: Pipelines are used to deploy models for inferencing. Each model is a pipeline step in a pipelines, where the inputs of the previous step are fed into the next. Pipeline steps can be ML models, Python scripts, or Arbitrary Python (these contain necessary models and artifacts for running a model).

For this tutorial, we will be providing pre-trained models in ONNX format. To see how to upload and deploy your particular model, see the Wallaroo Documentation site.

This tutorial includes a helper module CVDemoUtils, which converts images into a DataFrame format that our computer vision model expects.

Preliminary Steps

Before starting this module, ensure that the models have been downloaded.

To download the Wallaroo Computer Vision models, use the following link:

https://storage.googleapis.com/wallaroo-public-data/cv-demo-models/cv-retail-models.zip

Unzip the contents into the directory models.

Before we start, let’s load some libraries that we will need for this notebook (note that this may not be a complete list).

  • IMPORTANT NOTE: This tutorial is geared towards a Wallaroo 2023.2.1 environment.
import json
import IPython.display as display
import time
import matplotlib.pyplot as plt
from IPython.display import clear_output, display
import tifffile as tiff

import pandas as pd

import wallaroo
from wallaroo.object import EntityNotFoundError

import numpy as np
from matplotlib import pyplot as plt
import cv2
from tensorflow.keras.utils import normalize

# setting path - only needed when running this from the `with-code` folder.
import sys
sys.path.append('../')

from lib.TiffImageUtils import TiffUtils
tiff_utils = TiffUtils()

# ignoring warnings for demonstration
import warnings
warnings.filterwarnings('ignore')

import requests

Get ready to work with Wallaroo

With the libraries loaded, you can log into Wallaroo. This will provide access to your workspaces, workspaces shared with you from other users, and all other aspects of the Wallaroo environment.

Logging into Wallaroo via the cluster’s integrated JupyterLab is quite straight forward:

# Login through local Wallaroo instance 
wl = wallaroo.Client()

See the documentation if you are logging into Wallaroo some other way such as from a remote location. This tutorial assumes you’re logging in through the Wallaroo JupyterHub service.

Notice that the Wallaroo client connection is stored into a variable called wl. This variable can be anything you want it to be - you can have client = wallaroo.Client() or myWallarooClient = wallaroo.Client().

This variable is about to become your best friend - a lot of the commands you’ll be running will be through this variable, like wl.list_workspaces() to get all of the workspaces available to you in your Wallaroo environment, or wl.list_models() to show all of the models in your current workspace. We’ll go into these commands and more - just make sure you saved that Wallaroo client to a variable so you can use it for the other commands.

When we log into the Wallaroo through the SDK, the Client will provide a url to verify your authentication. Either click it, or copy and past that URL, then authenticate into your Wallaroo instance with your email address and password.

Login to Wallaroo Exercise

Time to login to your Wallaroo instance. By now you should be logged into your Wallaroo JupyterHub service and looking at this notebook.

Copy the code below and place it into the code block and run it. When prompted, select the authentication URL by either clicking it, or copying and pasting it into a browser. Log into your Wallaroo instance, and then the client will be set.

# Login through local Wallaroo instance 
wl = wallaroo.Client()
# put your wallaroo Client login code here.

# Login through local Wallaroo instance 
wl = wallaroo.Client()

Run Client Commands

We’re now logged into our Wallaroo instance, let’s run some commands to get used to working within the environment.

The following are going to be very useful as you work in Wallaroo.

List Workspaces

The command wallaroo.Client.list_workspaces gives a list of all of the workspaces you have access to in the Wallaroo environment. Here’s an example of running it. For this example, our Wallaroo client is stored in the wl variable, but this could have been named wallaroo_client or whatever you like.

wl.list_workspaces()
NameCreated AtUsersModelsPipelines
john.hummel@wallaroo.ai - Default Workspace2023-08-21 19:06:07[‘john.hummel@wallaroo.ai’]11
edge-publish-demojohn2023-08-21 20:54:35[‘john.hummel@wallaroo.ai’]11
biolabsworkspacegomj2023-08-22 15:11:11[‘john.hummel@wallaroo.ai’]11
biolabsworkspacedtrv2023-08-22 16:03:32[‘john.hummel@wallaroo.ai’]11
biolabsworkspacejohn2023-08-22 16:07:15[‘john.hummel@wallaroo.ai’]11

Listing the workspaces will show the following fields:

  • name: The user created workspace name. Workspace names must be unique across the Wallaroo instance.
  • created_at: The date and time the workspace was created.
  • users: The users in the workspace. The user who created the workspace is always listed.
  • models: The number of models in the workspace.
  • pipelines: The number of pipelines in the workspace.

When you first login to Wallaroo, the workspace {your email address} - Default Workspace is created. This is assigned as your current workspace once you have logged into Wallaroo. We’ll cover workspaces in a moment - for now, just remember that every time you do wallaroo.Client(), that your default workspace is set as your current workspace, and any commands you issue will be directed to that workspace.

Notice that in that example above, you only see one default workspace - the one of the user who ran the list_workspaces command. Other users will have their own default workspaces. You can only see workspaces you have access to.

List Users

Users are listed with the wallaroo.Client.list_users command. This shows all of the user’s email addresses and names across the Wallaroo instance.

There’s more commands, but we’ll stop here until we’ve created our workspace and uploaded some models.

Client Commands Exercise

For this exercise, use your Wallaroo client variable and list the current workspaces and current users. You can do this with your Wallaroo client variable you saved in a previous step.

For example, if your Wallaroo client variable is wl, then the following would list the workspaces and then the users:

# list the workspaces available to you
workspaces = wl.list_workspaces()
print(workspaces)

# list the users
users = wl.list_users()
print(users)
# empty space to get workspaces and users

display(wl.list_workspaces())

display(wl.list_users())
NameCreated AtUsersModelsPipelines
john.hummel@wallaroo.ai - Default Workspace2023-09-27 15:17:23['john.hummel@wallaroo.ai']12
cv-arm-edge-demo-jch2023-09-27 15:17:44['john.hummel@wallaroo.ai']11
workshop-workspace-john-102023-09-27 16:29:36['john.hummel@wallaroo.ai']21
workshop-workspace-john-052023-09-27 16:50:52['john.hummel@wallaroo.ai']11
workshop-workspace-john-cv-yolo2023-09-28 16:06:55['john.hummel@wallaroo.ai']21
retail2023-09-28 19:44:06['john.hummel@wallaroo.ai']11
workshop-workspace-john-cv-medical2023-09-28 20:22:21['john.hummel@wallaroo.ai']11
[User({"id": "568f86b1-8b18-4db8-a40b-eca338fe371d", "email": "admin@keycloak", "username": "admin", "enabled": "True),
 User({"id": "b030ff9c-41eb-49b4-afdf-2ccbecb6be5d", "email": "john.hummel@wallaroo.ai", "username": "john.hummel@wallaroo.ai", "enabled": "True)]

Workspace Creation and Management

A Wallaroo workspace is place to organize the deployment artifacts for a project, and to collaborate with other team members. For more information, see the Wallaroo 101.

When you upload a ML model to Wallaroo, you upload it to a workspace. When a pipeline is created and models assigned to it, that pipeline is in side of a workspace. Wallaroo workload orchestrations? They’re assigned to a workspace.

.

When you first login, the SDK assigns you to your default workspace. Most of the commands issued through your Wallaroo client will target that workspace.

You can see what workspace you are currently in with the wallaroo.Client.get_current_workspace() method.

This shows you the following workspace fields:

  • name: The user created workspace name. Workspace names must be unique across the Wallaroo instance.
  • id: The numerical identifier of the workspace.
  • archived: Whether the workspace was archived or not.
  • created_by: The Keycloak ID of the user who created the workspace. This is in UUID format, and is used to identify specific users. Most of the time you’ll refer to users by their email address.

  • created_at: The date and time the workspace was created.
  • models: The models uploaded to the workspace and their details.
  • pipelines: The number of pipelines in the workspace and their details.

Get Current Workspace Exercise

Get your current workspace with the wallaroo.Client.get_current_workspace(). For example, if your Wallaroo client was saved to the variable wl, this command would be:

wl.get_current_workspace()

When done,

## blank space to get your current workspace

wl.get_current_workspace()
{'name': 'john.hummel@wallaroo.ai - Default Workspace', 'id': 1, 'archived': False, 'created_by': 'b030ff9c-41eb-49b4-afdf-2ccbecb6be5d', 'created_at': '2023-09-27T15:17:23.923705+00:00', 'models': [{'name': 'v5s6', 'versions': 2, 'owner_id': '""', 'last_update_time': datetime.datetime(2023, 9, 28, 19, 23, 3, 425821, tzinfo=tzutc()), 'created_at': datetime.datetime(2023, 9, 28, 18, 55, 24, 860357, tzinfo=tzutc())}], 'pipelines': [{'name': 'cv-arm-edge', 'create_time': datetime.datetime(2023, 9, 27, 15, 17, 45, 593032, tzinfo=tzutc()), 'definition': '[]'}, {'name': 'retailimage', 'create_time': datetime.datetime(2023, 9, 28, 18, 55, 14, 954338, tzinfo=tzutc()), 'definition': '[]'}]}

Create New Workspace

Workspaces are created with the wallaroo.Client.create_workspace(name), where name is the new name of the workspace. When a new workspace is created, the workspace user is assigned to the user that created it (in this case - you). For example, if the Wallaroo client is stored to the variable wl, then the following will create the new workspace ‘sparkly-bunnies, then store the workspace information into the variable workspace`:

workspace = wl.create_workspace('sparkly-bunnies')

Once this is created, this shows you the following workspace fields:

  • name: The user created workspace name. Workspace names must be unique across the Wallaroo instance, and must be DNS compliant. So ‘my-cool-workspace’ is ok, but ‘?? workspace’ is not. - are ok, but _ is not.
  • id: The numerical identifier of the workspace.
  • archived: Whether the workspace was archived or not.
  • created_by: The Keycloak ID of the user who created the workspace. This is in UUID format, and is used to identify specific users. Most of the time you’ll refer to users by their email address.

  • created_at: The date and time the workspace was created.
  • models: The models uploaded to the workspace and their details. For a new workspace, this will be empty [].
  • pipelines: The number of pipelines in the workspace and their details. For a new workspace, this will be empty [].

Workspace names must be unique. So the following will fail:

wl.create_workspace('sparkly-bunnies')
wl.create_workspace('sparkly-bunnies')
Exception: Failed to create workspace.

Create New Workspace Exercise

Now it’s time for us to create our own workspace. To make this easy, we’ll call the workspace ‘workshop-workspace-{firstname}’. If someone else has the same first name and is in this workshop, each of you decide who should change their name for this to work. Or - if it’s easier - change the firstname to something else like john1.

For example, if your Wallaroo client was saved to the variable wl, then the command to create a new workspace workshop-workspace-sample is:

wl.create_workspace('workshop-workspace-sample')

When you’re done, list the workspaces. You did that in a previous step, so you can copy that here.

# blank space to create your workspace

print(wl.create_workspace('workshop-workspace-john-cv-medical'))

# list all the workspaces here

wl.list_workspaces()

Retrieve Workspace

In the above example, you saw this:

workspace = wl.create_workspace('sparkly-bunnies')

This creates the workspace sparkly-bunnies, then assigns it to the variable workspace. We can display that workspace and see what it looks like. For example:

workspace = wl.create_workspace('sparkly-bunnies')

print(workspace)
{'name': 'sparkly-bunnies', 'id': 9, 'archived': False, 'created_by': '66d3b2c4-9b22-4429-b16e-3bcdc1ac28e3', 'created_at': '2023-08-22T17:30:40.475785+00:00', 'models': [], 'pipelines': []}

If we had created a workspace earlier, and want to work with it, we will have to retrieve it. We do that with the wallaroo.Client.list_workspaces method, which returns a List. Then we can set the workspace we want to a variable.

Here’s an example. We start with list_workspaces:

wl.list_workspaces()
NameCreated AtUsersModelsPipelines
john.hummel@wallaroo.ai - Default Workspace2023-08-21 19:06:07[‘john.hummel@wallaroo.ai’]11
sparkly-bunnies2023-08-22 17:30:40[‘john.hummel@wallaroo.ai’]00
workshop-workspace-sample2023-08-22 17:38:07[‘john.hummel@wallaroo.ai’]00
workshop-workspace-john2023-08-22 17:38:13[‘john.hummel@wallaroo.ai’]00

In our case, we want to use sparkly-bunnies. If we count down from the top, it’s the 2nd in the List. Since lists start at 0, it’s as position 1. For those not experienced with Lists, here’s how it looks.

positionworkspace
0john.hummel@wallaroo.ai - Default Workspace
1sparkly-bunnies
2workshop-workspace-sample

And so on.

To store a specific workspace from list_workspaces, we assign it to a variable based on its position. If we want to store the workspace sparkly-bunnies to a variable workspace, we do it like so:

workspace = wl.list_workspaces()[1]
print(workspace)

{'name': 'sparkly-bunnies', 'id': 9, 'archived': False, 'created_by': '66d3b2c4-9b22-4429-b16e-3bcdc1ac28e3', 'created_at': '2023-08-22T17:30:40.475785+00:00', 'models': [], 'pipelines': []}

For those more familiar with lists, you can do things like filter by name like so:

def get_workspace(name, client):
    workspace = None
    for ws in client.list_workspaces():
        if ws.name() == name:
            workspace= ws

We’ll introduce helper functions like this later, but know for now if you know how to use a List, then you can retrieve a workspace.

Retrieve Workspace Exercise

Retrieve the workspace to a variable you created earlier through the following steps:

  1. List the workspaces.
  2. Determine the position of the workspace from the list - remember the list positions start at 0.
  3. Assign the workspace to a variable that you’ll use later.

For example, if the workspace is position 1 in the list_workspaces list, then you would retrieve the workspace like so:

workspace = wl.list_workspaces()[1]
# blank space to retrieve your workspace

display(wl.list_workspaces())
workspace = wl.list_workspaces()[-1]
print(workspace)
NameCreated AtUsersModelsPipelines
john.hummel@wallaroo.ai - Default Workspace2023-09-27 15:17:23['john.hummel@wallaroo.ai']12
cv-arm-edge-demo-jch2023-09-27 15:17:44['john.hummel@wallaroo.ai']11
workshop-workspace-john-102023-09-27 16:29:36['john.hummel@wallaroo.ai']21
workshop-workspace-john-052023-09-27 16:50:52['john.hummel@wallaroo.ai']11
workshop-workspace-john-cv-yolo2023-09-28 16:06:55['john.hummel@wallaroo.ai']21
retail2023-09-28 19:44:06['john.hummel@wallaroo.ai']11
workshop-workspace-john-cv-medical2023-09-28 20:22:21['john.hummel@wallaroo.ai']11
{'name': 'workshop-workspace-john-cv-medical', 'id': 10, 'archived': False, 'created_by': 'b030ff9c-41eb-49b4-afdf-2ccbecb6be5d', 'created_at': '2023-09-28T20:22:21.506043+00:00', 'models': [{'name': 'mitochondria-detector', 'versions': 1, 'owner_id': '""', 'last_update_time': datetime.datetime(2023, 9, 28, 20, 25, 2, 359367, tzinfo=tzutc()), 'created_at': datetime.datetime(2023, 9, 28, 20, 25, 2, 359367, tzinfo=tzutc())}], 'pipelines': [{'name': 'cv-mitochondria', 'create_time': datetime.datetime(2023, 9, 28, 20, 25, 17, 379732, tzinfo=tzutc()), 'definition': '[]'}]}

Set the Current Workspace

We mentioned earlier that when you login to Wallaroo, the SDK assigns you to the default workspace - the one named {your email address} - Default Workspace - replacing your email address in the front.

Usually you’ll want to work in some other workspace - perhaps one that you’re a part of with other users, or one you set up yourself for test purposes. It is highly recommended that workspaces be divided by project or some specific goal where the same models are used for different purposes.

We’ve gone over how to create a workspace, and how to retrieve a workspace that was previously created. Now we’ll use that to set our current workspace with the wallaroo.Client.set_current_workspace(workspace).

The current workspace is where your SDK commands are routed to. When you give the upload models command - they are uploaded to your current workspace. Build a pipeline? Associated to the current workspace.

So we’re going to make sure that what we’re doing is done in the right workspace with two commands:

  • wallaroo.Client.get_current_workspace(): Shows what the current workspace.
  • wallaroo.Client.set_current_workspace(workspace): Sets the current workspace to the target workspace.

For example, if your workspace is saved to a variable as shown in the previous step, we can change from the default workspace and set our current workspace to the new one as follows:

# show the current workspace
print(wl.get_current_workspace())
{'name': 'john.hummel@wallaroo.ai - Default Workspace', 'id': 1, 'archived': False, 'created_by': '66d3b2c4-9b22-4429-b16e-3bcdc1ac28e3', 'created_at': '2023-08-21T19:06:07.404363+00:00', 'models': [{'name': 'm1', 'versions': 1, 'owner_id': '""', 'last_update_time': datetime.datetime(2023, 8, 21, 19, 38, 36, 672465, tzinfo=tzutc()), 'created_at': datetime.datetime(2023, 8, 21, 19, 38, 36, 672465, tzinfo=tzutc())}], 'pipelines': [{'name': 'p1', 'create_time': datetime.datetime(2023, 8, 21, 19, 38, 44, 314377, tzinfo=tzutc()), 'definition': '[]'}]}

# change to the new workspace
wl.set_current_workspace(workspace)

# show the new current workspace
print(wl.get_current_workspace())
{'name': 'workshop-workspace-john', 'id': 12, 'archived': False, 'created_by': '66d3b2c4-9b22-4429-b16e-3bcdc1ac28e3', 'created_at': '2023-08-22T17:38:13.612187+00:00', 'models': [], 'pipelines': []}

Setting your current workspace to the one you want to work in is an important step. We highly recommend that once a Wallaroo client connection is established, the next task should be setting whatever workspace is the proper one to work in as the current workspace, then proceeding with any other tasks.

Set the Current Workspace Exercise

Previously you created a workspace and retrieved it to a variable. Using the * wallaroo.Client.get_current_workspace() and wallaroo.Client.set_current_workspace(workspace) methods:

  1. Get your current workspace.
  2. Set your current workspace to the one created in the previous steps.
  3. Get your current workspace again to verify that the change was made.

For example, if your Wallaroo client was stored as the variable wl, and your new workspace saved to the variable workspace, you can change your current workspace to the new one with the following:

wl.set_current_workspace(workspace)
wl.get_current_workspace()
print(wl.get_current_workspace())
wl.set_current_workspace(workspace)
print(wl.get_current_workspace())
{'name': 'john.hummel@wallaroo.ai - Default Workspace', 'id': 1, 'archived': False, 'created_by': 'b030ff9c-41eb-49b4-afdf-2ccbecb6be5d', 'created_at': '2023-09-27T15:17:23.923705+00:00', 'models': [{'name': 'v5s6', 'versions': 2, 'owner_id': '""', 'last_update_time': datetime.datetime(2023, 9, 28, 19, 23, 3, 425821, tzinfo=tzutc()), 'created_at': datetime.datetime(2023, 9, 28, 18, 55, 24, 860357, tzinfo=tzutc())}], 'pipelines': [{'name': 'cv-arm-edge', 'create_time': datetime.datetime(2023, 9, 27, 15, 17, 45, 593032, tzinfo=tzutc()), 'definition': '[]'}, {'name': 'retailimage', 'create_time': datetime.datetime(2023, 9, 28, 18, 55, 14, 954338, tzinfo=tzutc()), 'definition': '[]'}]}
{'name': 'workshop-workspace-john-cv-medical', 'id': 10, 'archived': False, 'created_by': 'b030ff9c-41eb-49b4-afdf-2ccbecb6be5d', 'created_at': '2023-09-28T20:22:21.506043+00:00', 'models': [{'name': 'mitochondria-detector', 'versions': 1, 'owner_id': '""', 'last_update_time': datetime.datetime(2023, 9, 28, 20, 25, 2, 359367, tzinfo=tzutc()), 'created_at': datetime.datetime(2023, 9, 28, 20, 25, 2, 359367, tzinfo=tzutc())}], 'pipelines': [{'name': 'cv-mitochondria', 'create_time': datetime.datetime(2023, 9, 28, 20, 25, 17, 379732, tzinfo=tzutc()), 'definition': '[]'}]}

Upload a Model

Now that we have our current workspace set, it’s time to start uploading some models.

We already have some ONNX models available in the folder ./models. All three do the same thing: predict house price values based on some values. We have those values stored in the folder ./data.

ML Models are uploaded to the current Wallaroo workspace the wallaroo.Client.upload_model method. Wallaroo supports different model types, as well as Arbitrary Python and containerized models.

For full details, see Wallaroo SDK Essentials Guide: Model Uploads and Registrations.

Wallaroo supports ONNX models as part of the default runtime, so these will run in Wallaroo without additional configurations.

When uploading models, the following is needed:

  • The name of the model. This needs to be unique across the workspace.
  • The path to the ML model file. For example, ./models/xgb_model.onnx.
  • The framework of the model. These are listed through the wallaroo.framework.Framework list. For these examples we will be using wallaroo.framework.Framework.ONNX to specify we are using ONNX models.
  • The input_schema and output_schema. For ONNX models, we can skip this. For non-native runtime models, that has to be specified in Apache Arrow schema format. See the Wallaroo SDK Essentials Guide: Model Uploads and Registrations for full details.

Here’s an example of uploading a model to a Wallaroo workspace and assigning it the name ‘house-price-prime’, with the Wallaroo client assigned to the variable wl, then retrieving the model version from Wallaroo once the upload it complete:

house_price_model_version = wl.upload_model('house-price-prime',
                                            './models/xgb_model.onnx',
                                            framework=wallaroo.framework.Framework.ONNX)
house_price_model_version
Namehouse-price-prime
Versioncc3ba784-ffdf-4a0f-982a-9a8ac4db8ba9
File Namexgb_model.onnx
SHA31e92d6ccb27b041a324a7ac22cf95d9d6cc3aa7e8263a229f7c4aec4938657c
Statusready
Image PathNone
Updated At2023-22-Aug 19:55:26

We store this new version of a model to the variable house_price_model_version. This is used for later processes involving pipeline deployments, generating assays, and so on.

In Wallaroo, you have the model, which is based on the name parameter. Each model has one or more versions.

IMPORTANT NOTE: Models in Wallaroo are organized by name. If a model is uploaded with the same name, it will create a new version of the model with the same name. For example, the following will create a model named house-price-prime with two versions.

# set the model from the XGB model converted to ONNX
house_price_model_version = wl.upload_model('house-price-prime',
                                            './models/xgb_model.onnx',
                                            framework=wallaroo.framework.Framework.ONNX)
print(house_price_model_version)
{'name': 'house-price-prime', 'version': '83d89260-9aac-41ea-b2b4-79aae48b5a65', 'file_name': 'xgb_model.onnx', 'image_path': None, 'last_update_time': datetime.datetime(2023, 8, 22, 19, 59, 2, 26718, tzinfo=tzutc())}

# create the new model version to the model converted from an RF model
house_price_model_version = wl.upload_model('house-price-prime',
                                            './models/rf_model.onnx',
                                            framework=wallaroo.framework.Framework.ONNX)
print(house_price_model_version)
{'name': 'house-price-prime', 'version': 'c86fd309-7c28-4e95-9d3e-831fefa51a12', 'file_name': 'rf_model.onnx', 'image_path': None, 'last_update_time': datetime.datetime(2023, 8, 22, 19, 59, 3, 381581, tzinfo=tzutc())}

Notice that the model is the same - house-price-prime - but the model version changes each time we do an upload_model. This allows you to change the model version to a totally different flavor and framework if you desire.

Upload a Model Exercise

For this exercise, upload the model /models/mobilenet.pt.onnx and assign it a name, with the framework =wallaroo.framework.Framework.ONNX. This model has an additional configuration option that we’ll include in the code below: batch_config="single", indicating that the model expects just one inference row at a time.

Here’s an example of uploading the mobilenet model.

from wallaroo.framework import Framework

wl.upload_model('mitochondria-detector',
                "./models/mitochondria_epochs_15.onnx",
                framework=Framework.ONNX)
from wallaroo.framework import Framework

wl.upload_model('mitochondria-detector',
                "../models/mitochondria_epochs_15.onnx",
                framework=Framework.ONNX)
Namemitochondria-detector
Version11a34cce-0eaa-4cdd-8739-43bd18b67a04
File Namemitochondria_epochs_15.onnx
SHAe80fcdaf563a183b0c32c027dcb3890a64e1764d6d7dcd29524cd270dd42e7bd
Statusready
Image PathNone
Updated At2023-29-Sep 17:40:06

Retrieve Model Version

Once a model is uploaded to Wallaroo, we can list the models in a workspace with the wallaroo.workspace.models() method. This returns a List of all of the models and how many versions are associated with that model.

Here’s an example:

workspace.models()
[{'name': 'house-price-prime', 'versions': 3, 'owner_id': '""', 'last_update_time': datetime.datetime(2023, 8, 22, 19, 59, 3, 381581, tzinfo=tzutc()), 'created_at': datetime.datetime(2023, 8, 22, 19, 55, 26, 603685, tzinfo=tzutc())}]

We can retrieve the model by specifying the position on the list. In our case, we only have one model, so we can retrieve it to a model by specifying position 0. For example:

my_model = workspace.models()[0]
my_model
Namehouse-price-prime
# of Versions3
Owner ID""
Last Updated2023-08-22 19:59:03.381581+00:00
Created At2023-08-22 19:55:26.603685+00:00

And finally - we retrieve what we really need, the model version by using the wallaroo.model.version() method, which returns the list of versions of the model:

my_model = workspace.models()[0]
my_model.versions()

[{'name': 'house-price-prime', 'version': 'cc3ba784-ffdf-4a0f-982a-9a8ac4db8ba9', 'file_name': 'xgb_model.onnx', 'image_path': None, 'last_update_time': datetime.datetime(2023, 8, 22, 19, 55, 26, 603685, tzinfo=tzutc())},
 {'name': 'house-price-prime', 'version': '83d89260-9aac-41ea-b2b4-79aae48b5a65', 'file_name': 'xgb_model.onnx', 'image_path': None, 'last_update_time': datetime.datetime(2023, 8, 22, 19, 59, 2, 26718, tzinfo=tzutc())},
 {'name': 'house-price-prime', 'version': 'c86fd309-7c28-4e95-9d3e-831fefa51a12', 'file_name': 'rf_model.onnx', 'image_path': None, 'last_update_time': datetime.datetime(2023, 8, 22, 19, 59, 3, 381581, tzinfo=tzutc())}]

As you can see - the most recent version is the last element in the list, or the [-1] position. We can now retrieve the most recent version of the model version as follows:

my_model = workspace.models()[0]
my_model_version = my_model.versions()[-1]
my_model_version
Namehouse-price-prime
Versionc86fd309-7c28-4e95-9d3e-831fefa51a12
File Namerf_model.onnx
SHAe22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6
Statusready
Image PathNone
Updated At2023-22-Aug 19:59:03

The model version is an important concept - that is what is added to a pipeline for deployments and other uses.

Retrieve Model Version Exercise

This exercise will have you retrieving the model you uploaded earlier. For example, if the Wallaroo client was stored as wl, and the workspace saved to workspace, then the command to get the current model version would be:

my_model = workspace.models()[0]
my_model_version = my_model.versions()[-1]
my_model_version
# blank space to retrieve the model version and store it

my_model = workspace.models()[0]
my_model_version = my_model.versions()[-1]
my_model_version
Namemitochondria-detector
Version11a34cce-0eaa-4cdd-8739-43bd18b67a04
File Namemitochondria_epochs_15.onnx
SHAe80fcdaf563a183b0c32c027dcb3890a64e1764d6d7dcd29524cd270dd42e7bd
Statusready
Image PathNone
Updated At2023-29-Sep 17:40:06

Build a Pipeline

Pipelines are the method of taking submitting data and processing that data through the models. Each pipeline can have one or more steps that submit the data from the previous step to the next one. Information can be submitted to a pipeline as a file, or through the pipeline’s URL.

Each pipeline step is a model version. These can be ML models like we uploaded earlier, or they can be Python scripts that manipulate the data into a format needed for another model in the chain.

When an inference is performed, data is submitted to the pipeline. The pipeline then submits the data to the first step, receives the output, then transmits that data to the next step. When all steps are complete, the pipeline returns the final values to the requesting client.

For this workshop, we will focus one a very simple pipeline: one ML model.

Pipeline are created in the current workspace with the wallaroo.Client.build_pipeline(name) command, where the name is unique to the workspace. For example, to create the pipeline named houseprice-estimator and the Wallaroo client is saved to wl the command would be:

pipeline = wl.build_pipeline('houseprice-estimator')
pipeline

name | houseprice-estimator
created | 2023-08-22 20:39:35.853683+00:00
last_updated | 2023-08-22 20:39:35.853683+00:00
deployed | (none)
tags |
versions | f42c0457-e4f3-4370-b152-0a220347de11
steps |

Just like models, pipelines have version. Each time pipeline steps are changed, a new version is created.

See Wallaroo SDK Essentials Guide: Pipeline Management for full details.

Build a Pipeline

Build your own pipeline! Use the wallaroo.Client.build_pipeline(name) command and create a pipeline named houseprice-pipeline. Recall that this creates the pipeline in the current workspace, so verify that the current workspace is the one you want to create a pipeline in.

For example, if the Wallaroo client is saved to the variable wl, the command would be:

wl.build_pipeline('cv-retail')
# blank space for you to create the pipeline

wl.build_pipeline('cv-mitochondria')
namecv-mitochondria
created2023-09-28 20:25:17.379732+00:00
last_updated2023-09-29 17:40:15.164148+00:00
deployedFalse
tags
versionscf4bdfb4-1eec-46f8-9af4-b16dea894de6, 507cb1eb-8034-4b5b-bc96-2427730a6407, 50ed1d2f-6dba-411c-9579-1090791b33bd, 6208c9cf-fcd6-4b20-bbfc-f6ce714596e3
stepsmitochondria-detector
publishedFalse

Retrieve a Pipeline

Pipelines that are associated with a workspace are retrieved wallaroo.Client.list_pipelines method - this returns a List of pipelines. From there, a variable is assigned to the pipeline in the list we want to work with.

The list_pipelines method returns a list of pipelines and their details as follows:

wl.list_pipelines()
namecreatedlast_updateddeployedtagsversionssteps
houseprice-estimator2023-22-Aug 20:39:352023-22-Aug 20:39:35(unknown)f42c0457-e4f3-4370-b152-0a220347de11
biolabspipeline2023-22-Aug 16:07:202023-22-Aug 16:24:40False4c6dceb7-e692-4b8b-b615-4f7873eb020b, 59d0babe-bc1d-4dbb-959f-711c74f7b05d, ae834c0d-7a5b-4f87-9e2e-1f06f3cd25e7, 7c438222-28d8-4fca-9a70-eabee8a0fac5biolabsmodel
biolabspipeline2023-22-Aug 16:03:332023-22-Aug 16:03:38False4e103a7d-cd4d-464b-b182-61d4041518a8, ec2a0fd6-21d4-4843-b7c3-65b1e5be1b85, 516f3848-be98-40d7-8564-a1e48eecb7a8biolabsmodel
biolabspipelinegomj2023-22-Aug 15:11:122023-22-Aug 15:42:44False1dc9f89f-82aa-4a71-b21a-75dc8d5e4e51, 152d12f2-1200-46ad-ad04-60078c5aa284, 6ca59ffd-802e-4ad5-bd9a-35146b9fbda5, bdab08cc-3e99-4afc-b22d-657e33b76f29, 3c8feb0d-3124-4018-8dfa-06162156d51ebiolabsmodelgomj
edge-pipeline2023-21-Aug 20:54:372023-22-Aug 19:06:46False2be013d9-a438-453c-a013-3fd8e6218394, a02b6af5-4235-42af-92c6-5ae678b35be4, e721ccad-11d8-4874-8388-4211c4957d18, d642e766-cffb-451f-b197-e058bedbdd5f, eb586aba-4908-4bff-84e1-bdeb1fa4b7d3, 2163d718-a5ea-41e3-b69f-095efa858462ccfraud
p12023-21-Aug 19:38:442023-21-Aug 19:38:44(unknown)5f93e90a-e8d6-4e8a-8a1a-22eee80a3e13, 5f78247f-7bf9-445b-98a6-e146fb22b8e9

Just like with a model version, we can set a variable to the pipeline by assigning it from its position in the list. In my case, if we want to retrieve the pipeline houseprice-estimator at position 0, we do so as follows:

this_pipeline = wl.list_pipelines()[0]
this_pipeline

Retrieve a Pipeline Exercise

For this exercise, retrieve your the pipeline you built in the previous step and store it into the variable my_pipeline. You’ll need to use list_pipelines() to get the list, then see where the position of the pipeline is in the list and assign it to the variable. Here’s an example if the Wallaroo client is stored in the variable wl and our pipeline is at position 0 of the list_pipelines list, the command would be:

my_pipeline = wl.list_pipelines()[0]
my_pipeline
# empty space to retrieve your pipeline

my_pipeline = wl.list_pipelines()[0]
my_pipeline
namecv-mitochondria
created2023-09-28 20:25:17.379732+00:00
last_updated2023-09-29 17:40:15.164148+00:00
deployedFalse
tags
versionscf4bdfb4-1eec-46f8-9af4-b16dea894de6, 507cb1eb-8034-4b5b-bc96-2427730a6407, 50ed1d2f-6dba-411c-9579-1090791b33bd, 6208c9cf-fcd6-4b20-bbfc-f6ce714596e3
stepsmitochondria-detector
publishedFalse

Add Model Step

Models are added to a pipeline as pipeline steps. There are different kinds of pipeline steps that can host one or more models.

For this workshop, we will use the method wallaroo.pipeline.add_model_step(model_version). This adds a single step to a Pipeline. Pipeline steps start at 0 and increment from there. We can see the steps in our pipeline with the wallaroo.pipeline.steps() method.

# add modelA as a pipeline steps
pipeline.add_model_step(modelA)

# display the steps
pipeline.steps()

[{'ModelInference': {'models': [{'name': 'house-price-prime', 'version': 'c86fd309-7c28-4e95-9d3e-831fefa51a12', 'sha': 'e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6'}]}}]

For now, there’s three commands you should know:

  • wallaroo.pipelineadd_model_step(model_version): Add a step to a pipeline from a model version.
  • wallaroo.pipeline.steps(): Display the current steps in the pipeline.
  • wallaroo.pipeline.clear(): Clear all pipeline steps.

Pipeline steps are not saved in the Wallaroo instance until the pipeline has been deployed - more on that shortly. So you can add steps, clear them, add new ones - they all stay in your local script until you issue the command to deploy the pipeline. More on that later.

The second thing to watch out for it every time add_model_step is performed on a pipeline, another step is created. For example, if we have a pipeline with two models, modelA and modelB, then the following creates two steps in the same pipelines:

# clear the steps
pipeline.clear()

# add modelA then modelB as pipeline steps
pipeline.add_model_step(modelA)
pipeline.add_model_step(modelB)

# display the steps
pipeline.steps()

[{'ModelInference': {'models': [{'name': 'house-price-prime', 'version': 'c86fd309-7c28-4e95-9d3e-831fefa51a12', 'sha': 'e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6'}]}},
 {'ModelInference': {'models': [{'name': 'house-price-gbr', 'version': '248a6eab-b159-4821-830b-22cc137a1ace', 'sha': 'ed6065a79d841f7e96307bb20d5ef22840f15da0b587efb51425c7ad60589d6a'}]}}]

This means the data from the inference is fed first to modelA, and that output is fed into modelB. The problem is if the input data from modelA doesn’t match what modelB expects, the process will fail.

Because of this, check your pipeline steps before you deploy a pipeline, clear them if you need to.

Add Pipeline Step Exercise

We have our model version uploaded from the previous steps, and we have our pipeline. Time to put them together and create a pipeline step with our model version.

Just for practice, do the following:

  1. Clear the pipeline steps.
  2. Add the sample model uploaded earlier. In our examples, that was my_model_version.
  3. Show the current pipeline steps.

Here’s an example with the Wallaroo client stored to wl, with the pipeline my_pipeline and my_model_version:

my_pipeline.clear()
my_pipeline.add_model_step(my_model_version)
my_pipeline.steps()
my_pipeline.clear()
my_pipeline.add_model_step(my_model_version)
my_pipeline.steps()
[{'ModelInference': {'models': [{'name': 'mitochondria-detector', 'version': '11a34cce-0eaa-4cdd-8739-43bd18b67a04', 'sha': 'e80fcdaf563a183b0c32c027dcb3890a64e1764d6d7dcd29524cd270dd42e7bd'}]}}]

Deploy a Pipeline

Now we reach what we’ve been aiming for: deploying a pipeline.

By now, you’ve seen how workspaces contain the models, pipelines, and other artifacts. You’ve uploaded a model and retrieved the latest version of the model. You’ve built a pipeline and added the model version as a pipeline step.

Now we will deploy the pipeline. Deploying a pipeline allocated resources from the cluster to that pipeline for it’s use. The amount of resources has a default value of 4 CPUs, but for this workshop we’ll be adjusting that to just 0.5 cpus and 1 GB RAM per pipeline.

  • IMPORTANT NOTE: Please stick to these resource configurations when using a Wallaroo instance with other users. Otherwise, pipelines might not deploy if the available resources are used up.

To deploy a pipeline, we do two things:

  • Create a deployment configuration: This is an optional step, but we will make it mandatory for this workshop to allow other users to work in the same Wallaroo instance without running out of resources.
  • Deploy the pipeline with the deployment configuration: This is the active step that saves the pipeline steps, and allocates system resources to the pipeline for performing inferences.

Deployment configurations are made with the wallaroo.DeploymentConfigBuilder() class, and then we assign the resource settings from there. This is saved to a variable so we can apply it to our pipeline deployment.

Here’s an example of setting up a deployment with just 0.5 cpu and 1Gi RAM:

deploy_config = wallaroo.DeploymentConfigBuilder().replica_count(1).cpus(0.5).memory("1Gi").build()

Notice the replica_count(1) configuration - this tells Wallaroo to only spin up one replica for this pipeline. In a production environment, we could spin multiple replicas either manually or automatically as more resources are needed to improve performance.

Now we deploy the pipeline with our deployment configuration with the wallaroo.pipeline.deploy(deploy_configuration) method. If our pipeline variable is my_pipeline, then we would deploy it as follows:

deploy_config = wallaroo.DeploymentConfigBuilder().replica_count(1).cpus(0.5).memory("1Gi").build()
my_pipeline.deploy(deployment_config=deploy_config)

We can check the status of the pipeline deployment with the wallaroo.pipeline.status() method:

my_pipeline.status()

{'status': 'Running',
 'details': [],
 'engines': [{'ip': '10.244.3.83',
   'name': 'engine-6d4fccf5cb-dmwfl',
   'status': 'Running',
   'reason': None,
   'details': [],
   'pipeline_statuses': {'pipelines': [{'id': 'houseprice-estimator',
      'status': 'Running'}]},
   'model_statuses': {'models': [{'name': 'house-price-prime',
      'version': 'c86fd309-7c28-4e95-9d3e-831fefa51a12',
      'sha': 'e22a0831aafd9917f3cc87a15ed267797f80e2afa12ad7d8810ca58f173b8cc6',
      'status': 'Running'}]}}],
 'engine_lbs': [{'ip': '10.244.4.100',
   'name': 'engine-lb-584f54c899-sswnw',
   'status': 'Running',
   'reason': None,
   'details': []}],
 'sidekicks': []}

Deploy a Pipeline Exercise

This exercise will have you deploy your pipeline with the deployment settings we listed above. For example, if your pipeline was called my_pipeline, then your deployment will look like this:

deploy_config = wallaroo.DeploymentConfigBuilder().replica_count(1).cpus(0.5).memory("1Gi").build()
my_pipeline.deploy(deployment_config=deploy_config)
deploy_config = wallaroo.DeploymentConfigBuilder().replica_count(1).cpus(0.5).memory("1Gi").build()
my_pipeline.deploy(deployment_config=deploy_config)
my_pipeline.status()
{'status': 'Running',
 'details': [],
 'engines': [{'ip': '10.244.3.236',
   'name': 'engine-669fcff449-w5j62',
   'status': 'Running',
   'reason': None,
   'details': [],
   'pipeline_statuses': {'pipelines': [{'id': 'cv-mitochondria',
      'status': 'Running'}]},
   'model_statuses': {'models': [{'name': 'mitochondria-detector',
      'version': '11a34cce-0eaa-4cdd-8739-43bd18b67a04',
      'sha': 'e80fcdaf563a183b0c32c027dcb3890a64e1764d6d7dcd29524cd270dd42e7bd',
      'status': 'Running'}]}}],
 'engine_lbs': [{'ip': '10.244.4.12',
   'name': 'engine-lb-584f54c899-d4z4h',
   'status': 'Running',
   'reason': None,
   'details': []}],
 'sidekicks': []}

Pipeline Inference with DataFrames

Wallaroo deployed pipelines accept three types of data:

  • pandas DataFrames
  • Apache Arrow

We do this with one of two commands on a deployed pipeline.

  • wallaroo.pipeline.infer(input): Submits either JSON, a DataFrame, or Apache Arrow to the pipeline for inferences.
  • wallaroo.pipeline.infer_from_file(path): Submits either a JSON, a DataFrame in pandas Record format, or an Apache Arrow binary file inferences.

We’ll start with a single DataFrame stored in the file: ./data/singleton.df.json, which contains input data as a tensor:

[
    {
        "tensor": [
            4.0,
            3.0,
            3710.0,
            20000.0,
            2.0,
            0.0,
            2.0,
            5.0,
            10.0,
            2760.0,
            950.0,
            47.6696014404,
            -122.2610015869,
            3970.0,
            20000.0,
            79.0,
            0.0,
            0.0
        ]
    }
]

This translates into a pandas DataFrame, which is submitted with the pipeline.infer method.

df = pd.DataFrame([
    {
        "tensor": [
            4.0,
            3.0,
            3710.0,
            20000.0,
            2.0,
            0.0,
            2.0,
            5.0,
            10.0,
            2760.0,
            950.0,
            47.6696014404,
            -122.2610015869,
            3970.0,
            20000.0,
            79.0,
            0.0,
            0.0
        ]
    }
])

pipeline.infer(df)

When we use infer, Wallaroo determines whether the object is a pandas DataFrame or Apache arrow, and handles the inferences accordingly.

The data received through the SDK is always of the same type submitted: Submit a DataFrame, get a DataFrame with the data back. Submit an Arrow table file, get an Arrow table back. Here’s an example of submitting our sample DataFrame through a pipeline saved to the variable pipeline:

result = pipeline.infer(df)
display(result)
timein.tensorout.variablecheck_failures
02023-08-23 15:02:41.452[4.0, 3.0, 3710.0, 20000.0, 2.0, 0.0, 2.0, 5.0, 10.0, 2760.0, 950.0, 47.6696014404, -122.2610015869, 3970.0, 20000.0, 79.0, 0.0, 0.0][1514079.4]0

Let’s break down each of these fields:

  • Index (unnamed): This field doesn’t have a label, but in the example above this is the index. We only have one submission, so we have one result. If we had 20 inputs, we’d have 20 inference results, and each result would be aligned with each row we sent as an input.
  • time: The date and time the inference request.
  • in.{variable}: Every input to the inference request is listed as in.{variable_name}. For example, if our inputs were house_size_in_square_feet and year_house_built, then the inputs would be listed as in.house_size_in_square_feet and in.year_house_built.
  • out.{variable}: Every output to the inference is listed as out.{variable_name}. Our sample model outputs just one output: variable, but others such might output estimated_house_price, initial_offer_price, and in the inference result those would be listed as out.estimated_house_price and out.initial_offer_price.
  • check_failures: Indicates if any validation checks failed. This is covered in later sessions.

There’s additional data that an inference has that is retrieved by requesting it. For full details, see the Wallaroo SDK Essentials Guide: Inference Management.

Pipeline Inference with Files Exercise

The computer vision models accepts tensor inputs. We have converted the file ./patches/ms-01-atl-3-22-23_9-50/images/image_0_21.tif" to a DataFrame. See the helper file TiffImageUtils.py for the loadImageAndConvertTiff that converts TIFF images to DataFrames. This DataFrame of the image is stored to ./data/image_0_21.tif.df.json. We can perform a sample inferece using the wallaroo.pipeline.infer_from_file method like so:

my_pipeline.infer_from_file('./data/image_0_21.tif.df.json')
my_pipeline.infer_from_file('../data/image_0_21.tif.df.json')
timein.tensorout.conv2d_37check_failures
02023-09-29 18:13:48.458[0.0572924651, 0.0555159546, 0.0621778691, 0.0...[0.073827654, 0.04537511, 0.022077948, 0.02752...0
deploy_url = my_pipeline._deployment._url()

headers = wl.auth.auth_header()

headers['Content-Type']='application/json; format=pandas-records'
headers['Accept']='application/json; format=pandas-records'

dataFile = '../data/image_0_21.tif.df.json'

# test_df = pd.read_json('../data/image_0_21.tif.df.json')
# test_df.to_json('../data/image_0_21.tif.df.json', orient="records")
# display(test_df)

print(f'''
!curl -X POST {deploy_url} \\
    -H "Authorization:{headers['Authorization']}" \\
    -H "Content-Type:{headers['Content-Type']}" \\
    -H "Accept:{headers['Accept']}" \\
    --data @{dataFile} > curl_response.df
      ''')
!curl -X POST https://doc-test.api.wallarooexample.ai/v1/api/pipelines/infer/cv-mitochondria-25/cv-mitochondria \
    -H "Authorization:Bearer eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJURV9fTFgzRTZHZVdkRlBXaXVwaXN0OU0ySHR3dFVZbzk2NkJCUG1CUjJvIn0.eyJleHAiOjE2OTYwMTEyNjYsImlhdCI6MTY5NjAxMTIwNiwiYXV0aF90aW1lIjoxNjk2MDA5MDA4LCJqdGkiOiI5MGQwNWZjNC1iYTc3LTQ5ZjItYWY4Yi03ZDQ3ZDQyZDFiZGUiLCJpc3MiOiJodHRwczovL2RvYy10ZXN0LmtleWNsb2FrLndhbGxhcm9vY29tbXVuaXR5Lm5pbmphL2F1dGgvcmVhbG1zL21hc3RlciIsImF1ZCI6WyJtYXN0ZXItcmVhbG0iLCJhY2NvdW50Il0sInN1YiI6ImIwMzBmZjljLTQxZWItNDliNC1hZmRmLTJjY2JlY2I2YmU1ZCIsInR5cCI6IkJlYXJlciIsImF6cCI6InNkay1jbGllbnQiLCJzZXNzaW9uX3N0YXRlIjoiMTBlYTU4MGEtYTQyOS00YTkwLThmMzUtNTkwMmZmNGViMzQzIiwiYWNyIjoiMSIsInJlYWxtX2FjY2VzcyI6eyJyb2xlcyI6WyJkZWZhdWx0LXJvbGVzLW1hc3RlciIsIm9mZmxpbmVfYWNjZXNzIiwidW1hX2F1dGhvcml6YXRpb24iXX0sInJlc291cmNlX2FjY2VzcyI6eyJtYXN0ZXItcmVhbG0iOnsicm9sZXMiOlsibWFuYWdlLXVzZXJzIiwidmlldy11c2VycyIsInF1ZXJ5LWdyb3VwcyIsInF1ZXJ5LXVzZXJzIl19LCJhY2NvdW50Ijp7InJvbGVzIjpbIm1hbmFnZS1hY2NvdW50IiwibWFuYWdlLWFjY291bnQtbGlua3MiLCJ2aWV3LXByb2ZpbGUiXX19LCJzY29wZSI6InByb2ZpbGUgZW1haWwiLCJzaWQiOiIxMGVhNTgwYS1hNDI5LTRhOTAtOGYzNS01OTAyZmY0ZWIzNDMiLCJlbWFpbF92ZXJpZmllZCI6ZmFsc2UsImh0dHBzOi8vaGFzdXJhLmlvL2p3dC9jbGFpbXMiOnsieC1oYXN1cmEtdXNlci1pZCI6ImIwMzBmZjljLTQxZWItNDliNC1hZmRmLTJjY2JlY2I2YmU1ZCIsIngtaGFzdXJhLWRlZmF1bHQtcm9sZSI6InVzZXIiLCJ4LWhhc3VyYS1hbGxvd2VkLXJvbGVzIjpbInVzZXIiXSwieC1oYXN1cmEtdXNlci1ncm91cHMiOiJ7fSJ9LCJuYW1lIjoiSm9obiBIYW5zYXJpY2siLCJwcmVmZXJyZWRfdXNlcm5hbWUiOiJqb2huLmh1bW1lbEB3YWxsYXJvby5haSIsImdpdmVuX25hbWUiOiJKb2huIiwiZmFtaWx5X25hbWUiOiJIYW5zYXJpY2siLCJlbWFpbCI6ImpvaG4uaHVtbWVsQHdhbGxhcm9vLmFpIn0.K9oClU6gK6hWf1P__HYKr2GWn08T08h16Y2nA9fccp6MDKZzOb2RW5tdBJ9AtzkmMeChBG_TG-BLLsnZRwjpi_0V7vphMgxkp88p1QhbxIdgzQA5enGDDR_183DC0JC9ZGiRTTk3jCp9qoIgygN67959gONhGanH19S1SsOdNWzNltXcdE8lzclwutyKeR4bXMplgKXZP1VxGuM1SBN0SfmC1MPCR50wn8vI_bko011OyCJhd48OUUFuRvdjFjLqcD6J1Wqtc8mYvkphYBwuc2jmXwpW3vyIEQKs7NMATHXFH925awloE27XCT8sKLUvRGFN_MxVAuOPuiL9hJrU3w" \
    -H "Content-Type:application/json; format=pandas-records" \
    -H "Accept:application/json; format=pandas-records" \
    --data @../data/image_0_21.tif.df.json > curl_response.df
!curl -X POST https://doc-test.api.wallarooexample.ai/v1/api/pipelines/infer/cv-mitochondria-25/cv-mitochondria \
    -H "Authorization:Bearer eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJURV9fTFgzRTZHZVdkRlBXaXVwaXN0OU0ySHR3dFVZbzk2NkJCUG1CUjJvIn0.eyJleHAiOjE2OTYwMTEyNjYsImlhdCI6MTY5NjAxMTIwNiwiYXV0aF90aW1lIjoxNjk2MDA5MDA4LCJqdGkiOiI5MGQwNWZjNC1iYTc3LTQ5ZjItYWY4Yi03ZDQ3ZDQyZDFiZGUiLCJpc3MiOiJodHRwczovL2RvYy10ZXN0LmtleWNsb2FrLndhbGxhcm9vY29tbXVuaXR5Lm5pbmphL2F1dGgvcmVhbG1zL21hc3RlciIsImF1ZCI6WyJtYXN0ZXItcmVhbG0iLCJhY2NvdW50Il0sInN1YiI6ImIwMzBmZjljLTQxZWItNDliNC1hZmRmLTJjY2JlY2I2YmU1ZCIsInR5cCI6IkJlYXJlciIsImF6cCI6InNkay1jbGllbnQiLCJzZXNzaW9uX3N0YXRlIjoiMTBlYTU4MGEtYTQyOS00YTkwLThmMzUtNTkwMmZmNGViMzQzIiwiYWNyIjoiMSIsInJlYWxtX2FjY2VzcyI6eyJyb2xlcyI6WyJkZWZhdWx0LXJvbGVzLW1hc3RlciIsIm9mZmxpbmVfYWNjZXNzIiwidW1hX2F1dGhvcml6YXRpb24iXX0sInJlc291cmNlX2FjY2VzcyI6eyJtYXN0ZXItcmVhbG0iOnsicm9sZXMiOlsibWFuYWdlLXVzZXJzIiwidmlldy11c2VycyIsInF1ZXJ5LWdyb3VwcyIsInF1ZXJ5LXVzZXJzIl19LCJhY2NvdW50Ijp7InJvbGVzIjpbIm1hbmFnZS1hY2NvdW50IiwibWFuYWdlLWFjY291bnQtbGlua3MiLCJ2aWV3LXByb2ZpbGUiXX19LCJzY29wZSI6InByb2ZpbGUgZW1haWwiLCJzaWQiOiIxMGVhNTgwYS1hNDI5LTRhOTAtOGYzNS01OTAyZmY0ZWIzNDMiLCJlbWFpbF92ZXJpZmllZCI6ZmFsc2UsImh0dHBzOi8vaGFzdXJhLmlvL2p3dC9jbGFpbXMiOnsieC1oYXN1cmEtdXNlci1pZCI6ImIwMzBmZjljLTQxZWItNDliNC1hZmRmLTJjY2JlY2I2YmU1ZCIsIngtaGFzdXJhLWRlZmF1bHQtcm9sZSI6InVzZXIiLCJ4LWhhc3VyYS1hbGxvd2VkLXJvbGVzIjpbInVzZXIiXSwieC1oYXN1cmEtdXNlci1ncm91cHMiOiJ7fSJ9LCJuYW1lIjoiSm9obiBIYW5zYXJpY2siLCJwcmVmZXJyZWRfdXNlcm5hbWUiOiJqb2huLmh1bW1lbEB3YWxsYXJvby5haSIsImdpdmVuX25hbWUiOiJKb2huIiwiZmFtaWx5X25hbWUiOiJIYW5zYXJpY2siLCJlbWFpbCI6ImpvaG4uaHVtbWVsQHdhbGxhcm9vLmFpIn0.K9oClU6gK6hWf1P__HYKr2GWn08T08h16Y2nA9fccp6MDKZzOb2RW5tdBJ9AtzkmMeChBG_TG-BLLsnZRwjpi_0V7vphMgxkp88p1QhbxIdgzQA5enGDDR_183DC0JC9ZGiRTTk3jCp9qoIgygN67959gONhGanH19S1SsOdNWzNltXcdE8lzclwutyKeR4bXMplgKXZP1VxGuM1SBN0SfmC1MPCR50wn8vI_bko011OyCJhd48OUUFuRvdjFjLqcD6J1Wqtc8mYvkphYBwuc2jmXwpW3vyIEQKs7NMATHXFH925awloE27XCT8sKLUvRGFN_MxVAuOPuiL9hJrU3w" \
    -H "Content-Type:application/json; format=pandas-records" \
    -H "Accept:application/json; format=pandas-records" \
    --data @../data/image_0_21.tif.df.json > curl_response.df
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 2338k  100 1513k  100  824k  1049k   571k  0:00:01  0:00:01 --:--:-- 1624k--:--:-- --:--:-- --:--:--  917k

Undeploying Your Pipeline

You should always undeploy your pipelines when you are done with them, or don’t need them for a while. This releases the resources that the pipeline is using for other processes to use. You can always redeploy the pipeline when you need it again. As a reminder, here are the commands to deploy and undeploy a pipeline:


# when the pipeline is deployed, it's ready to receive data and infer
pipeline.deploy()

# "turn off" the pipeline and releaase its resources
pipeline.undeploy()

If you are continuing on to the next notebook now, you can leave the pipeline deployed to keep working; but if you are taking a break, then you should undeploy.

## blank space to undeploy the pipeline, if needed

my_pipeline.undeploy()
namecv-mitochondria
created2023-09-28 20:25:17.379732+00:00
last_updated2023-09-29 18:08:12.573990+00:00
deployedFalse
tags
versions4b6dab7d-b3ff-4f14-8425-7d9b6de76637, 66e72bc0-a3e3-4872-bc20-19b992c194b4, cf4bdfb4-1eec-46f8-9af4-b16dea894de6, 507cb1eb-8034-4b5b-bc96-2427730a6407, 50ed1d2f-6dba-411c-9579-1090791b33bd, 6208c9cf-fcd6-4b20-bbfc-f6ce714596e3
stepsmitochondria-detector
publishedFalse

Congratulations!

You have now

  • Created a workspace and set it as the current workspace.
  • Uploaded an ONNX model.
  • Created a Wallaroo pipeline, and set the most recent version of the uploaded model as a pipeline step.
  • Successfully send data to your pipeline for inference through the SDK and through an API call.

3.1.2 - Healthcare: Data Connections

How to use Wallaroo to set data connections within Wallaroo for inferencing and other use cases.

This tutorial and the assets can be downloaded as part of the Wallaroo Workshops repository.

Wallaroo Connections are definitions set by MLOps engineers that are used by other Wallaroo users for connection information to a data source.

This provides MLOps engineers a method of creating and updating connection information for data stores: databases, Kafka topics, etc. Wallaroo Connections are composed of three main parts:

  • Name: The unique name of the connection.
  • Type: A user defined string that designates the type of connection. This is used to organize connections.
  • Details: Details are a JSON object containing the information needed to make the connection. This can include data sources, authentication tokens, etc.

Wallaroo Connections are only used to store the connection information used by other processes to create and use external connections. The user still has to provide the libraries and other elements to actually make and use the conneciton.

The primary advantage is Wallaroo connections allow scripts and other code to retrieve the connection details directly from their Wallaroo instance, then refer to those connection details. They don’t need to know what those details actually - they can refer to them in their code to make their code more flexible.

For this step, we will use a Google BigQuery dataset to retrieve the inference information, predict the next month of sales, then store those predictions into another table. This will use the Wallaroo Connection feature to create a Connection, assign it to our workspace, then perform our inferences by using the Connection details to connect to the BigQuery dataset and tables.

Prerequisites

  • A Wallaroo instance version 2023.2.1 or greater.

References

Preliminaries

In the blocks below we will preload some required libraries.

For convenience, the following helper functions are defined to retrieve previously created workspaces, models, and pipelines:

  • get_workspace(name, client): This takes in the name and the Wallaroo client being used in this session, and returns the workspace matching name. If no workspaces are found matching the name, raises a KeyError and returns None.
  • get_model_version(model_name, workspace): Retrieves the most recent model version from the model matching the model_name within the provided workspace. If no model matches that name, raises a KeyError and returns None.
  • get_pipeline(pipeline_name, workspace): Retrieves the most pipeline from the workspace matching the pipeline_name within the provided workspace. If no model matches that name, raises a KeyError and returns None.
import json
import os
import datetime

import wallaroo
from wallaroo.object import EntityNotFoundError
from wallaroo.framework import Framework

# used to display dataframe information without truncating
from IPython.display import display
import pandas as pd
import numpy as np

pd.set_option('display.max_colwidth', None)

import time
import pyarrow as pa
## convenience functions from the previous notebooks

# return the workspace called <name> through the Wallaroo client.
def get_workspace(name, client):
    workspace = None
    for ws in client.list_workspaces():
        if ws.name() == name:
            workspace= ws
            return workspace
    # if no workspaces were found
    if workspace==None:
        raise KeyError(f"Workspace {name} was not found.")
    return workspace

# returns the most recent model version in a workspace for the matching `model_name`
def get_model_version(model_name, workspace):
    modellist = workspace.models()
    model_version = [m.versions()[-1] for m in modellist if m.name() == model_name]
    # if no models match, return None
    if len(modellist) <= 0:
        raise KeyError(f"Model {mname} not found in this workspace")
        return None
    return model_version[0]

# get a pipeline by name in the workspace
def get_pipeline(pipeline_name, workspace):
    plist = workspace.pipelines()
    pipeline = [p for p in plist if p.name() == pipeline_name]
    if len(pipeline) <= 0:
        raise KeyError(f"Pipeline {pipeline_name} not found in this workspace")
        return None
    return pipeline[0]

Connect to the Wallaroo Instance

The first step is to connect to Wallaroo through the Wallaroo client. The Python library is included in the Wallaroo install and available through the Jupyter Hub interface provided with your Wallaroo environment.

This is accomplished using the wallaroo.Client() command, which provides a URL to grant the SDK permission to your specific Wallaroo environment. When displayed, enter the URL into a browser and confirm permissions. Store the connection into a variable that can be referenced later.

If logging into the Wallaroo instance through the internal JupyterHub service, use wl = wallaroo.Client(). For more information on Wallaroo Client settings, see the Client Connection guide.

## blank space to log in 

wl = wallaroo.Client()
Please log into the following URL in a web browser:
https://doc-test.keycloak.wallarooexample.ai/auth/realms/master/device?user_code=WVDT-DNFQ

Login successful!

Set Configurations

Set the workspace, pipeline, and model used from Notebook 1. The helper functions will make this task easier.

Set Configurations References

# retrieve the previous workspace, model, and pipeline version

workspace_name = "workshop-workspace-john-cv-medical"

workspace = get_workspace(workspace_name, wl)

# set your current workspace to the workspace that you just created
wl.set_current_workspace(workspace)

# optionally, examine your current workspace
wl.get_current_workspace()

model_name = 'mitochondria-detector'

prime_model_version = get_model_version(model_name, workspace)

pipeline_name = 'cv-mitochondria'

pipeline = get_pipeline(pipeline_name, workspace)

display(workspace)
display(prime_model_version)
display(pipeline)
{'name': 'workshop-workspace-john-cv-medical', 'id': 10, 'archived': False, 'created_by': 'b030ff9c-41eb-49b4-afdf-2ccbecb6be5d', 'created_at': '2023-09-28T20:22:21.506043+00:00', 'models': [{'name': 'mitochondria-detector', 'versions': 2, 'owner_id': '""', 'last_update_time': datetime.datetime(2023, 9, 29, 17, 40, 6, 205146, tzinfo=tzutc()), 'created_at': datetime.datetime(2023, 9, 28, 20, 25, 2, 359367, tzinfo=tzutc())}], 'pipelines': [{'name': 'cv-mitochondria', 'create_time': datetime.datetime(2023, 9, 28, 20, 25, 17, 379732, tzinfo=tzutc()), 'definition': '[]'}]}
Namemitochondria-detector
Version11a34cce-0eaa-4cdd-8739-43bd18b67a04
File Namemitochondria_epochs_15.onnx
SHAe80fcdaf563a183b0c32c027dcb3890a64e1764d6d7dcd29524cd270dd42e7bd
Statusready
Image PathNone
Updated At2023-29-Sep 17:40:06
namecv-mitochondria
created2023-09-28 20:25:17.379732+00:00
last_updated2023-09-29 18:08:12.573990+00:00
deployedFalse
tags
versions4b6dab7d-b3ff-4f14-8425-7d9b6de76637, 66e72bc0-a3e3-4872-bc20-19b992c194b4, cf4bdfb4-1eec-46f8-9af4-b16dea894de6, 507cb1eb-8034-4b5b-bc96-2427730a6407, 50ed1d2f-6dba-411c-9579-1090791b33bd, 6208c9cf-fcd6-4b20-bbfc-f6ce714596e3
stepsmitochondria-detector
publishedFalse

Deploy the Pipeline with the Model Version Step

As per the other workshops:

  1. Clear the pipeline of all steps.
  2. Add the model version as a pipeline step.
  3. Deploy the pipeline with the following deployment configuration:
deploy_config = wallaroo.DeploymentConfigBuilder().replica_count(1).cpus(0.5).memory("1Gi").build()
pipeline.clear()
pipeline.add_model_step(prime_model_version)

deploy_config = wallaroo.DeploymentConfigBuilder().replica_count(1).cpus(0.5).memory("1Gi").build()
pipeline.deploy(deployment_config=deploy_config)
namecv-mitochondria
created2023-09-28 20:25:17.379732+00:00
last_updated2023-09-29 19:07:47.787981+00:00
deployedTrue
tags
versionsac4bd826-f47f-48b7-8319-01fbc6622899, 4b6dab7d-b3ff-4f14-8425-7d9b6de76637, 66e72bc0-a3e3-4872-bc20-19b992c194b4, cf4bdfb4-1eec-46f8-9af4-b16dea894de6, 507cb1eb-8034-4b5b-bc96-2427730a6407, 50ed1d2f-6dba-411c-9579-1090791b33bd, 6208c9cf-fcd6-4b20-bbfc-f6ce714596e3
stepsmitochondria-detector
publishedFalse

Create the Connection

For this demonstration, the connection set to a specific file on a GitHub repository. The connection details can be anything that can be stored in JSON: connection URLs, tokens, etc.

This connection will set a URL to pull a file from GitHub, then use the file contents to perform an inference.

Wallaroo connections are created through the Wallaroo Client create_connection(name, type, details) method. See the Wallaroo SDK Essentials Guide: Data Connections Management guide for full details.

Note that connection names must be unique across the Wallaroo instance - if needed, use random characters at the end to make sure your connection doesn’t have the same name as a previously created connection.

Here’s an example connection used to retrieve the same CSV file used in ./data/test_data.df.json: https://raw.githubusercontent.com/WallarooLabs/Workshops/main/Linear%20Regression/Real%20Estate/data/test_data.df.json

Create the Connection Exercise

# set the connection information for other steps
# suffix is used to create a unique data connection

forecast_connection_input_name = f'house-price-data'
forecast_connection_input_type = "HTTP"
forecast_connection_input_argument = { 
    "url": "https://raw.githubusercontent.com/WallarooLabs/Workshops/main/Linear%20Regression/Real%20Estate/data/test_data.df.json"
    }

wl.create_connection(forecast_connection_input_name, forecast_connection_input_type, forecast_connection_input_argument)
# set the connection information for other steps
# suffix is used to create a unique data connection

medical_imaging_connection_input_name = f'house-price-data-source-medical-jch'
medical_imaging_input_type = "HTTP"
medical_imaging_input_argument = { 
    "url": "https://raw.githubusercontent.com/WallarooLabs/Workshops/20230927-foundations-v2/Computer%20Vision/Healthcare/data/image_0_21.tif.df.json"
    }

wl.create_connection(medical_imaging_connection_input_name, medical_imaging_input_type, medical_imaging_input_argument)
FieldValue
Namehouse-price-data-source-medical-jch
Connection TypeHTTP
Details*****
Created At2023-09-29T19:15:01.906357+00:00
Linked Workspaces[]

List Connections

Connections for the entire Wallaroo instance are listed with Wallaroo Client list_connections() method.

List Connections Exercise

Here’s an example of listing the connections when the Wallaroo client is wl.

wl.list_connections()
# list the connections here

wl.list_connections()
nameconnection typedetailscreated atlinked workspaces
house-price-data-source-medicalHTTP*****2023-09-29T19:08:24.238046+00:00[]
house-price-data-source-medical-johnHTTP*****2023-09-29T19:09:36.352483+00:00['workshop-workspace-john-cv-medical']
house-price-data-source-medical-jchHTTP*****2023-09-29T19:15:01.906357+00:00[]

Get Connection by Name

To retrieve a previosly created conneciton, we can assign it to a variable with the method Wallaroo Client.get_connection(connection_name). Then we can display the connection itself. Notice that when displaying a connection, the details section will be hidden, but they are retrieved with connection.details(). Here’s an example:

myconnection = client.get_connection("My amazing connection")
display(myconnection)
display(myconnection.details()

Use that code to retrieve your new connection.

Get Connection by Name Example

Here’s an example based on the Wallaroo client saved as wl.

wl.get_connection(forecast_connection_input_name)
# get the connection by name

this_connection = wl.get_connection(medical_imaging_connection_input_name)
this_connection
FieldValue
Namehouse-price-data-source-medical-jch
Connection TypeHTTP
Details*****
Created At2023-09-29T19:15:01.906357+00:00
Linked Workspaces[]

Add Connection to Workspace

We’ll now add the connection to our workspace so it can be retrieved by other workspace users. The method Workspace add_connection(connection_name) adds a Data Connection to a workspace. The method Workspace list_connections() displays a list of connections attached to the workspace.

Add Connection to Workspace Exercise

Use the connection we just created, and add it to the sample workspace. Here’s a code example where the workspace is saved to the variable workspace and the connection is saved as forecast_connection_input_name.

workspace.add_connection(forecast_connection_input_name)
workspace.add_connection(medical_imaging_connection_input_name)
workspace.list_connections()
nameconnection typedetailscreated atlinked workspaces
house-price-data-source-medical-johnHTTP*****2023-09-29T19:09:36.352483+00:00['workshop-workspace-john-cv-medical']
house-price-data-source-medical-jchHTTP*****2023-09-29T19:15:01.906357+00:00['workshop-workspace-john-cv-medical']

Retrieve Connection from Workspace

To simulate a data scientist’s procedural flow, we’ll now retrieve the connection from the workspace. Specific connections are retrieved by specifying their position in the returned list.

For example, if we have two connections in a workspace and we want the second one, we can assign it to a variable with list_connections[1].

Create a new variable and retrieve the connection we just assigned to the workspace.

Retrieve Connection from Workspace Exercise

Retrieve the connection that was just associated with the workspace. You’ll use the list_connections method, then assign a variable to the connection. Here’s an example if the connection is the most recently one added to the workspace workspace.

forecast_connection = workspace.list_connections()[-1]
medical_connection = workspace.list_connections()[-1]
display(medical_connection)
FieldValue
Namehouse-price-data-source-medical-jch
Connection TypeHTTP
Details*****
Created At2023-09-29T19:15:01.906357+00:00
Linked Workspaces['workshop-workspace-john-cv-medical']

Run Inference with Connection

Connections can be used for different purposes: uploading new models, engine configurations - any place that data is needed. This exercise will use the data connection to perform an inference through our deployed pipeline.

Run Inference with Connection Exercise

We’ll now retrieve sample data through the Wallaroo connection, and perform a sample inference. The connection details are retrieved through the Connection details() method. Use them to retrieve the pandas record file and convert it to a DataFrame, and use it with our sample model.

Here’s a code example that uses the Python requests library to retrieve the file information, then turns it into a DataFrame for the inference request.

display(forecast_connection.details()['url'])

import requests

response = requests.get(
                    forecast_connection.details()['url']
                )

# display(response.json())

df = pd.DataFrame(response.json())

pipeline.infer(df)
display(medical_connection.details()['url'])

import requests

response = requests.get(
    medical_connection.details()['url']
)

df = pd.DataFrame(response.json())
display(df)

result = pipeline.infer(df)
display(result)
'https://raw.githubusercontent.com/WallarooLabs/Workshops/20230927-foundations-v2/Computer%20Vision/Healthcare/data/image_0_21.tif.df.json'
tensor
0[0.0572924651, 0.0555159546, 0.0621778691, 0.053739444, 0.04885404, 0.053739444, 0.0541835717, 0.0381949768, 0.0448568913, 0.0413038702, 0.0444127637, 0.0492981677, 0.0497422953, 0.0426362531, 0.0515188059, 0.0479657848, 0.0404156149, 0.039083232, 0.0470775295, 0.0413038702, 0.0461892742, 0.039083232, 0.0448568913, 0.0519629335, 0.0484099124, 0.043968636, 0.0453010189, 0.0448568913, 0.0519629335, 0.0444127637, 0.0484099124, 0.0479657848, 0.0546276993, 0.0461892742, 0.0635102521, 0.0515188059, 0.0612896139, 0.0519629335, 0.0546276993, 0.0666191455, 0.0541835717, 0.0577365928, 0.0568483375, 0.0555159546, 0.04885404, 0.053739444, 0.043968636, 0.0466334019, 0.0421921255, 0.0510746782, 0.0492981677, 0.0457451466, 0.0670632731, 0.0577365928, 0.0595131033, 0.0675074008, 0.0732810601, 0.0661750179, 0.0635102521, 0.0755016982, 0.0803871022, 0.0728369324, 0.0777223364, 0.0848283786, 0.084384251, 0.0866048892, 0.0768340812, 0.0821636128, 0.0803871022, 0.0866048892, 0.0848283786, 0.0777223364, 0.0834959957, 0.0786105917, 0.0799429746, 0.0808312299, 0.0817194852, 0.0839401233, 0.0808312299, 0.0786105917, 0.0777223364, 0.0759458259, 0.0715045495, 0.074613443, 0.064842635, 0.0595131033, 0.0617337415, 0.058624848, 0.059957231, 0.0652867626, 0.0715045495, 0.0675074008, 0.0728369324, 0.0728369324, 0.0732810601, 0.0750575706, 0.0728369324, 0.0723928048, 0.0723928048, 0.0675074008, ...]
timein.tensorout.conv2d_37check_failures
02023-09-29 19:15:27.809[0.0572924651, 0.0555159546, 0.0621778691, 0.053739444, 0.04885404, 0.053739444, 0.0541835717, 0.0381949768, 0.0448568913, 0.0413038702, 0.0444127637, 0.0492981677, 0.0497422953, 0.0426362531, 0.0515188059, 0.0479657848, 0.0404156149, 0.039083232, 0.0470775295, 0.0413038702, 0.0461892742, 0.039083232, 0.0448568913, 0.0519629335, 0.0484099124, 0.043968636, 0.0453010189, 0.0448568913, 0.0519629335, 0.0444127637, 0.0484099124, 0.0479657848, 0.0546276993, 0.0461892742, 0.0635102521, 0.0515188059, 0.0612896139, 0.0519629335, 0.0546276993, 0.0666191455, 0.0541835717, 0.0577365928, 0.0568483375, 0.0555159546, 0.04885404, 0.053739444, 0.043968636, 0.0466334019, 0.0421921255, 0.0510746782, 0.0492981677, 0.0457451466, 0.0670632731, 0.0577365928, 0.0595131033, 0.0675074008, 0.0732810601, 0.0661750179, 0.0635102521, 0.0755016982, 0.0803871022, 0.0728369324, 0.0777223364, 0.0848283786, 0.084384251, 0.0866048892, 0.0768340812, 0.0821636128, 0.0803871022, 0.0866048892, 0.0848283786, 0.0777223364, 0.0834959957, 0.0786105917, 0.0799429746, 0.0808312299, 0.0817194852, 0.0839401233, 0.0808312299, 0.0786105917, 0.0777223364, 0.0759458259, 0.0715045495, 0.074613443, 0.064842635, 0.0595131033, 0.0617337415, 0.058624848, 0.059957231, 0.0652867626, 0.0715045495, 0.0675074008, 0.0728369324, 0.0728369324, 0.0732810601, 0.0750575706, 0.0728369324, 0.0723928048, 0.0723928048, 0.0675074008, ...][0.073827654, 0.04537511, 0.022077948, 0.027527481, 0.021322101, 0.022883624, 0.02370426, 0.030836225, 0.042639196, 0.05585119, 0.07217789, 0.08134478, 0.08961046, 0.103625864, 0.12554172, 0.13468736, 0.16203326, 0.17829657, 0.21441683, 0.20686802, 0.22815448, 0.228313, 0.25762054, 0.24422944, 0.2527454, 0.24923241, 0.25063625, 0.22561434, 0.19528994, 0.1826241, 0.13989443, 0.11508188, 0.05944249, 0.0459204, 0.01766479, 0.011624932, 0.0019857585, 0.001327306, 0.0003760457, 0.00037619472, 5.8710575e-05, 6.005168e-05, 2.2798777e-05, 4.0113926e-05, 1.4781952e-05, 2.2858381e-05, 1.79708e-05, 3.990531e-05, 3.2246113e-05, 6.175041e-05, 4.014373e-05, 6.827712e-05, 2.8342009e-05, 4.389882e-05, 1.7046928e-05, 2.7179718e-05, 6.2286854e-06, 8.434057e-06, 2.5331974e-06, 5.453825e-06, 1.1920929e-06, 1.9669533e-06, 7.748604e-07, 1.9967556e-06, 7.748604e-07, 1.5497208e-06, 8.34465e-07, 2.2649765e-06, 1.1622906e-06, 2.503395e-06, 1.4305115e-06, 4.142523e-06, 2.4735928e-06, 5.096197e-06, 3.0696392e-06, 8.046627e-06, 5.185604e-06, 1.1056662e-05, 7.7188015e-06, 1.8835068e-05, 1.3649464e-05, 2.8192997e-05, 2.1010637e-05, 5.221367e-05, 4.3094158e-05, 8.9138746e-05, 6.3210726e-05, 0.00012665987, 6.726384e-05, 0.00010934472, 5.0485134e-05, 9.301305e-05, 3.7252903e-05, 5.8323145e-05, 2.5242567e-05, 4.926324e-05, 1.8715858e-05, 3.0636787e-05, 1.2457371e-05, 2.6792288e-05, ...]0

Cleaning up.

Now that the workshop is complete, don’t forget to undeploy your pipeline to free up the resources.

pipeline.undeploy()
namecv-mitochondria
created2023-09-28 20:25:17.379732+00:00
last_updated2023-09-29 19:07:47.787981+00:00
deployedFalse
tags
versionsac4bd826-f47f-48b7-8319-01fbc6622899, 4b6dab7d-b3ff-4f14-8425-7d9b6de76637, 66e72bc0-a3e3-4872-bc20-19b992c194b4, cf4bdfb4-1eec-46f8-9af4-b16dea894de6, 507cb1eb-8034-4b5b-bc96-2427730a6407, 50ed1d2f-6dba-411c-9579-1090791b33bd, 6208c9cf-fcd6-4b20-bbfc-f6ce714596e3
stepsmitochondria-detector
publishedFalse

Congratulations!

In this workshop you have:

  • Deployed a single step house price prediction pipeline and sent data to it.
  • Create a new Wallaroo connection.
  • Assigned the connection to a workspace.
  • Retrieved the connection from the workspace.
  • Used the data connection to retrieve information from outside of Wallaroo, and use it for an inference.

Great job!

3.1.3 - Healthcare: Edge Deployment

How to use Wallaroo to deploy a computer vision healthcare model on edge devices for inferences.

This tutorial and the assets can be downloaded as part of the Wallaroo Workshops repository.

For this workshop, we will take a Wallaroo pipeline and publish it to an Open Container (OCI) Registry. The registry details are stored in the Wallaroo instance as the Edge Registry.

In this set of exercises, you will:

  1. Use a pre-trained model and deploy it to Wallaroo.
  2. Perform sample inferences.
  3. Publish the pipeline to the Edge Registry.
  4. See the steps to deploy the published pipeline to an Edge device and perform inferences through it.

Deployment to the Edge allows data scientists to work in Wallaroo to test their models in Wallaroo, then once satisfied with the results publish those pipelines. DevOps engineers then take those published pipeline details from the Edge registry and deploy them into Docker and Kubernetes environments.

This workshop will demonstrate the following concepts:

  • Wallaroo Workspaces: Workspaces are environments were users upload models, create pipelines and other artifacts. The workspace should be considered the fundamental area where work is done. Workspaces are shared with other users to give them access to the same models, pipelines, etc.
  • Wallaroo Model Upload and Registration: ML Models are uploaded to Wallaroo through the SDK or the MLOps API to a workspace. ML models include default runtimes (ONNX, Python Step, and TensorFlow) that are run directly through the Wallaroo engine, and containerized runtimes (Hugging Face, PyTorch, etc) that are run through in a container through the Wallaroo engine.
  • Wallaroo Pipelines: Pipelines are used to deploy models for inferencing. Each model is a pipeline step in a pipelines, where the inputs of the previous step are fed into the next. Pipeline steps can be ML models, Python scripts, or Arbitrary Python (these contain necessary models and artifacts for running a model).
  • Pipeline Edge Publication: How to publish a Wallaroo pipeline to an OCI registry, then deploy that pipeline into other environments.

For this tutorial, we will be providing pre-trained models in ONNX format, and have connected a sample Edge Registry to our Wallaroo instance.

For more Wallaroo procedures, see the Wallaroo Documentation site.

Preliminaries

In the blocks below we will preload some required libraries.

For convenience, the following helper functions are defined to retrieve previously created workspaces, models, and pipelines:

  • get_workspace(name, client): This takes in the name and the Wallaroo client being used in this session, and returns the workspace matching name. If no workspaces are found matching the name, raises a KeyError and returns None.
  • get_model_version(model_name, workspace): Retrieves the most recent model version from the model matching the model_name within the provided workspace. If no model matches that name, raises a KeyError and returns None.
  • get_pipeline(pipeline_name, workspace): Retrieves the most pipeline from the workspace matching the pipeline_name within the provided workspace. If no model matches that name, raises a KeyError and returns None.
import json
import os
import datetime

import wallaroo
from wallaroo.object import EntityNotFoundError
from wallaroo.framework import Framework

# used to display dataframe information without truncating
from IPython.display import display
import pandas as pd
import numpy as np

pd.set_option('display.max_colwidth', None)

import time
import pyarrow as pa
## convenience functions from the previous notebooks

# return the workspace called <name> through the Wallaroo client.
def get_workspace(name, client):
    workspace = None
    for ws in client.list_workspaces():
        if ws.name() == name:
            workspace= ws
            return workspace
    # if no workspaces were found
    if workspace==None:
        raise KeyError(f"Workspace {name} was not found.")
    return workspace

# returns the most recent model version in a workspace for the matching `model_name`
def get_model_version(model_name, workspace):
    modellist = workspace.models()
    model_version = [m.versions()[-1] for m in modellist if m.name() == model_name]
    # if no models match, return None
    if len(modellist) <= 0:
        raise KeyError(f"Model {mname} not found in this workspace")
        return None
    return model_version[0]

# get a pipeline by name in the workspace
def get_pipeline(pipeline_name, workspace):
    plist = workspace.pipelines()
    pipeline = [p for p in plist if p.name() == pipeline_name]
    if len(pipeline) <= 0:
        raise KeyError(f"Pipeline {pipeline_name} not found in this workspace")
        return None
    return pipeline[0]

Connect to the Wallaroo Instance

The first step is to connect to Wallaroo through the Wallaroo client. The Python library is included in the Wallaroo install and available through the Jupyter Hub interface provided with your Wallaroo environment.

This is accomplished using the wallaroo.Client() command, which provides a URL to grant the SDK permission to your specific Wallaroo environment. When displayed, enter the URL into a browser and confirm permissions. Store the connection into a variable that can be referenced later.

If logging into the Wallaroo instance through the internal JupyterHub service, use wl = wallaroo.Client(). For more information on Wallaroo Client settings, see the Client Connection guide.

## blank space to log in 

wl = wallaroo.Client()

Set Configurations

Set the workspace, pipeline, and model used from Notebook 1. The helper functions will make this task easier.

Set Configurations References

# retrieve the previous workspace, model, and pipeline version

workspace_name = "workshop-workspace-john-cv-medical"

workspace = get_workspace(workspace_name, wl)

# set your current workspace to the workspace that you just created
wl.set_current_workspace(workspace)

# optionally, examine your current workspace
wl.get_current_workspace()

model_name = 'mitochondria-detector'

prime_model_version = get_model_version(model_name, workspace)

pipeline_name = 'cv-mitochondria'

pipeline = get_pipeline(pipeline_name, workspace)

display(workspace)
display(prime_model_version)
display(pipeline)
{'name': 'workshop-workspace-john-cv-medical', 'id': 10, 'archived': False, 'created_by': 'b030ff9c-41eb-49b4-afdf-2ccbecb6be5d', 'created_at': '2023-09-28T20:22:21.506043+00:00', 'models': [{'name': 'mitochondria-detector', 'versions': 2, 'owner_id': '""', 'last_update_time': datetime.datetime(2023, 9, 29, 17, 40, 6, 205146, tzinfo=tzutc()), 'created_at': datetime.datetime(2023, 9, 28, 20, 25, 2, 359367, tzinfo=tzutc())}], 'pipelines': [{'name': 'cv-mitochondria', 'create_time': datetime.datetime(2023, 9, 28, 20, 25, 17, 379732, tzinfo=tzutc()), 'definition': '[]'}]}
Namemitochondria-detector
Version11a34cce-0eaa-4cdd-8739-43bd18b67a04
File Namemitochondria_epochs_15.onnx
SHAe80fcdaf563a183b0c32c027dcb3890a64e1764d6d7dcd29524cd270dd42e7bd
Statusready
Image PathNone
Updated At2023-29-Sep 17:40:06
namecv-mitochondria
created2023-09-28 20:25:17.379732+00:00
last_updated2023-09-29 19:07:47.787981+00:00
deployedFalse
tags
versionsac4bd826-f47f-48b7-8319-01fbc6622899, 4b6dab7d-b3ff-4f14-8425-7d9b6de76637, 66e72bc0-a3e3-4872-bc20-19b992c194b4, cf4bdfb4-1eec-46f8-9af4-b16dea894de6, 507cb1eb-8034-4b5b-bc96-2427730a6407, 50ed1d2f-6dba-411c-9579-1090791b33bd, 6208c9cf-fcd6-4b20-bbfc-f6ce714596e3
stepsmitochondria-detector
publishedFalse

Deploy the Pipeline with the Model Version Step

As per the other workshops:

  1. Clear the pipeline of all steps.
  2. Add the model version as a pipeline step.
  3. Deploy the pipeline with the following deployment configuration:
deploy_config = wallaroo.DeploymentConfigBuilder().replica_count(1).cpus(0.5).memory("1Gi").build()
pipeline.clear()
pipeline.add_model_step(prime_model_version)

deploy_config = wallaroo.DeploymentConfigBuilder().replica_count(1).cpus(0.5).memory("1Gi").build()
pipeline.deploy(deployment_config=deploy_config)
namecv-mitochondria
created2023-09-28 20:25:17.379732+00:00
last_updated2023-09-29 19:23:37.594749+00:00
deployedTrue
tags
versionsd271be10-fadd-4408-97aa-c57e6ae4e35a, ac4bd826-f47f-48b7-8319-01fbc6622899, 4b6dab7d-b3ff-4f14-8425-7d9b6de76637, 66e72bc0-a3e3-4872-bc20-19b992c194b4, cf4bdfb4-1eec-46f8-9af4-b16dea894de6, 507cb1eb-8034-4b5b-bc96-2427730a6407, 50ed1d2f-6dba-411c-9579-1090791b33bd, 6208c9cf-fcd6-4b20-bbfc-f6ce714596e3
stepsmitochondria-detector
publishedFalse

Sample Inference

Verify the pipeline is deployed properly with a sample inference with the file ./data/test_data.df.json.

# sample inference from previous code here

pipeline.infer_from_file('../data/image_0_21.tif.df.json')
timein.tensorout.conv2d_37check_failures
02023-09-29 19:23:58.276[0.0572924651, 0.0555159546, 0.0621778691, 0.053739444, 0.04885404, 0.053739444, 0.0541835717, 0.0381949768, 0.0448568913, 0.0413038702, 0.0444127637, 0.0492981677, 0.0497422953, 0.0426362531, 0.0515188059, 0.0479657848, 0.0404156149, 0.039083232, 0.0470775295, 0.0413038702, 0.0461892742, 0.039083232, 0.0448568913, 0.0519629335, 0.0484099124, 0.043968636, 0.0453010189, 0.0448568913, 0.0519629335, 0.0444127637, 0.0484099124, 0.0479657848, 0.0546276993, 0.0461892742, 0.0635102521, 0.0515188059, 0.0612896139, 0.0519629335, 0.0546276993, 0.0666191455, 0.0541835717, 0.0577365928, 0.0568483375, 0.0555159546, 0.04885404, 0.053739444, 0.043968636, 0.0466334019, 0.0421921255, 0.0510746782, 0.0492981677, 0.0457451466, 0.0670632731, 0.0577365928, 0.0595131033, 0.0675074008, 0.0732810601, 0.0661750179, 0.0635102521, 0.0755016982, 0.0803871022, 0.0728369324, 0.0777223364, 0.0848283786, 0.084384251, 0.0866048892, 0.0768340812, 0.0821636128, 0.0803871022, 0.0866048892, 0.0848283786, 0.0777223364, 0.0834959957, 0.0786105917, 0.0799429746, 0.0808312299, 0.0817194852, 0.0839401233, 0.0808312299, 0.0786105917, 0.0777223364, 0.0759458259, 0.0715045495, 0.074613443, 0.064842635, 0.0595131033, 0.0617337415, 0.058624848, 0.059957231, 0.0652867626, 0.0715045495, 0.0675074008, 0.0728369324, 0.0728369324, 0.0732810601, 0.0750575706, 0.0728369324, 0.0723928048, 0.0723928048, 0.0675074008, ...][0.073827654, 0.04537511, 0.022077948, 0.027527481, 0.021322101, 0.022883624, 0.02370426, 0.030836225, 0.042639196, 0.05585119, 0.07217789, 0.08134478, 0.08961046, 0.103625864, 0.12554172, 0.13468736, 0.16203326, 0.17829657, 0.21441683, 0.20686802, 0.22815448, 0.228313, 0.25762054, 0.24422944, 0.2527454, 0.24923241, 0.25063625, 0.22561434, 0.19528994, 0.1826241, 0.13989443, 0.11508188, 0.05944249, 0.0459204, 0.01766479, 0.011624932, 0.0019857585, 0.001327306, 0.0003760457, 0.00037619472, 5.8710575e-05, 6.005168e-05, 2.2798777e-05, 4.0113926e-05, 1.4781952e-05, 2.2858381e-05, 1.79708e-05, 3.990531e-05, 3.2246113e-05, 6.175041e-05, 4.014373e-05, 6.827712e-05, 2.8342009e-05, 4.389882e-05, 1.7046928e-05, 2.7179718e-05, 6.2286854e-06, 8.434057e-06, 2.5331974e-06, 5.453825e-06, 1.1920929e-06, 1.9669533e-06, 7.748604e-07, 1.9967556e-06, 7.748604e-07, 1.5497208e-06, 8.34465e-07, 2.2649765e-06, 1.1622906e-06, 2.503395e-06, 1.4305115e-06, 4.142523e-06, 2.4735928e-06, 5.096197e-06, 3.0696392e-06, 8.046627e-06, 5.185604e-06, 1.1056662e-05, 7.7188015e-06, 1.8835068e-05, 1.3649464e-05, 2.8192997e-05, 2.1010637e-05, 5.221367e-05, 4.3094158e-05, 8.9138746e-05, 6.3210726e-05, 0.00012665987, 6.726384e-05, 0.00010934472, 5.0485134e-05, 9.301305e-05, 3.7252903e-05, 5.8323145e-05, 2.5242567e-05, 4.926324e-05, 1.8715858e-05, 3.0636787e-05, 1.2457371e-05, 2.6792288e-05, ...]0

Undeploying Your Pipeline

You should always undeploy your pipelines when you are done with them, or don’t need them for a while. This releases the resources that the pipeline is using for other processes to use. You can always redeploy the pipeline when you need it again. As a reminder, here are the commands to deploy and undeploy a pipeline:


# "turn off" the pipeline and releaase its resources
my_pipeline.undeploy()
# blank space to undeploy the pipeline
pipeline.undeploy()
namecv-mitochondria
created2023-09-28 20:25:17.379732+00:00
last_updated2023-09-29 19:23:37.594749+00:00
deployedFalse
tags
versionsd271be10-fadd-4408-97aa-c57e6ae4e35a, ac4bd826-f47f-48b7-8319-01fbc6622899, 4b6dab7d-b3ff-4f14-8425-7d9b6de76637, 66e72bc0-a3e3-4872-bc20-19b992c194b4, cf4bdfb4-1eec-46f8-9af4-b16dea894de6, 507cb1eb-8034-4b5b-bc96-2427730a6407, 50ed1d2f-6dba-411c-9579-1090791b33bd, 6208c9cf-fcd6-4b20-bbfc-f6ce714596e3
stepsmitochondria-detector
publishedFalse

Publish the Pipeline for Edge Deployment

It worked! For a demo, we’ll take working once as “tested”. So now that we’ve tested our pipeline, we are ready to publish it for edge deployment.

Publishing it means assembling all of the configuration files and model assets and pushing them to an Open Container Initiative (OCI) repository set in the Wallaroo instance as the Edge Registry service. DevOps engineers then retrieve that image and deploy it through Docker, Kubernetes, or similar deployments.

See Edge Deployment Registry Guide for details on adding an OCI Registry Service to Wallaroo as the Edge Deployment Registry.

This is done through the SDK command wallaroo.pipeline.publish(deployment_config) which has the following parameters and returns.

Publish a Pipeline Parameters

The publish method takes the following parameters. The containerized pipeline will be pushed to the Edge registry service with the model, pipeline configurations, and other artifacts needed to deploy the pipeline.

ParameterTypeDescription
deployment_configwallaroo.deployment_config.DeploymentConfig (Optional)Sets the pipeline deployment configuration. For example: For more information on pipeline deployment configuration, see the Wallaroo SDK Essentials Guide: Pipeline Deployment Configuration.

Publish a Pipeline Returns

FieldTypeDescription
idintegerNumerical Wallaroo id of the published pipeline.
pipeline version idintegerNumerical Wallaroo id of the pipeline version published.
statusstringThe status of the pipeline publication. Values include:
  • PendingPublish: The pipeline publication is about to be uploaded or is in the process of being uploaded.
  • Published: The pipeline is published and ready for use.
Engine URLstringThe URL of the published pipeline engine in the edge registry.
Pipeline URLstringThe URL of the published pipeline in the edge registry.
Helm Chart URLstringThe URL of the helm chart for the published pipeline in the edge registry.
Helm Chart ReferencestringThe help chart reference.
Helm Chart VersionstringThe version of the Helm Chart of the published pipeline. This is also used as the Docker tag.
Engine Configwallaroo.deployment_config.DeploymentConfigThe pipeline configuration included with the published pipeline.
Created AtDateTimeWhen the published pipeline was created.
Updated AtDateTimeWhen the published pipeline was updated.

Publish the Pipeline for Edge Deployment Exercise

We will now publish the pipeline to our Edge Deployment Registry with the pipeline.publish(deployment_config) command. deployment_config is an optional field that specifies the pipeline deployment. This can be overridden by the DevOps engineer during deployment.

In this example, assuming that the pipeline was saved to the variable my_pipeline, we would publish it to the Edge Registry already stored in the Wallaroo instance and store the pipeline publish to the variable my_pub with the following command:

my_pub=pipeline.publish(deploy_config)
# display the publish
my_pub
## blank space to publish the pipeline

my_pub=pipeline.publish(deploy_config)
# display the publish
my_pub
Waiting for pipeline publish... It may take up to 600 sec.
Pipeline is Publishing...Published.
ID5
Pipeline Version63f71352-93bc-4e4a-85f6-0a0bf603124c
StatusPublished
Engine URLghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/standalone-mini:v2023.3.0-3854
Pipeline URLghcr.io/wallaroolabs/doc-samples/pipelines/cv-mitochondria:63f71352-93bc-4e4a-85f6-0a0bf603124c
Helm Chart URLghcr.io/wallaroolabs/doc-samples/charts/cv-mitochondria
Helm Chart Referenceghcr.io/wallaroolabs/doc-samples/charts@sha256:ef2632b42423105004744d93ebe6c164f4df4b733f3b3698946f4804bbf69477
Helm Chart Version0.0.1-63f71352-93bc-4e4a-85f6-0a0bf603124c
Engine Config{'engine': {'resources': {'limits': {'cpu': 1.0, 'memory': '512Mi'}, 'requests': {'cpu': 1.0, 'memory': '512Mi'}}}, 'engineAux': {'images': {}}, 'enginelb': {'resources': {'limits': {'cpu': 1.0, 'memory': '512Mi'}, 'requests': {'cpu': 1.0, 'memory': '512Mi'}}}}
User Images[]
Created Byjohn.hummel@wallaroo.ai
Created At2023-09-29 19:26:55.535747+00:00
Updated At2023-09-29 19:26:55.535747+00:00

List Published Pipelines

The method wallaroo.client.list_pipelines() shows a list of all pipelines in the Wallaroo instance, and includes the published field that indicates whether the pipeline was published to the registry (True), or has not yet been published (False).

List Published Pipelines Exercise

List all pipelines and see which ones are published or not. For example, if your client was saved to the variable wl, then the following will list the pipelines and display which ones are published.

wl.list_pipelines()
# list the pipelines and view which are published

wl.list_pipelines()
namecreatedlast_updateddeployedtagsversionsstepspublished
cv-mitochondria2023-28-Sep 20:25:172023-29-Sep 19:26:53False63f71352-93bc-4e4a-85f6-0a0bf603124c, d271be10-fadd-4408-97aa-c57e6ae4e35a, ac4bd826-f47f-48b7-8319-01fbc6622899, 4b6dab7d-b3ff-4f14-8425-7d9b6de76637, 66e72bc0-a3e3-4872-bc20-19b992c194b4, cf4bdfb4-1eec-46f8-9af4-b16dea894de6, 507cb1eb-8034-4b5b-bc96-2427730a6407, 50ed1d2f-6dba-411c-9579-1090791b33bd, 6208c9cf-fcd6-4b20-bbfc-f6ce714596e3mitochondria-detectorTrue
retailimage2023-28-Sep 19:44:332023-28-Sep 19:54:59False26afe601-6515-48ca-9a37-d063ab1e1ea2, 1d806c89-ecc6-4207-b98f-c56eccd16c43, 11835eda-4e10-49c0-baab-63862c16d1ef, 57bf2bfb-009b-42b9-b926-742f8bbb8d3c, 891fe58d-902b-49bd-94d3-c2196a8efd3b, db0d489b-d8fa-41d3-b46f-a9623b28e336, f039eaf3-d0dd-4ab7-a767-852db5241ff0, 2f5cd92d-ecc8-4e75-aee5-1605c1f23f0ev5s6False
retailimage2023-28-Sep 18:55:142023-28-Sep 19:23:05Trued64dabed-7f7a-4f41-a307-e7995d7b8144, 8d257d18-2ca1-46b9-a40e-1f3d7f308dc1, e84586a7-05bb-4d67-a696-f04e80df8b58, 95c2157a-2722-4a5b-b564-d3a709c6238f, fa351ab0-fe77-4fc0-b521-ba15e92a91d7v5s6False
cv-yolo2023-28-Sep 16:07:292023-28-Sep 18:47:35True5f889757-89c5-4475-a579-937639779ab3, f9981617-7734-4f2d-905a-62333c600fe7, b21ac721-49e3-402d-b6c0-af139d51299a, 3f277cc7-351d-4d10-bdb2-c770c0dc1ac2house-price-primeFalse
houseprice-estimator2023-27-Sep 16:51:152023-27-Sep 16:53:56False07cac6a2-140d-4a5e-b7ec-264f5fbf9dc3, bd389561-2c4f-492b-a82b-896cf76c2acf, 37bcce00-28d9-4d28-b637-33acf4021103, 146a3e4a-057b-4bd2-94f7-ebadc133df3d, 996a9877-142f-4934-aa4a-7696d3662297, a79802b5-42f4-4fb6-bd6b-3da560d39d73house-price-primeFalse
aloha-fraud-detector2023-27-Sep 16:29:552023-27-Sep 18:28:05Falsee2a42011-d319-476f-bc32-9b6cccae4870, be15dcad-5a78-4493-b568-ee4502fa1791, b74a8b3a-8128-4356-a6ff-434c2b283cc8, 6d72feb7-76b5-4121-b401-9dfd4b978745, c22e3aa7-8efa-41c1-8844-cc4e7d1147c5, 739269a7-7890-4774-9597-fda5f80a3a6d, aa362e18-7f7e-4dc6-9069-3207e9d2f605, 79865932-5b89-4b2a-bfb1-cb9ebeb5125f, 4727b985-db36-44f7-a1a3-7f1886bbf894, 07cbfcae-1fa2-4746-b585-55349d230b45, 03824313-6bbb-4ccd-95ea-64340f789b9c, 9ce54998-a667-43b3-8198-b2d95e0d2879, 8a416842-5675-455a-b638-29fe7dbb5ba1aloha-primeTrue
cv-arm-edge2023-27-Sep 15:20:152023-27-Sep 15:20:15(unknown)86dd133a-c12f-478b-af9a-30a7e4850fc4True
cv-arm-edge2023-27-Sep 15:17:452023-27-Sep 15:17:45(unknown)97a92779-0a5d-4c2b-bcf1-7afd60ac83d5False

List Publishes from a Pipeline

All publishes created from a pipeline are displayed with the wallaroo.pipeline.publishes method. The pipeline_version_id is used to know what version of the pipeline was used in that specific publish. This allows for pipelines to be updated over time, and newer versions to be sent and tracked to the Edge Deployment Registry service.

List Publishes Parameters

N/A

List Publishes Returns

A List of the following fields:

FieldTypeDescription
idintegerNumerical Wallaroo id of the published pipeline.
pipeline_version_idintegerNumerical Wallaroo id of the pipeline version published.
engine_urlstringThe URL of the published pipeline engine in the edge registry.
pipeline_urlstringThe URL of the published pipeline in the edge registry.
created_bystringThe email address of the user that published the pipeline.
Created AtDateTimeWhen the published pipeline was created.
Updated AtDateTimeWhen the published pipeline was updated.

List Publishes from a Pipeline Exercise

List all of the publishes from our pipeline. For example, if our pipeline is my_pipeline, then we would list all publishes from the pipeline with the following:

my_pipeline.publishes()
pipeline.publishes()
idpipeline_version_nameengine_urlpipeline_urlcreated_bycreated_atupdated_at
563f71352-93bc-4e4a-85f6-0a0bf603124cghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/standalone-mini:v2023.3.0-3854ghcr.io/wallaroolabs/doc-samples/pipelines/cv-mitochondria:63f71352-93bc-4e4a-85f6-0a0bf603124cjohn.hummel@wallaroo.ai2023-29-Sep 19:26:552023-29-Sep 19:26:55

Congratulations!

You have now

  • Created a workspace and set it as the current workspace.
  • Uploaded an ONNX model.
  • Created a Wallaroo pipeline, and set the most recent version of the uploaded model as a pipeline step.
  • Successfully send data to your pipeline for inference through the SDK and through an API call.

DevOps - Pipeline Edge Deployment

Once a pipeline is deployed to the Edge Registry service, it can be deployed in environments such as Docker, Kubernetes, or similar container running services by a DevOps engineer.

Docker Deployment

First, the DevOps engineer must authenticate to the same OCI Registry service used for the Wallaroo Edge Deployment registry.

For more details, check with the documentation on your artifact service. The following are provided for the three major cloud services:

For the deployment, the engine URL is specified with the following environmental variables:

{published engine url}
-e DEBUG=true -e OCI_REGISTRY={your registry server} \
-e CONFIG_CPUS=4 \ # optional number of CPUs to use
-e OCI_USERNAME={registry username} \
-e OCI_PASSWORD={registry token here} \
-e PIPELINE_URL={published pipeline url}

Docker Deployment Example

Using our sample environment, here’s sample deployment using Docker with a computer vision ML model, the same used in the Wallaroo Use Case Tutorials Computer Vision: Retail tutorials.

docker run -p 8080:8080 \
    -e DEBUG=true -e OCI_REGISTRY={your registry server} \
    -e CONFIG_CPUS=4 \
    -e OCI_USERNAME=oauth2accesstoken \
    -e OCI_PASSWORD={registry token here} \
    -e PIPELINE_URL={your registry server}/pipelines/edge-cv-retail:bf70eaf7-8c11-4b46-b751-916a43b1a555 \
    {your registry server}/engine:v2023.3.0-main-3707

Docker Compose Deployment

For users who prefer to use docker compose, the following sample compose.yaml file is used to launch the Wallaroo Edge pipeline. This is the same used in the Wallaroo Use Case Tutorials Computer Vision: Retail tutorials.

services:
  engine:
    image: {Your Engine URL}
    ports:
      - 8080:8080
    environment:
      PIPELINE_URL: {Your Pipeline URL}
      OCI_REGISTRY: {Your Edge Registry URL}
      OCI_USERNAME:  {Your Registry Username}
      OCI_PASSWORD: {Your Token or Password}
      CONFIG_CPUS: 4

For example:

services:
  engine:
    image: sample-registry.com/engine:v2023.3.0-main-3707
    ports:
      - 8080:8080
    environment:
      PIPELINE_URL: sample-registry.com/pipelines/edge-cv-retail:bf70eaf7-8c11-4b46-b751-916a43b1a555
      OCI_REGISTRY: sample-registry.com
      OCI_USERNAME:  _json_key_base64
      OCI_PASSWORD: abc123
      CONFIG_CPUS: 4

Docker Compose Deployment Example

The deployment and undeployment is then just a simple docker compose up and docker compose down. The following shows an example of deploying the Wallaroo edge pipeline using docker compose.

docker compose up
[+] Running 1/1
 ✔ Container cv_data-engine-1  Recreated                                                                                                                                                                 0.5s
Attaching to cv_data-engine-1
cv_data-engine-1  | Wallaroo Engine - Standalone mode
cv_data-engine-1  | Login Succeeded
cv_data-engine-1  | Fetching manifest and config for pipeline: sample-registry.com/pipelines/edge-cv-retail:bf70eaf7-8c11-4b46-b751-916a43b1a555
cv_data-engine-1  | Fetching model layers
cv_data-engine-1  | digest: sha256:c6c8869645962e7711132a7e17aced2ac0f60dcdc2c7faa79b2de73847a87984
cv_data-engine-1  |   filename: c6c8869645962e7711132a7e17aced2ac0f60dcdc2c7faa79b2de73847a87984
cv_data-engine-1  |   name: resnet-50
cv_data-engine-1  |   type: model
cv_data-engine-1  |   runtime: onnx
cv_data-engine-1  |   version: 693e19b5-0dc7-4afb-9922-e3f7feefe66d
cv_data-engine-1  |
cv_data-engine-1  | Fetched
cv_data-engine-1  | Starting engine
cv_data-engine-1  | Looking for preexisting `yaml` files in //modelconfigs
cv_data-engine-1  | Looking for preexisting `yaml` files in //pipelines

Helm Deployment

Published pipelines can be deployed through the use of helm charts.

Helm deployments take up to two steps - the first step is in retrieving the required values.yaml and making updates to override.

  1. Pull the helm charts from the published pipeline. The two fields are the Helm Chart URL and the Helm Chart version to specify the OCI . This typically takes the format of:
helm pull oci://{published.helm_chart_url} --version {published.helm_chart_version}
  1. Extract the tgz file and copy the values.yaml and copy the values used to edit engine allocations, etc. The following are required for the deployment to run:
ociRegistry:
  registry: {your registry service}
  username:  {registry username here}
  password: {registry token here}

Store this into another file, suc as local-values.yaml.

  1. Create the namespace to deploy the pipeline to. For example, the namespace wallaroo-edge-pipeline would be:
kubectl create -n wallaroo-edge-pipeline
  1. Deploy the helm installation with helm install through one of the following options:

    1. Specify the tgz file that was downloaded and the local values file. For example:

      helm install --namespace {namespace} --values {local values file} {helm install name} {tgz path}
      
    2. Specify the expended directory from the downloaded tgz file.

      helm install --namespace {namespace} --values {local values file} {helm install name} {helm directory path}
      
    3. Specify the Helm Pipeline Helm Chart and the Pipeline Helm Version.

      helm install --namespace {namespace} --values {local values file} {helm install name} oci://{published.helm_chart_url} --version {published.helm_chart_version}
      
  2. Once deployed, the DevOps engineer will have to forward the appropriate ports to the svc/engine-svc service in the specific pipeline. For example, using kubectl port-forward to the namespace ccfraud that would be:

    kubectl port-forward svc/engine-svc -n ccfraud01 8080 --address 0.0.0.0`
    

The following code segment generates a docker run template based on the previously published pipeline, assuming our publish was listed as my_pub.

docker_deploy = f'''
docker run -p 8080:8080 \\
    -e DEBUG=true \\
    -e OCI_REGISTRY=$REGISTRYURL \\
    -e CONFIG_CPUS=4 \\
    -e OCI_USERNAME=$REGISTRYUSERNAME \\
    -e OCI_PASSWORD=$REGISTRYPASSWORD \\
    -e PIPELINE_URL={my_pub.pipeline_url} \\
    {my_pub.engine_url}
'''

print(docker_deploy)
docker run -p 8080:8080 \
    -e DEBUG=true \
    -e OCI_REGISTRY=$REGISTRYURL \
    -e CONFIG_CPUS=4 \
    -e OCI_USERNAME=$REGISTRYUSERNAME \
    -e OCI_PASSWORD=$REGISTRYPASSWORD \
    -e PIPELINE_URL=ghcr.io/wallaroolabs/doc-samples/pipelines/cv-mitochondria:63f71352-93bc-4e4a-85f6-0a0bf603124c \
    ghcr.io/wallaroolabs/doc-samples/engines/proxy/wallaroo/ghcr.io/wallaroolabs/standalone-mini:v2023.3.0-3854

Docker Compose Deployment Exercise

Use the docker compose up command on your own compose.yaml using the sample above, replacing the OCI_USERNAME and OCI_PASSWORD with the values provided by your instructor.

Edge Deployed Pipeline API Endpoints

Once deployed, we can check the pipelines and models available. We’ll use a curl command, but any HTTP based request will work the same way.

Pipelines Endpoints

The endpoint /pipelines returns:

  • id (String): The name of the pipeline.
  • status (String): The status as either Running, or Error if there are any issues.

For this example, the deployment is made on a machine called testboy.local. Replace this URL with the URL of you edge deployment.

Pipelines Endpoints Exercise

Use the following curl command to view the pipeline data. For example, if the pipeline was deployed on localhost, then the command would be:

!curl locahost:8080/pipelines
# blank space to run the command - replace testboy.local with the host

!curl localhost:8080/pipelines
{"pipelines":[{"id":"cv-mitochondria","status":"Running"}]}

Models Endpoints

The endpoint /models returns a List of models with the following fields:

  • name (String): The model name.
  • sha (String): The sha hash value of the ML model.
  • status (String): The status of either Running or Error if there are any issues.
  • version (String): The model version. This matches the version designation used by Wallaroo to track model versions in UUID format.

Models Endpoints Exercise

Use the following curl command to view the models data. For example, if the pipeline was deployed on localhost, then the command would be:

!curl locahost:8080/models
# blank space to run the command - replace testboy.local with the host

!curl localhost:8080/models
{"models":[{"name":"mitochondria-detector","sha":"e80fcdaf563a183b0c32c027dcb3890a64e1764d6d7dcd29524cd270dd42e7bd","status":"Running","version":"11a34cce-0eaa-4cdd-8739-43bd18b67a04"}]}

Edge Deployed Inference

The inference endpoint takes the following pattern:

  • /pipelines/{pipeline-name}: The pipeline-name is the same as returned from the /pipelines endpoint as id.

Wallaroo inference endpoint URLs accept the following data inputs through the Content-Type header:

  • Content-Type: application/vnd.apache.arrow.file: For Apache Arrow tables.
  • Content-Type: application/json; format=pandas-records: For pandas DataFrame in record format.

It returns a application/json; format=pandas-records - the same pandas record we’ve been working with.

Edge Deployed Inference Exercise

Perform an inference on the deployed pipeline using curl. This command will look like this:

!curl -X POST localhost:8080/pipelines/{YOUR PIPELINE NAME} -H "Content-Type: application/json; format=pandas-records" --data @../data/singleton.df.json
!curl -X POST localhost:8080/pipelines/cv-mitochondria -H "Content-Type: application/json; format=pandas-records" --data @../data/image_0_21.tif.df.json
[{"check_failures":[],"elapsed":[40924125,357825417],"model_name":"mitochondria-detector","model_version":"11a34cce-0eaa-4cdd-8739-43bd18b67a04","original_data":null,"outputs":[{"Float":{"data":[0.07382762432098389,0.045375049114227295,0.02207791805267334,0.027527421712875366,0.021322041749954224,0.022883564233779907,0.023704230785369873,0.03083619475364685,0.04263922572135925,0.055851101875305176,0.07217776775360107,0.08134463429450989,0.08961024880409241,0.10362571477890015,0.12554147839546204,0.13468709588050842,0.16203296184539795,0.17829623818397522,0.21441656351089478,0.2068677544593811,0.2281542420387268,0.22831282019615173,0.25762036442756653,0.24422922730445862,0.25274521112442017,0.24923226237297058,0.25063613057136536,0.22561416029930115,0.19528985023498535,0.1826239824295044,0.1398943066596985,0.115081787109375,0.05944240093231201,0.045920342206954956,0.01766476035118103,0.011624962091445923,0.0019857585430145264,0.0013273060321807861,0.00037604570388793945,0.00037613511085510254,0.000058710575103759766,0.000060051679611206055,0.000022798776626586914,0.00004011392593383789,0.00001481175422668457,0.00002276897430419922,0.000017970800399780273,0.000039905309677124023,0.00003224611282348633,0.00006175041198730469,0.000040203332901000977,0.00006827712059020996,0.000028401613235473633,0.000043898820877075195,0.00001704692840576172,0.000027179718017578125,6.22868537902832e-6,8.493661880493164e-6,2.5331974029541016e-6,5.3942203521728516e-6,1.1920928955078125e-6,1.8477439880371094e-6,7.748603820800781e-7,2.0265579223632812e-6,7.748603820800781e-7,1.519918441772461e-6,7.748603820800781e-7,2.205371856689453e-6,1.1622905731201172e-6,2.4437904357910156e-6,1.5497207641601562e-6,4.1425228118896484e-6,2.384185791015625e-6,5.0961971282958984e-6,3.069639205932617e-6,8.046627044677734e-6,5.185604095458984e-6,0.000011056661605834961,7.718801498413086e-6,0.000018835067749023438,0.000013649463653564453,0.000028192996978759766,0.000020951032638549805,0.00005227327346801758,0.00004309415817260742,0.00008913874626159668,0.00006321072578430176,0.0001266002655029297,0.00006726384162902832,0.0001093149185180664,0.00005048513412475586,0.00009301304817199707,0.000037342309951782227,0.000058323144912719727,0.00002530217170715332,0.00004926323890686035,0.000018715858459472656,0.00003063678741455078,0.00001245737075805664,0.000026881694793701172,9.566545486450195e-6,0.000016182661056518555,6.496906280517578e-6,0.000013977289199829102,4.887580871582031e-6,8.404254913330078e-6,3.4868717193603516e-6,9.119510650634766e-6,4.0531158447265625e-6,8.13603401184082e-6,4.26173210144043e-6,0.000010520219802856445,4.6193599700927734e-6,8.672475814819336e-6,4.0531158447265625e-6,0.00001043081283569336,4.798173904418945e-6,9.179115295410156e-6,4.4405460357666016e-6,9.5367431640625e-6,3.1888484954833984e-6,5.304813385009766e-6,2.2351741790771484e-6,6.020069122314453e-6,2.562999725341797e-6,5.0067901611328125e-6,3.039836883544922e-6,7.659196853637695e-6,3.814697265625e-6,7.62939453125e-6,5.751848220825195e-6,0.000016242265701293945,0.000012755393981933594,0.000028699636459350586,0.000024050474166870117,0.000056177377700805664,0.00003841519355773926,0.00007605552673339844,0.0000514984130859375,0.0001150667667388916,0.0000813603401184082,0.0001557469367980957,0.00013962388038635254,0.0003027021884918213,0.00026109814643859863,0.00046062469482421875,0.0003737807273864746,0.0006524324417114258,0.00047200918197631836,0.0007236003875732422,0.0005968809127807617,0.001073688268661499,0.0008858442306518555,0.0013785064220428467,0.0010051727294921875,0.0015411972999572754,0.0009672641754150391,0.0013799965381622314,0.0009815096855163574,0.0014969110488891602,0.0009140968322753906,0.0012715458869934082,0.0009190738201141357,0.001569896936416626,0.001228630542755127,0.001904219388961792,0.001717597246170044,0.0025345981121063232,0.001650691032409668,0.002280682325363159,0.0013093352317810059,0.0017162859439849854,0.0007543265819549561,0.0009396374225616455,0.00037479400634765625,0.00045818090438842773,0.00010633468627929688,0.00011935830116271973,0.000033020973205566406,0.00005266070365905762,0.000014662742614746094,0.000021517276763916016,7.867813110351562e-6,0.00001558661460876465,5.632638931274414e-6,0.00001004338264465332,3.844499588012695e-6,9.059906005859375e-6,3.337860107421875e-6,6.079673767089844e-6,2.3543834686279297e-6,5.155801773071289e-6,1.7583370208740234e-6,3.1888484954833984e-6,1.2516975402832031e-6,3.1888484954833984e-6,1.2814998626708984e-6,2.7120113372802734e-6,1.1622905731201172e-6,2.9206275939941406e-6,1.0132789611816406e-6,1.9371509552001953e-6,7.152557373046875e-7,2.1457672119140625e-6,8.940696716308594e-7,1.6987323760986328e-6,8.344650268554688e-7,2.4139881134033203e-6,1.0728836059570312e-6,2.205371856689453e-6,1.0132789611816406e-6,3.129243850708008e-6,1.3709068298339844e-6,2.950429916381836e-6,1.5795230865478516e-6,4.172325134277344e-6,1.6987323760986328e-6,3.427267074584961e-6,1.6391277313232422e-6,4.26173210144043e-6,1.6987323760986328e-6,3.248453140258789e-6,1.6987323760986328e-6,4.231929779052734e-6,1.8477439880371094e-6,3.904104232788086e-6,2.205371856689453e-6,5.781650543212891e-6,3.0100345611572266e-6,6.258487701416016e-6,3.993511199951172e-6,9.208917617797852e-6,3.993511199951172e-6,7.063150405883789e-6,4.32133674621582e-6,0.00001049041748046875,6.079673767089844e-6,0.000011712312698364258,9.149312973022461e-6,0.000020116567611694336,0.000012099742889404297,0.000022619962692260742,0.000011801719665527344,0.000023037195205688477,8.165836334228516e-6,0.000013560056686401367,5.930662155151367e-6,0.000014156103134155273,5.245208740234375e-6,8.732080459594727e-6,0.000010520219802856445,0.000029712915420532227,0.00010520219802856445,0.00015032291412353516,0.0009121596813201904,0.008487939834594727,0.01903444528579712,0.015768975019454956,0.006120860576629639,0.007659435272216797,0.005146831274032593,0.006676912307739258,0.005835622549057007,0.011198669672012329,0.015312910079956055,0.02710878849029541,0.030075490474700928,0.04533454775810242,0.04933196306228638,0.06992730498313904,0.07720574736595154,0.0979054868221283,0.11467188596725464,0.1456666886806488,0.15871182084083557,0.17429497838020325,0.18464037775993347,0.20827051997184753,0.21409878134727478,0.22273945808410645,0.2207934558391571,0.23194774985313416,0.2132304310798645,0.20275938510894775,0.16805043816566467,0.15242788195610046,0.09788960218429565,0.0824669897556305,0.030963361263275146,0.021536976099014282,0.0055491626262664795,0.002815157175064087,0.0002790093421936035,0.0001405477523803711,0.000028699636459350586,0.000020772218704223633,1.6391277313232422e-6,1.7881393432617188e-6,5.364418029785156e-7,9.834766387939453e-7,2.384185791015625e-7,4.76837158203125e-7,2.980232238769531e-7,1.1622905731201172e-6,7.450580596923828e-7,1.8775463104248047e-6,9.238719940185547e-7,2.2649765014648438e-6,5.960464477539062e-7,1.1622905731201172e-6,4.172325134277344e-7,5.066394805908203e-7,8.940696716308594e-8,5.960464477539063e-8,2.9802322387695312e-8,0.0,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,2.9802322387695312e-8,-5.960464477539063e-8,0.0,-5.960464477539063e-8,0.0,5.960464477539063e-8,0.0,-5.960464477539063e-8,1.1920928955078125e-7,5.960464477539063e-8,1.4901161193847656e-7,1.1920928955078125e-7,2.980232238769531e-7,2.086162567138672e-7,5.364418029785156e-7,4.172325134277344e-7,1.3709068298339844e-6,1.0132789611816406e-6,2.4139881134033203e-6,1.7285346984863281e-6,4.172325134277344e-6,1.8775463104248047e-6,3.069639205932617e-6,1.3709068298339844e-6,2.652406692504883e-6,8.344650268554688e-7,1.2516975402832031e-6,5.364418029785156e-7,1.0728836059570312e-6,2.682209014892578e-7,5.960464477539062e-7,2.682209014892578e-7,4.172325134277344e-7,1.1920928955078125e-7,2.384185791015625e-7,5.960464477539063e-8,2.384185791015625e-7,2.9802322387695312e-8,1.4901161193847656e-7,2.9802322387695312e-8,1.1920928955078125e-7,2.9802322387695312e-8,5.960464477539063e-8,8.940696716308594e-8,1.4901161193847656e-7,2.9802322387695312e-8,1.1920928955078125e-7,0.0,1.4901161193847656e-7,2.9802322387695312e-8,1.1920928955078125e-7,2.9802322387695312e-8,1.7881393432617188e-7,2.9802322387695312e-8,8.940696716308594e-8,5.960464477539063e-8,1.4901161193847656e-7,0.0,8.940696716308594e-8,2.9802322387695312e-8,1.1920928955078125e-7,8.940696716308594e-8,1.1920928955078125e-7,1.4901161193847656e-7,3.2782554626464844e-7,2.682209014892578e-7,6.258487701416016e-7,4.470348358154297e-7,1.5497207641601562e-6,9.834766387939453e-7,2.205371856689453e-6,1.519918441772461e-6,4.0531158447265625e-6,2.7120113372802734e-6,6.318092346191406e-6,5.334615707397461e-6,0.000016570091247558594,0.000012278556823730469,0.000028014183044433594,0.000019252300262451172,0.000043898820877075195,0.000026345252990722656,0.000050008296966552734,0.00003546476364135742,0.00008806586265563965,0.00006219744682312012,0.00011432170867919922,0.00007334351539611816,0.00013592839241027832,0.00006815791130065918,0.00011029839515686035,0.00007399916648864746,0.00012424588203430176,0.00006267428398132324,0.00009271502494812012,0.00006693601608276367,0.00014257431030273438,0.00010475516319274902,0.00017255544662475586,0.00015208125114440918,0.0002727508544921875,0.00015491247177124023,0.00019225478172302246,0.0001189112663269043,0.0001501142978668213,0.00004869699478149414,0.00005125999450683594,0.000020712614059448242,0.00002047419548034668,3.0100345611572266e-6,2.8312206268310547e-6,8.344650268554688e-7,1.043081283569336e-6,2.384185791015625e-7,2.980232238769531e-7,1.1920928955078125e-7,1.7881393432617188e-7,0.0,1.4901161193847656e-7,8.940696716308594e-8,5.960464477539063e-8,0.0,1.1920928955078125e-7,5.960464477539063e-8,2.9802322387695312e-8,-5.960464477539063e-8,2.9802322387695312e-8,-5.960464477539063e-8,1.1920928955078125e-7,-5.960464477539063e-8,2.9802322387695312e-8,-5.960464477539063e-8,5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,5.960464477539063e-8,-5.960464477539063e-8,2.9802322387695312e-8,-5.960464477539063e-8,0.0,-5.960464477539063e-8,2.9802322387695312e-8,-5.960464477539063e-8,2.9802322387695312e-8,-5.960464477539063e-8,8.940696716308594e-8,-5.960464477539063e-8,2.9802322387695312e-8,-5.960464477539063e-8,2.9802322387695312e-8,-5.960464477539063e-8,8.940696716308594e-8,-5.960464477539063e-8,2.9802322387695312e-8,-5.960464477539063e-8,5.960464477539063e-8,0.0,5.960464477539063e-8,8.940696716308594e-8,1.1920928955078125e-7,2.9802322387695312e-8,1.1920928955078125e-7,2.9802322387695312e-8,1.4901161193847656e-7,8.940696716308594e-8,2.086162567138672e-7,1.1920928955078125e-7,4.76837158203125e-7,2.384185791015625e-7,4.76837158203125e-7,2.086162567138672e-7,4.470348358154297e-7,2.384185791015625e-7,2.086162567138672e-7,0.0,1.7881393432617188e-7,5.960464477539063e-8,1.1920928955078125e-7,1.7881393432617188e-7,6.854534149169922e-7,1.5497207641601562e-6,4.649162292480469e-6,0.00002422928810119629,0.0004768073558807373,0.019365936517715454,0.006869584321975708,0.0028767287731170654,0.005936235189437866,0.0049648284912109375,0.006009161472320557,0.00574067234992981,0.011233299970626831,0.019848257303237915,0.038144856691360474,0.044866204261779785,0.06367120146751404,0.07567378878593445,0.11070913076400757,0.12412858009338379,0.14877453446388245,0.1621476411819458,0.21278256177902222,0.22975540161132812,0.237778902053833,0.2452886998653412,0.27987587451934814,0.29475080966949463,0.2924577593803406,0.2875986695289612,0.3092496693134308,0.2973715662956238,0.27760231494903564,0.24046960473060608,0.22444981336593628,0.16208112239837646,0.14094603061676025,0.06233406066894531,0.03945353627204895,0.009642094373703003,0.006227940320968628,0.0006011426448822021,0.00020512938499450684,0.00002250075340270996,0.000025212764739990234,2.5331974029541016e-6,1.2516975402832031e-6,1.7881393432617188e-7,6.258487701416016e-7,1.4901161193847656e-7,2.682209014892578e-7,5.960464477539063e-8,6.258487701416016e-7,5.364418029785156e-7,1.519918441772461e-6,4.470348358154297e-7,1.6391277313232422e-6,5.662441253662109e-7,8.344650268554688e-7,1.7881393432617188e-7,3.8743019104003906e-7,8.940696716308594e-8,0.0,-5.960464477539063e-8,5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,2.9802322387695312e-8,8.940696716308594e-8,2.9802322387695312e-8,1.1920928955078125e-7,8.940696716308594e-8,2.086162567138672e-7,8.940696716308594e-8,4.76837158203125e-7,4.172325134277344e-7,1.0132789611816406e-6,4.172325134277344e-7,1.519918441772461e-6,7.748603820800781e-7,1.2814998626708984e-6,2.682209014892578e-7,9.238719940185547e-7,2.980232238769531e-7,3.8743019104003906e-7,5.960464477539063e-8,3.5762786865234375e-7,1.1920928955078125e-7,1.7881393432617188e-7,-5.960464477539063e-8,1.4901161193847656e-7,8.940696716308594e-8,5.960464477539063e-8,-5.960464477539063e-8,5.960464477539063e-8,0.0,2.9802322387695312e-8,-5.960464477539063e-8,8.940696716308594e-8,2.9802322387695312e-8,0.0,-5.960464477539063e-8,8.940696716308594e-8,0.0,2.9802322387695312e-8,-5.960464477539063e-8,8.940696716308594e-8,2.9802322387695312e-8,0.0,-5.960464477539063e-8,0.0,2.9802322387695312e-8,2.9802322387695312e-8,-5.960464477539063e-8,-5.960464477539063e-8,2.9802322387695312e-8,2.9802322387695312e-8,-5.960464477539063e-8,2.9802322387695312e-8,2.9802322387695312e-8,8.940696716308594e-8,2.9802322387695312e-8,1.4901161193847656e-7,1.7881393432617188e-7,4.172325134277344e-7,1.4901161193847656e-7,7.748603820800781e-7,5.662441253662109e-7,1.4007091522216797e-6,5.662441253662109e-7,2.115964889526367e-6,2.0265579223632812e-6,4.4405460357666016e-6,2.3245811462402344e-6,9.894371032714844e-6,9.626150131225586e-6,0.000019878149032592773,8.374452590942383e-6,0.00002771615982055664,0.00001952052116394043,0.000031441450119018555,0.000014841556549072266,0.00005125999450683594,0.000042438507080078125,0.00007572770118713379,0.00003129243850708008,0.00008016824722290039,0.000052303075790405273,0.00007238984107971191,0.0000292360782623291,0.00006756186485290527,0.00004413723945617676,0.00005391240119934082,0.000028520822525024414,0.00007167458534240723,0.00007113814353942871,0.00011593103408813477,0.00006783008575439453,0.0001494288444519043,0.00009083747863769531,0.00011375546455383301,0.00003966689109802246,0.00007265806198120117,0.00002276897430419922,0.000022113323211669922,4.291534423828125e-6,8.195638656616211e-6,1.2218952178955078e-6,1.043081283569336e-6,1.7881393432617188e-7,3.5762786865234375e-7,5.960464477539063e-8,5.960464477539063e-8,-5.960464477539063e-8,8.940696716308594e-8,0.0,0.0,-5.960464477539063e-8,8.940696716308594e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,0.0,-5.960464477539063e-8,0.0,-5.960464477539063e-8,2.9802322387695312e-8,0.0,8.940696716308594e-8,-5.960464477539063e-8,5.960464477539063e-8,0.0,5.960464477539063e-8,-5.960464477539063e-8,1.7881393432617188e-7,8.940696716308594e-8,1.1920928955078125e-7,8.940696716308594e-8,3.5762786865234375e-7,2.086162567138672e-7,3.5762786865234375e-7,1.4901161193847656e-7,3.2782554626464844e-7,1.1920928955078125e-7,1.4901161193847656e-7,5.960464477539063e-8,1.1920928955078125e-7,1.1920928955078125e-7,8.940696716308594e-8,0.0,1.4901161193847656e-7,5.960464477539062e-7,1.4007091522216797e-6,6.079673767089844e-6,0.00011983513832092285,0.012609094381332397,0.006560146808624268,0.004257470369338989,0.004214078187942505,0.007582992315292358,0.009072840213775635,0.012098908424377441,0.01607614755630493,0.040244877338409424,0.06905850768089294,0.09675511717796326,0.10755527019500732,0.16143950819969177,0.19566985964775085,0.2433457374572754,0.23540717363357544,0.27886611223220825,0.31143811345100403,0.36904218792915344,0.3417057991027832,0.3793210983276367,0.3993215262889862,0.4438783526420593,0.4056882858276367,0.44072848558425903,0.43993353843688965,0.45827212929725647,0.3982585072517395,0.40280428528785706,0.3471537232398987,0.3344241976737976,0.2377834916114807,0.18997779488563538,0.11782237887382507,0.05232158303260803,0.02003958821296692,0.004132479429244995,0.0011143088340759277,0.00017240643501281738,0.0000521242618560791,0.00001481175422668457,4.947185516357422e-6,1.519918441772461e-6,7.748603820800781e-7,7.450580596923828e-7,7.748603820800781e-7,6.258487701416016e-7,6.556510925292969e-7,2.1457672119140625e-6,4.172325134277344e-6,2.7120113372802734e-6,2.175569534301758e-6,2.4437904357910156e-6,2.9802322387695312e-6,8.344650268554688e-7,4.172325134277344e-7,2.980232238769531e-7,2.384185791015625e-7,0.0,5.960464477539063e-8,2.9802322387695312e-8,0.0,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,2.9802322387695312e-8,-5.960464477539063e-8,-5.960464477539063e-8,2.9802322387695312e-8,0.0,2.9802322387695312e-8,2.9802322387695312e-8,8.940696716308594e-8,1.4901161193847656e-7,1.1920928955078125e-7,1.4901161193847656e-7,2.682209014892578e-7,4.172325134277344e-7,3.5762786865234375e-7,3.5762786865234375e-7,8.642673492431641e-7,1.7881393432617188e-6,1.1622905731201172e-6,1.0132789611816406e-6,1.5795230865478516e-6,2.3543834686279297e-6,9.238719940185547e-7,6.258487701416016e-7,7.450580596923828e-7,8.344650268554688e-7,3.2782554626464844e-7,2.086162567138672e-7,2.682209014892578e-7,3.2782554626464844e-7,5.960464477539063e-8,1.4901161193847656e-7,1.7881393432617188e-7,2.086162567138672e-7,8.940696716308594e-8,8.940696716308594e-8,5.960464477539063e-8,5.960464477539063e-8,0.0,1.1920928955078125e-7,5.960464477539063e-8,1.1920928955078125e-7,2.9802322387695312e-8,2.9802322387695312e-8,5.960464477539063e-8,1.1920928955078125e-7,-5.960464477539063e-8,2.9802322387695312e-8,8.940696716308594e-8,1.7881393432617188e-7,5.960464477539063e-8,1.1920928955078125e-7,8.940696716308594e-8,1.7881393432617188e-7,2.9802322387695312e-8,2.9802322387695312e-8,1.1920928955078125e-7,1.7881393432617188e-7,1.1920928955078125e-7,1.4901161193847656e-7,2.086162567138672e-7,3.8743019104003906e-7,2.682209014892578e-7,3.2782554626464844e-7,7.152557373046875e-7,1.6093254089355469e-6,1.1920928955078125e-6,1.2814998626708984e-6,2.3245811462402344e-6,4.082918167114258e-6,3.0100345611572266e-6,3.0100345611572266e-6,6.3478946685791016e-6,0.000011473894119262695,9.804964065551758e-6,0.000011354684829711914,0.000022292137145996094,0.00003835558891296387,0.000027447938919067383,0.000027954578399658203,0.00003975629806518555,0.000058710575103759766,0.00004366040229797363,0.000049114227294921875,0.00007936358451843262,0.0001239180564880371,0.00008219480514526367,0.00008115172386169434,0.00010210275650024414,0.00013172626495361328,0.00007715821266174316,0.00006759166717529297,0.00008153915405273438,0.00009328126907348633,0.0000692903995513916,0.00006857514381408691,0.00013023614883422852,0.00018420815467834473,0.0001328587532043457,0.00011068582534790039,0.00013339519500732422,0.00013932585716247559,0.00006335973739624023,0.00003695487976074219,0.000030517578125,0.000023812055587768555,6.943941116333008e-6,2.9206275939941406e-6,1.7881393432617188e-6,1.1920928955078125e-6,3.2782554626464844e-7,1.7881393432617188e-7,1.7881393432617188e-7,1.7881393432617188e-7,8.940696716308594e-8,0.0,8.940696716308594e-8,1.7881393432617188e-7,2.9802322387695312e-8,5.960464477539063e-8,-5.960464477539063e-8,2.9802322387695312e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,2.9802322387695312e-8,-5.960464477539063e-8,-5.960464477539063e-8,2.9802322387695312e-8,8.940696716308594e-8,5.960464477539063e-8,0.0,2.9802322387695312e-8,2.9802322387695312e-8,2.9802322387695312e-8,2.9802322387695312e-8,0.0,8.940696716308594e-8,2.9802322387695312e-8,0.0,2.9802322387695312e-8,8.940696716308594e-8,0.0,2.9802322387695312e-8,5.960464477539063e-8,1.4901161193847656e-7,8.940696716308594e-8,5.960464477539063e-8,1.1920928955078125e-7,2.980232238769531e-7,1.4901161193847656e-7,1.4901161193847656e-7,4.172325134277344e-7,6.854534149169922e-7,5.364418029785156e-7,5.066394805908203e-7,1.0132789611816406e-6,1.7881393432617188e-6,8.940696716308594e-7,5.960464477539062e-7,7.152557373046875e-7,8.046627044677734e-7,2.086162567138672e-7,1.7881393432617188e-7,2.384185791015625e-7,2.682209014892578e-7,3.5762786865234375e-7,2.980232238769531e-7,1.0132789611816406e-6,1.1324882507324219e-6,3.6656856536865234e-6,0.0000737309455871582,0.010358065366744995,0.005437970161437988,0.0034765005111694336,0.005830526351928711,0.008596241474151611,0.010094285011291504,0.01952183246612549,0.025001704692840576,0.05651816725730896,0.08724719285964966,0.15226075053215027,0.1656276285648346,0.2200118601322174,0.24699527025222778,0.32215696573257446,0.32163751125335693,0.3472454845905304,0.37721964716911316,0.42665228247642517,0.47229403257369995,0.47733646631240845,0.5262007713317871,0.5577277541160583,0.5758357048034668,0.5524232387542725,0.5830184817314148,0.5903913378715515,0.583803117275238,0.5094260573387146,0.527377724647522,0.4989054501056671,0.4174478352069855,0.3025966286659241,0.2256428599357605,0.15567639470100403,0.07851725816726685,0.013051539659500122,0.003643035888671875,0.0007829070091247559,0.0003039240837097168,0.00003400444984436035,0.000011801719665527344,4.4405460357666016e-6,3.2782554626464844e-6,8.642673492431641e-7,9.238719940185547e-7,1.1622905731201172e-6,1.6987323760986328e-6,2.115964889526367e-6,4.947185516357422e-6,5.3942203521728516e-6,6.318092346191406e-6,3.248453140258789e-6,4.798173904418945e-6,1.7881393432617188e-6,1.4007091522216797e-6,2.682209014892578e-7,2.086162567138672e-7,1.1920928955078125e-7,8.940696716308594e-8,0.0,5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,0.0,-5.960464477539063e-8,0.0,0.0,2.9802322387695312e-8,0.0,0.0,0.0,0.0,2.9802322387695312e-8,1.1920928955078125e-7,5.960464477539063e-8,1.7881393432617188e-7,5.960464477539063e-8,2.980232238769531e-7,2.980232238769531e-7,4.76837158203125e-7,4.172325134277344e-7,9.834766387939453e-7,8.344650268554688e-7,1.2516975402832031e-6,7.152557373046875e-7,1.341104507446289e-6,7.152557373046875e-7,9.238719940185547e-7,3.2782554626464844e-7,4.76837158203125e-7,2.086162567138672e-7,3.5762786865234375e-7,1.1920928955078125e-7,2.384185791015625e-7,1.1920928955078125e-7,1.7881393432617188e-7,5.960464477539063e-8,1.1920928955078125e-7,5.960464477539063e-8,1.1920928955078125e-7,8.940696716308594e-8,5.960464477539063e-8,0.0,8.940696716308594e-8,2.9802322387695312e-8,5.960464477539063e-8,8.940696716308594e-8,5.960464477539063e-8,2.9802322387695312e-8,8.940696716308594e-8,5.960464477539063e-8,5.960464477539063e-8,8.940696716308594e-8,5.960464477539063e-8,8.940696716308594e-8,1.1920928955078125e-7,8.940696716308594e-8,2.384185791015625e-7,1.1920928955078125e-7,1.4901161193847656e-7,1.7881393432617188e-7,2.086162567138672e-7,2.682209014892578e-7,4.172325134277344e-7,2.980232238769531e-7,6.854534149169922e-7,9.5367431640625e-7,1.4007091522216797e-6,1.3709068298339844e-6,2.9802322387695312e-6,3.5762786865234375e-6,4.231929779052734e-6,3.1888484954833984e-6,6.377696990966797e-6,7.68899917602539e-6,9.03010368347168e-6,8.225440979003906e-6,0.000015854835510253906,0.000019550323486328125,0.000026851892471313477,0.000023692846298217773,0.000043004751205444336,0.00004145503044128418,0.000052809715270996094,0.00004073977470397949,0.00006404519081115723,0.0000642538070678711,0.00008669495582580566,0.00007745623588562012,0.00013467669486999512,0.00013244152069091797,0.0001564323902130127,0.00011426210403442383,0.00016605854034423828,0.00013303756713867188,0.00013956427574157715,0.00008916854858398438,0.00011989474296569824,0.00011351704597473145,0.00013712048530578613,0.00012025237083435059,0.00018095970153808594,0.00013777613639831543,0.00014930963516235352,0.00007256865501403809,0.00008687376976013184,0.00003546476364135742,0.00003427267074584961,0.000011056661605834961,0.00001099705696105957,3.039836883544922e-6,2.8312206268310547e-6,5.662441253662109e-7,5.662441253662109e-7,1.4901161193847656e-7,1.7881393432617188e-7,8.940696716308594e-8,5.960464477539063e-8,2.9802322387695312e-8,0.0,-5.960464477539063e-8,0.0,2.9802322387695312e-8,2.9802322387695312e-8,-5.960464477539063e-8,2.9802322387695312e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,0.0,-5.960464477539063e-8,2.9802322387695312e-8,0.0,1.1920928955078125e-7,0.0,5.960464477539063e-8,-5.960464477539063e-8,5.960464477539063e-8,2.9802322387695312e-8,5.960464477539063e-8,0.0,8.940696716308594e-8,0.0,8.940696716308594e-8,2.9802322387695312e-8,2.9802322387695312e-8,2.9802322387695312e-8,8.940696716308594e-8,8.940696716308594e-8,5.960464477539063e-8,8.940696716308594e-8,1.4901161193847656e-7,1.1920928955078125e-7,2.980232238769531e-7,1.4901161193847656e-7,3.5762786865234375e-7,4.76837158203125e-7,7.450580596923828e-7,6.258487701416016e-7,1.0132789611816406e-6,1.5497207641601562e-6,2.115964889526367e-6,1.7285346984863281e-6,3.129243850708008e-6,2.9206275939941406e-6,2.7418136596679688e-6,1.341104507446289e-6,1.7285346984863281e-6,7.450580596923828e-7,6.854534149169922e-7,3.5762786865234375e-7,5.066394805908203e-7,1.0132789611816406e-6,1.1026859283447266e-6,7.748603820800781e-7,5.364418029785156e-7,2.7418136596679688e-6,0.00003993511199951172,0.007946997880935669,0.0065888166427612305,0.0035552978515625,0.005549013614654541,0.010588198900222778,0.014325201511383057,0.022171765565872192,0.030603408813476562,0.060056209564208984,0.107767254114151,0.15644559264183044,0.18695029616355896,0.24611195921897888,0.28754913806915283,0.3376401662826538,0.3699960708618164,0.38343483209609985,0.4579521715641022,0.49760758876800537,0.5615085363388062,0.5633670687675476,0.6168015599250793,0.6384363770484924,0.6603186130523682,0.6769681572914124,0.6870238184928894,0.6929091811180115,0.6851742267608643,0.6614933013916016,0.6645145416259766,0.6243987679481506,0.5672487616539001,0.4701089560985565,0.36398035287857056,0.2811383008956909,0.14001783728599548,0.041773468255996704,0.008580207824707031,0.0019318163394927979,0.0005396604537963867,0.0000711977481842041,0.00001850724220275879,7.212162017822266e-6,3.814697265625e-6,1.7881393432617188e-6,1.2814998626708984e-6,1.1622905731201172e-6,1.3709068298339844e-6,2.7120113372802734e-6,5.692243576049805e-6,4.976987838745117e-6,6.467103958129883e-6,5.751848220825195e-6,6.467103958129883e-6,2.384185791015625e-6,1.5497207641601562e-6,5.364418029785156e-7,3.5762786865234375e-7,1.1920928955078125e-7,8.940696716308594e-8,2.9802322387695312e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,0.0,-5.960464477539063e-8,2.9802322387695312e-8,5.960464477539063e-8,8.940696716308594e-8,5.960464477539063e-8,0.0,2.9802322387695312e-8,8.940696716308594e-8,8.940696716308594e-8,8.940696716308594e-8,8.940696716308594e-8,1.7881393432617188e-7,1.1920928955078125e-7,1.4901161193847656e-7,1.7881393432617188e-7,2.384185791015625e-7,2.384185791015625e-7,2.980232238769531e-7,4.76837158203125e-7,7.152557373046875e-7,5.662441253662109e-7,6.854534149169922e-7,7.152557373046875e-7,9.238719940185547e-7,5.960464477539062e-7,4.76837158203125e-7,4.172325134277344e-7,4.172325134277344e-7,2.384185791015625e-7,1.4901161193847656e-7,1.1920928955078125e-7,1.1920928955078125e-7,5.960464477539063e-8,1.1920928955078125e-7,0.0,1.1920928955078125e-7,0.0,5.960464477539063e-8,5.960464477539063e-8,5.960464477539063e-8,5.960464477539063e-8,5.960464477539063e-8,5.960464477539063e-8,5.960464477539063e-8,1.1920928955078125e-7,2.9802322387695312e-8,8.940696716308594e-8,5.960464477539063e-8,2.9802322387695312e-8,1.4901161193847656e-7,5.960464477539063e-8,1.7881393432617188e-7,5.960464477539063e-8,1.1920928955078125e-7,2.384185791015625e-7,2.384185791015625e-7,1.4901161193847656e-7,1.7881393432617188e-7,2.980232238769531e-7,5.066394805908203e-7,4.172325134277344e-7,5.066394805908203e-7,8.344650268554688e-7,1.341104507446289e-6,1.5497207641601562e-6,2.115964889526367e-6,3.6954879760742188e-6,6.377696990966797e-6,5.8710575103759766e-6,6.67572021484375e-6,7.569789886474609e-6,0.000012189149856567383,0.000010758638381958008,0.000013023614883422852,0.000015497207641601562,0.000023305416107177734,0.000021755695343017578,0.000027120113372802734,0.00003212690353393555,0.0000489354133605957,0.00003960728645324707,0.000049233436584472656,0.000054836273193359375,0.00007316470146179199,0.00006395578384399414,0.00008228421211242676,0.00010442733764648438,0.00016164779663085938,0.00014284253120422363,0.0001774132251739502,0.0001754462718963623,0.00022479891777038574,0.00016823410987854004,0.00017392635345458984,0.00014477968215942383,0.00016349554061889648,0.00013503432273864746,0.00015166401863098145,0.00014674663543701172,0.000176161527633667,0.00012007355690002441,0.00009721517562866211,0.00005167722702026367,0.000044465065002441406,0.000022351741790771484,0.00001558661460876465,7.033348083496094e-6,4.857778549194336e-6,1.8477439880371094e-6,1.1622905731201172e-6,5.364418029785156e-7,2.980232238769531e-7,1.1920928955078125e-7,5.960464477539063e-8,2.9802322387695312e-8,2.9802322387695312e-8,0.0,0.0,0.0,2.9802322387695312e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,0.0,2.9802322387695312e-8,1.1920928955078125e-7,8.940696716308594e-8,8.940696716308594e-8,8.940696716308594e-8,5.960464477539063e-8,2.9802322387695312e-8,8.940696716308594e-8,5.960464477539063e-8,5.960464477539063e-8,2.9802322387695312e-8,0.0,8.940696716308594e-8,2.9802322387695312e-8,8.940696716308594e-8,2.9802322387695312e-8,2.9802322387695312e-8,5.960464477539063e-8,8.940696716308594e-8,5.960464477539063e-8,1.1920928955078125e-7,2.384185791015625e-7,1.4901161193847656e-7,2.980232238769531e-7,2.384185791015625e-7,4.76837158203125e-7,5.066394805908203e-7,6.556510925292969e-7,9.238719940185547e-7,1.519918441772461e-6,1.7285346984863281e-6,2.205371856689453e-6,3.4570693969726562e-6,5.424022674560547e-6,4.023313522338867e-6,3.874301910400391e-6,3.5762786865234375e-6,3.4570693969726562e-6,1.519918441772461e-6,1.1920928955078125e-6,1.519918441772461e-6,1.7285346984863281e-6,2.8312206268310547e-6,1.1622905731201172e-6,1.0728836059570312e-6,6.556510925292969e-7,2.175569534301758e-6,0.00002625584602355957,0.01220753788948059,0.004626601934432983,0.003471285104751587,0.0070021748542785645,0.014959156513214111,0.01746085286140442,0.02938714623451233,0.03883546590805054,0.08243522047996521,0.11089247465133667,0.15245765447616577,0.1858280897140503,0.247615247964859,0.2726067006587982,0.2902860641479492,0.34948164224624634,0.38967251777648926,0.45027443766593933,0.4841017723083496,0.55417799949646,0.6112068295478821,0.6538516283035278,0.6896997094154358,0.7066283226013184,0.7549391984939575,0.7595844268798828,0.781314492225647,0.765680730342865,0.7850497364997864,0.7501745223999023,0.7543432712554932,0.6825673580169678,0.6082164645195007,0.5047599673271179,0.3394518196582794,0.21655914187431335,0.07528024911880493,0.020449161529541016,0.00489276647567749,0.0011260509490966797,0.00015717744827270508,0.00002855062484741211,9.864568710327148e-6,4.470348358154297e-6,2.115964889526367e-6,1.1622905731201172e-6,9.238719940185547e-7,1.7881393432617188e-6,2.682209014892578e-6,3.874301910400391e-6,4.5299530029296875e-6,7.718801498413086e-6,6.4373016357421875e-6,5.900859832763672e-6,3.0994415283203125e-6,2.2649765014648438e-6,6.258487701416016e-7,2.980232238769531e-7,1.1920928955078125e-7,1.1920928955078125e-7,8.940696716308594e-8,5.960464477539063e-8,-5.960464477539063e-8,5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,2.9802322387695312e-8,2.9802322387695312e-8,5.960464477539063e-8,5.960464477539063e-8,2.9802322387695312e-8,2.9802322387695312e-8,0.0,2.9802322387695312e-8,5.960464477539063e-8,5.960464477539063e-8,8.940696716308594e-8,8.940696716308594e-8,1.1920928955078125e-7,5.960464477539063e-8,1.4901161193847656e-7,8.940696716308594e-8,2.980232238769531e-7,3.2782554626464844e-7,3.5762786865234375e-7,3.5762786865234375e-7,5.662441253662109e-7,4.172325134277344e-7,4.76837158203125e-7,3.5762786865234375e-7,4.76837158203125e-7,2.384185791015625e-7,2.682209014892578e-7,1.1920928955078125e-7,1.4901161193847656e-7,1.1920928955078125e-7,5.960464477539063e-8,8.940696716308594e-8,8.940696716308594e-8,5.960464477539063e-8,8.940696716308594e-8,8.940696716308594e-8,5.960464477539063e-8,8.940696716308594e-8,8.940696716308594e-8,2.9802322387695312e-8,8.940696716308594e-8,8.940696716308594e-8,0.0,1.1920928955078125e-7,8.940696716308594e-8,2.9802322387695312e-8,2.9802322387695312e-8,8.940696716308594e-8,8.940696716308594e-8,8.940696716308594e-8,5.960464477539063e-8,8.940696716308594e-8,1.1920928955078125e-7,1.7881393432617188e-7,1.7881393432617188e-7,2.384185791015625e-7,2.980232238769531e-7,3.5762786865234375e-7,4.76837158203125e-7,6.258487701416016e-7,9.238719940185547e-7,1.2516975402832031e-6,1.5795230865478516e-6,2.205371856689453e-6,4.023313522338867e-6,5.602836608886719e-6,7.212162017822266e-6,7.599592208862305e-6,0.000011295080184936523,0.00001290440559387207,0.0000152587890625,0.000015050172805786133,0.00002065300941467285,0.00001990795135498047,0.000022739171981811523,0.000021189451217651367,0.000032961368560791016,0.00003316998481750488,0.0000407099723815918,0.000037550926208496094,0.00005397200584411621,0.000056743621826171875,0.00006848573684692383,0.00006374716758728027,0.00008937716484069824,0.0001246631145477295,0.00015929341316223145,0.00015404820442199707,0.00020051002502441406,0.000213623046875,0.00024503469467163086,0.0001926422119140625,0.00021573901176452637,0.00017258524894714355,0.0001818835735321045,0.0001455843448638916,0.00017333030700683594,0.00012862682342529297,0.0001284182071685791,0.0000718235969543457,0.00007042288780212402,0.000027745962142944336,0.000019043684005737305,8.285045623779297e-6,9.506940841674805e-6,2.950429916381836e-6,1.9371509552001953e-6,6.258487701416016e-7,7.748603820800781e-7,2.384185791015625e-7,1.4901161193847656e-7,8.940696716308594e-8,8.940696716308594e-8,0.0,2.9802322387695312e-8,-5.960464477539063e-8,2.9802322387695312e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,2.9802322387695312e-8,2.9802322387695312e-8,2.9802322387695312e-8,5.960464477539063e-8,8.940696716308594e-8,0.0,1.1920928955078125e-7,1.4901161193847656e-7,1.7881393432617188e-7,5.960464477539063e-8,8.940696716308594e-8,8.940696716308594e-8,5.960464477539063e-8,8.940696716308594e-8,8.940696716308594e-8,2.9802322387695312e-8,0.0,8.940696716308594e-8,8.940696716308594e-8,2.9802322387695312e-8,5.960464477539063e-8,5.960464477539063e-8,1.4901161193847656e-7,1.4901161193847656e-7,2.980232238769531e-7,2.682209014892578e-7,2.980232238769531e-7,4.172325134277344e-7,7.748603820800781e-7,8.344650268554688e-7,1.0132789611816406e-6,1.430511474609375e-6,3.069639205932617e-6,3.9637088775634766e-6,4.827976226806641e-6,4.559755325317383e-6,6.9141387939453125e-6,5.0961971282958984e-6,4.559755325317383e-6,3.3974647521972656e-6,3.159046173095703e-6,2.682209014892578e-6,2.4139881134033203e-6,4.887580871582031e-6,2.1457672119140625e-6,1.6391277313232422e-6,5.364418029785156e-7,1.4007091522216797e-6,0.000025779008865356445,0.01100584864616394,0.006026625633239746,0.0056504905223846436,0.007992446422576904,0.019365817308425903,0.027635425329208374,0.04271203279495239,0.051993370056152344,0.082439124584198,0.12886828184127808,0.16220343112945557,0.18660083413124084,0.22162732481956482,0.2594706416130066,0.2767264246940613,0.3345777988433838,0.36323481798171997,0.437751829624176,0.4922510087490082,0.5791779160499573,0.6164031028747559,0.6882603764533997,0.7163942456245422,0.7706749439239502,0.7949490547180176,0.8298821449279785,0.8374290466308594,0.8597403764724731,0.8530761003494263,0.857259213924408,0.8332648277282715,0.8064218759536743,0.7266439199447632,0.6464816331863403,0.48782649636268616,0.2838912010192871,0.12234228849411011,0.04395046830177307,0.0075501203536987305,0.0015063285827636719,0.00035962462425231934,0.000055283308029174805,0.000015556812286376953,4.172325134277344e-6,2.4437904357910156e-6,1.4007091522216797e-6,1.1026859283447266e-6,1.0728836059570312e-6,2.0563602447509766e-6,3.6954879760742188e-6,4.738569259643555e-6,4.380941390991211e-6,5.692243576049805e-6,6.496906280517578e-6,3.248453140258789e-6,1.0728836059570312e-6,6.258487701416016e-7,3.2782554626464844e-7,1.1920928955078125e-7,2.9802322387695312e-8,2.9802322387695312e-8,2.9802322387695312e-8,5.960464477539063e-8,-5.960464477539063e-8,0.0,2.9802322387695312e-8,2.9802322387695312e-8,-5.960464477539063e-8,-5.960464477539063e-8,0.0,2.9802322387695312e-8,5.960464477539063e-8,0.0,0.0,0.0,2.9802322387695312e-8,8.940696716308594e-8,5.960464477539063e-8,8.940696716308594e-8,0.0,5.960464477539063e-8,1.1920928955078125e-7,1.4901161193847656e-7,1.7881393432617188e-7,2.086162567138672e-7,2.980232238769531e-7,2.682209014892578e-7,2.384185791015625e-7,3.8743019104003906e-7,3.5762786865234375e-7,2.980232238769531e-7,1.4901161193847656e-7,2.086162567138672e-7,2.384185791015625e-7,1.7881393432617188e-7,5.960464477539063e-8,5.960464477539063e-8,5.960464477539063e-8,8.940696716308594e-8,0.0,5.960464477539063e-8,5.960464477539063e-8,5.960464477539063e-8,2.9802322387695312e-8,8.940696716308594e-8,8.940696716308594e-8,8.940696716308594e-8,1.1920928955078125e-7,0.0,8.940696716308594e-8,0.0,8.940696716308594e-8,2.9802322387695312e-8,5.960464477539063e-8,8.940696716308594e-8,8.940696716308594e-8,5.960464477539063e-8,1.1920928955078125e-7,1.7881393432617188e-7,8.940696716308594e-8,1.7881393432617188e-7,2.384185791015625e-7,2.980232238769531e-7,2.980232238769531e-7,5.364418029785156e-7,7.748603820800781e-7,1.0132789611816406e-6,9.238719940185547e-7,1.5497207641601562e-6,2.4139881134033203e-6,3.1888484954833984e-6,2.771615982055664e-6,5.245208740234375e-6,8.404254913330078e-6,9.447336196899414e-6,7.361173629760742e-6,0.000013530254364013672,0.000017702579498291016,0.000016957521438598633,0.000011831521987915039,0.000017702579498291016,0.00002065300941467285,0.00001963973045349121,0.000016480684280395508,0.000028342008590698242,0.00003981590270996094,0.00003832578659057617,0.00003319978713989258,0.00005316734313964844,0.00007459521293640137,0.00007426738739013672,0.0000648200511932373,0.00012156367301940918,0.0001685619354248047,0.00016775727272033691,0.0001328587532043457,0.00020840764045715332,0.0002446472644805908,0.00020065903663635254,0.00013467669486999512,0.0001824498176574707,0.0001787245273590088,0.00014457106590270996,0.0000908970832824707,0.0001049041748046875,0.00008940696716308594,0.00005048513412475586,0.000025093555450439453,0.000017911195755004883,0.000012069940567016602,5.632638931274414e-6,2.7418136596679688e-6,1.8477439880371094e-6,1.1920928955078125e-6,5.066394805908203e-7,2.384185791015625e-7,2.384185791015625e-7,1.1920928955078125e-7,5.960464477539063e-8,2.9802322387695312e-8,5.960464477539063e-8,8.940696716308594e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,0.0,2.9802322387695312e-8,2.9802322387695312e-8,0.0,8.940696716308594e-8,1.1920928955078125e-7,8.940696716308594e-8,1.4901161193847656e-7,2.682209014892578e-7,2.086162567138672e-7,5.960464477539063e-8,1.4901161193847656e-7,1.4901161193847656e-7,1.1920928955078125e-7,0.0,0.0,5.960464477539063e-8,2.9802322387695312e-8,2.9802322387695312e-8,2.9802322387695312e-8,5.960464477539063e-8,5.960464477539063e-8,2.9802322387695312e-8,1.4901161193847656e-7,1.7881393432617188e-7,1.4901161193847656e-7,1.1920928955078125e-7,2.384185791015625e-7,3.2782554626464844e-7,3.5762786865234375e-7,2.980232238769531e-7,6.556510925292969e-7,9.834766387939453e-7,1.3709068298339844e-6,1.341104507446289e-6,2.8014183044433594e-6,4.32133674621582e-6,4.5299530029296875e-6,4.172325134277344e-6,5.21540641784668e-6,6.16908073425293e-6,5.245208740234375e-6,3.635883331298828e-6,6.258487701416016e-6,7.957220077514648e-6,8.940696716308594e-6,1.8477439880371094e-6,1.2516975402832031e-6,4.172325134277344e-7,1.341104507446289e-6,0.000022977590560913086,0.015229076147079468,0.009210675954818726,0.009308010339736938,0.01744842529296875,0.036470890045166016,0.046214282512664795,0.0756581723690033,0.08698135614395142,0.12448090314865112,0.1544351577758789,0.17177972197532654,0.19549289345741272,0.20874163508415222,0.23521113395690918,0.2371140718460083,0.2953578233718872,0.304996132850647,0.3638531565666199,0.4059973359107971,0.5056499242782593,0.6077192425727844,0.6735491752624512,0.7407839894294739,0.7669102549552917,0.8240711688995361,0.8442282676696777,0.8753251433372498,0.8722432255744934,0.8957279920578003,0.8766826391220093,0.8813464641571045,0.835584282875061,0.806495189666748,0.7107901573181152,0.5928788781166077,0.4055368900299072,0.1668093204498291,0.07454398274421692,0.01721969246864319,0.00577971339225769,0.0007209479808807373,0.00017240643501281738,0.00003084540367126465,0.000017344951629638672,2.592802047729492e-6,1.8775463104248047e-6,1.0728836059570312e-6,1.519918441772461e-6,1.8477439880371094e-6,4.291534423828125e-6,4.172325134277344e-6,6.288290023803711e-6,5.453824996948242e-6,0.000010192394256591797,3.129243850708008e-6,1.996755599975586e-6,4.470348358154297e-7,4.172325134277344e-7,1.4901161193847656e-7,8.940696716308594e-8,-5.960464477539063e-8,2.9802322387695312e-8,5.960464477539063e-8,5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,2.9802322387695312e-8,0.0,2.9802322387695312e-8,8.940696716308594e-8,0.0,2.9802322387695312e-8,5.960464477539063e-8,8.940696716308594e-8,0.0,2.9802322387695312e-8,2.9802322387695312e-8,5.960464477539063e-8,2.9802322387695312e-8,5.960464477539063e-8,5.960464477539063e-8,1.1920928955078125e-7,5.960464477539063e-8,1.1920928955078125e-7,1.7881393432617188e-7,2.086162567138672e-7,2.384185791015625e-7,2.086162567138672e-7,1.4901161193847656e-7,3.2782554626464844e-7,2.086162567138672e-7,1.7881393432617188e-7,1.4901161193847656e-7,1.1920928955078125e-7,5.960464477539063e-8,1.4901161193847656e-7,8.940696716308594e-8,5.960464477539063e-8,8.940696716308594e-8,8.940696716308594e-8,8.940696716308594e-8,5.960464477539063e-8,1.4901161193847656e-7,8.940696716308594e-8,8.940696716308594e-8,1.7881393432617188e-7,5.960464477539063e-8,5.960464477539063e-8,8.940696716308594e-8,8.940696716308594e-8,2.9802322387695312e-8,5.960464477539063e-8,2.9802322387695312e-8,8.940696716308594e-8,2.9802322387695312e-8,0.0,5.960464477539063e-8,1.7881393432617188e-7,5.960464477539063e-8,1.4901161193847656e-7,1.4901161193847656e-7,4.470348358154297e-7,2.980232238769531e-7,5.364418029785156e-7,5.662441253662109e-7,1.2814998626708984e-6,1.1026859283447266e-6,1.4901161193847656e-6,1.6391277313232422e-6,3.6954879760742188e-6,3.3676624298095703e-6,4.4405460357666016e-6,4.976987838745117e-6,0.000010579824447631836,8.910894393920898e-6,0.000011324882507324219,0.000011652708053588867,0.00002142786979675293,0.000014692544937133789,0.00001749396324157715,0.000013232231140136719,0.00001952052116394043,0.000013470649719238281,0.00001823902130126953,0.000020682811737060547,0.00003981590270996094,0.000034421682357788086,0.000046819448471069336,0.000050514936447143555,0.00008979439735412598,0.00008347630500793457,0.00010201334953308105,0.00010776519775390625,0.0001836717128753662,0.0001703500747680664,0.00019559264183044434,0.00016379356384277344,0.00023373961448669434,0.00017279386520385742,0.00018683075904846191,0.00012817978858947754,0.00016605854034423828,0.00010368227958679199,0.00011083483695983887,0.00006073713302612305,0.0000673532485961914,0.00002518296241760254,0.000023424625396728516,9.03010368347168e-6,8.52346420288086e-6,2.5331974029541016e-6,2.4437904357910156e-6,9.238719940185547e-7,8.642673492431641e-7,2.980232238769531e-7,2.980232238769531e-7,1.4901161193847656e-7,1.7881393432617188e-7,2.9802322387695312e-8,8.940696716308594e-8,0.0,8.940696716308594e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,0.0,8.940696716308594e-8,8.940696716308594e-8,8.940696716308594e-8,2.384185791015625e-7,2.682209014892578e-7,3.5762786865234375e-7,3.5762786865234375e-7,7.152557373046875e-7,4.76837158203125e-7,5.364418029785156e-7,2.682209014892578e-7,4.76837158203125e-7,2.086162567138672e-7,2.086162567138672e-7,8.940696716308594e-8,5.960464477539063e-8,2.9802322387695312e-8,8.940696716308594e-8,0.0,5.960464477539063e-8,2.9802322387695312e-8,8.940696716308594e-8,8.940696716308594e-8,1.7881393432617188e-7,1.7881393432617188e-7,2.086162567138672e-7,1.4901161193847656e-7,2.980232238769531e-7,2.682209014892578e-7,4.172325134277344e-7,3.2782554626464844e-7,8.642673492431641e-7,7.748603820800781e-7,1.3709068298339844e-6,1.9073486328125e-6,4.32133674621582e-6,3.7550926208496094e-6,5.0961971282958984e-6,5.125999450683594e-6,0.000010669231414794922,7.569789886474609e-6,7.510185241699219e-6,8.285045623779297e-6,0.000017940998077392578,0.00001633167266845703,9.417533874511719e-6,1.7881393432617188e-6,5.364418029785156e-7,1.7881393432617188e-6,0.000023484230041503906,0.02120184898376465,0.01817283034324646,0.015998661518096924,0.02803778648376465,0.06691443920135498,0.08451169729232788,0.1208716630935669,0.13313078880310059,0.17287245392799377,0.18420511484146118,0.19652223587036133,0.21177199482917786,0.2188510298728943,0.22367769479751587,0.23204296827316284,0.26334595680236816,0.29409539699554443,0.34131568670272827,0.3939729332923889,0.46515682339668274,0.5707066059112549,0.6790859699249268,0.7140561938285828,0.7843294143676758,0.8225016593933105,0.8649968504905701,0.8733683824539185,0.8991422057151794,0.9004687666893005,0.910798966884613,0.8924678564071655,0.8801893591880798,0.8322690725326538,0.7912240028381348,0.7096178531646729,0.5585001707077026,0.29506218433380127,0.13637295365333557,0.03838053345680237,0.011121183633804321,0.001711040735244751,0.00024372339248657227,0.00007495284080505371,0.00002372264862060547,6.318092346191406e-6,2.086162567138672e-6,1.3709068298339844e-6,1.3113021850585938e-6,2.8908252716064453e-6,4.589557647705078e-6,4.351139068603516e-6,7.212162017822266e-6,8.07642936706543e-6,9.98377799987793e-6,3.844499588012695e-6,2.0563602447509766e-6,8.046627044677734e-7,2.980232238769531e-7,1.7881393432617188e-7,8.940696716308594e-8,5.960464477539063e-8,2.9802322387695312e-8,-5.960464477539063e-8,2.9802322387695312e-8,0.0,8.940696716308594e-8,-5.960464477539063e-8,2.9802322387695312e-8,1.1920928955078125e-7,2.9802322387695312e-8,2.9802322387695312e-8,2.9802322387695312e-8,0.0,-5.960464477539063e-8,8.940696716308594e-8,2.9802322387695312e-8,5.960464477539063e-8,5.960464477539063e-8,2.9802322387695312e-8,8.940696716308594e-8,8.940696716308594e-8,1.1920928955078125e-7,8.940696716308594e-8,8.940696716308594e-8,1.7881393432617188e-7,1.4901161193847656e-7,5.960464477539063e-8,5.960464477539063e-8,1.4901161193847656e-7,1.7881393432617188e-7,1.7881393432617188e-7,5.960464477539063e-8,1.7881393432617188e-7,1.1920928955078125e-7,5.960464477539063e-8,8.940696716308594e-8,5.960464477539063e-8,8.940696716308594e-8,0.0,8.940696716308594e-8,8.940696716308594e-8,8.940696716308594e-8,5.960464477539063e-8,5.960464477539063e-8,5.960464477539063e-8,1.1920928955078125e-7,1.1920928955078125e-7,5.960464477539063e-8,1.1920928955078125e-7,8.940696716308594e-8,1.4901161193847656e-7,2.9802322387695312e-8,8.940696716308594e-8,8.940696716308594e-8,8.940696716308594e-8,5.960464477539063e-8,1.7881393432617188e-7,1.7881393432617188e-7,1.1920928955078125e-7,1.7881393432617188e-7,3.8743019104003906e-7,4.172325134277344e-7,4.470348358154297e-7,5.662441253662109e-7,1.1622905731201172e-6,1.4603137969970703e-6,1.519918441772461e-6,1.6093254089355469e-6,2.8014183044433594e-6,4.023313522338867e-6,3.933906555175781e-6,5.036592483520508e-6,7.033348083496094e-6,0.000010728836059570312,9.506940841674805e-6,0.000011861324310302734,0.000014334917068481445,0.000018417835235595703,0.000015139579772949219,0.00001481175422668457,0.000015288591384887695,0.00001570582389831543,0.000012516975402832031,0.0000152587890625,0.000025570392608642578,0.00003725290298461914,0.000036656856536865234,0.00004887580871582031,0.00007170438766479492,0.00009900331497192383,0.00009337067604064941,0.0001067817211151123,0.00013694167137145996,0.00017511844635009766,0.00017002224922180176,0.0001742541790008545,0.0001723170280456543,0.00018736720085144043,0.00015529990196228027,0.00014099478721618652,0.0001347064971923828,0.00012597441673278809,0.00009113550186157227,0.00008061528205871582,0.000060617923736572266,0.00004413723945617676,0.00002199411392211914,0.00001621246337890625,0.000010967254638671875,6.020069122314453e-6,2.4437904357910156e-6,1.6391277313232422e-6,1.2516975402832031e-6,6.258487701416016e-7,3.2782554626464844e-7,1.4901161193847656e-7,1.4901161193847656e-7,8.940696716308594e-8,8.940696716308594e-8,5.960464477539063e-8,2.9802322387695312e-8,0.0,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,2.9802322387695312e-8,0.0,8.940696716308594e-8,5.960464477539063e-8,1.4901161193847656e-7,2.980232238769531e-7,4.172325134277344e-7,5.662441253662109e-7,1.0728836059570312e-6,1.5497207641601562e-6,1.0132789611816406e-6,1.1324882507324219e-6,1.0132789611816406e-6,8.642673492431641e-7,4.470348358154297e-7,2.980232238769531e-7,2.682209014892578e-7,1.7881393432617188e-7,1.1920928955078125e-7,8.940696716308594e-8,1.4901161193847656e-7,5.960464477539063e-8,5.960464477539063e-8,5.960464477539063e-8,1.4901161193847656e-7,1.4901161193847656e-7,2.086162567138672e-7,1.1920928955078125e-7,2.086162567138672e-7,2.682209014892578e-7,2.980232238769531e-7,3.5762786865234375e-7,5.364418029785156e-7,7.152557373046875e-7,6.854534149169922e-7,1.0132789611816406e-6,1.9669532775878906e-6,3.516674041748047e-6,3.129243850708008e-6,4.410743713378906e-6,6.794929504394531e-6,9.47713851928711e-6,7.599592208862305e-6,8.970499038696289e-6,0.000014454126358032227,0.00002378225326538086,0.000025898218154907227,0.000010162591934204102,2.175569534301758e-6,1.043081283569336e-6,2.0563602447509766e-6,0.000025779008865356445,0.04437190294265747,0.024347543716430664,0.027952581644058228,0.054545849561691284,0.12353146076202393,0.1346849501132965,0.2108120322227478,0.2000553011894226,0.23579898476600647,0.23572248220443726,0.24180439114570618,0.21167996525764465,0.22697073221206665,0.21896010637283325,0.23285630345344543,0.22638121247291565,0.28170257806777954,0.30034926533699036,0.3651110529899597,0.41907286643981934,0.5827403664588928,0.6848248243331909,0.7517743110656738,0.7915359735488892,0.8500856161117554,0.8746166229248047,0.9008620977401733,0.9069416522979736,0.9277172088623047,0.918920636177063,0.9246749877929688,0.8939839005470276,0.8831466436386108,0.8201249241828918,0.7757402658462524,0.6292946338653564,0.37528160214424133,0.19007661938667297,0.06483417749404907,0.020119309425354004,0.0026406943798065186,0.0004496574401855469,0.00012350082397460938,0.0000349581241607666,5.841255187988281e-6,1.9371509552001953e-6,9.238719940185547e-7,1.5795230865478516e-6,2.771615982055664e-6,4.798173904418945e-6,4.470348358154297e-6,8.046627044677734e-6,8.225440979003906e-6,9.834766387939453e-6,3.7848949432373047e-6,2.3245811462402344e-6,4.76837158203125e-7,2.384185791015625e-7,5.960464477539063e-8,1.1920928955078125e-7,0.0,2.9802322387695312e-8,2.9802322387695312e-8,8.940696716308594e-8,2.9802322387695312e-8,2.9802322387695312e-8,2.9802322387695312e-8,2.9802322387695312e-8,5.960464477539063e-8,1.1920928955078125e-7,5.960464477539063e-8,2.9802322387695312e-8,2.9802322387695312e-8,2.9802322387695312e-8,5.960464477539063e-8,2.9802322387695312e-8,2.9802322387695312e-8,8.940696716308594e-8,5.960464477539063e-8,2.9802322387695312e-8,0.0,5.960464477539063e-8,2.9802322387695312e-8,8.940696716308594e-8,8.940696716308594e-8,8.940696716308594e-8,2.9802322387695312e-8,0.0,8.940696716308594e-8,5.960464477539063e-8,5.960464477539063e-8,1.7881393432617188e-7,0.0,8.940696716308594e-8,2.9802322387695312e-8,5.960464477539063e-8,8.940696716308594e-8,2.9802322387695312e-8,2.9802322387695312e-8,8.940696716308594e-8,-5.960464477539063e-8,8.940696716308594e-8,2.9802322387695312e-8,2.086162567138672e-7,5.960464477539063e-8,1.1920928955078125e-7,5.960464477539063e-8,5.960464477539063e-8,5.960464477539063e-8,5.960464477539063e-8,2.9802322387695312e-8,5.960464477539063e-8,8.940696716308594e-8,5.960464477539063e-8,5.960464477539063e-8,1.1920928955078125e-7,5.960464477539063e-8,1.7881393432617188e-7,1.1920928955078125e-7,2.384185791015625e-7,2.682209014892578e-7,3.8743019104003906e-7,4.172325134277344e-7,7.450580596923828e-7,8.642673492431641e-7,1.2218952178955078e-6,1.1622905731201172e-6,1.9073486328125e-6,2.6226043701171875e-6,4.023313522338867e-6,3.993511199951172e-6,5.930662155151367e-6,7.063150405883789e-6,0.000010907649993896484,8.463859558105469e-6,0.000012248754501342773,0.000011324882507324219,0.00001582503318786621,0.00001189112663269043,0.00001537799835205078,0.000010788440704345703,0.000011861324310302734,8.493661880493164e-6,0.000014513731002807617,0.00002014636993408203,0.000033974647521972656,0.0000337064266204834,0.0000527501106262207,0.00006589293479919434,0.00009775161743164062,0.0000787973403930664,0.00010883808135986328,0.00011223554611206055,0.00015804171562194824,0.00012293457984924316,0.00015687942504882812,0.00011882185935974121,0.0001392960548400879,0.00009116530418395996,0.00011739134788513184,0.00008279085159301758,0.00008946657180786133,0.00005453824996948242,0.00007081031799316406,0.00003427267074584961,0.000029772520065307617,0.000012069940567016602,0.000014543533325195312,5.841255187988281e-6,4.470348358154297e-6,1.5497207641601562e-6,1.7285346984863281e-6,5.662441253662109e-7,4.76837158203125e-7,2.086162567138672e-7,2.086162567138672e-7,8.940696716308594e-8,8.940696716308594e-8,0.0,2.9802322387695312e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,2.9802322387695312e-8,2.9802322387695312e-8,0.0,8.940696716308594e-8,1.4901161193847656e-7,1.4901161193847656e-7,3.5762786865234375e-7,4.470348358154297e-7,9.238719940185547e-7,1.4603137969970703e-6,2.4139881134033203e-6,1.8477439880371094e-6,2.384185791015625e-6,1.6689300537109375e-6,1.6093254089355469e-6,8.344650268554688e-7,6.854534149169922e-7,3.5762786865234375e-7,2.384185791015625e-7,1.1920928955078125e-7,1.1920928955078125e-7,1.1920928955078125e-7,1.1920928955078125e-7,1.4901161193847656e-7,1.1920928955078125e-7,1.7881393432617188e-7,1.4901161193847656e-7,1.7881393432617188e-7,2.086162567138672e-7,2.086162567138672e-7,2.980232238769531e-7,2.086162567138672e-7,3.5762786865234375e-7,3.5762786865234375e-7,5.364418029785156e-7,4.172325134277344e-7,9.834766387939453e-7,1.3113021850585938e-6,2.5331974029541016e-6,2.473592758178711e-6,4.112720489501953e-6,5.3942203521728516e-6,8.13603401184082e-6,8.225440979003906e-6,9.685754776000977e-6,7.450580596923828e-6,0.000010192394256591797,0.000018090009689331055,0.00001239776611328125,5.0067901611328125e-6,1.519918441772461e-6,2.384185791015625e-6,0.00003552436828613281,0.05901539325714111,0.04007616639137268,0.05077233910560608,0.08011108636856079,0.1567268669605255,0.2136884331703186,0.2723526954650879,0.28884637355804443,0.2944284677505493,0.32068705558776855,0.2870666980743408,0.25946229696273804,0.2522481083869934,0.2451545000076294,0.24863433837890625,0.24759051203727722,0.29470059275627136,0.3156857490539551,0.3646780252456665,0.45007753372192383,0.5760045647621155,0.6734926700592041,0.7332916259765625,0.7938055992126465,0.8433656692504883,0.8805509805679321,0.8963266611099243,0.9190279245376587,0.9253818392753601,0.9361651539802551,0.9278515577316284,0.9243308305740356,0.8905264735221863,0.8693556785583496,0.8082855939865112,0.704727053642273,0.4973960220813751,0.27022111415863037,0.09366169571876526,0.026001542806625366,0.00573459267616272,0.0008111894130706787,0.0001844167709350586,0.000029802322387695312,7.599592208862305e-6,1.4901161193847656e-6,1.0132789611816406e-6,8.344650268554688e-7,2.7120113372802734e-6,4.32133674621582e-6,5.841255187988281e-6,5.4836273193359375e-6,7.808208465576172e-6,7.897615432739258e-6,3.427267074584961e-6,8.642673492431641e-7,4.76837158203125e-7,1.4901161193847656e-7,1.4901161193847656e-7,2.9802322387695312e-8,2.9802322387695312e-8,5.960464477539063e-8,5.960464477539063e-8,-5.960464477539063e-8,2.9802322387695312e-8,5.960464477539063e-8,2.9802322387695312e-8,0.0,2.9802322387695312e-8,2.9802322387695312e-8,0.0,-5.960464477539063e-8,5.960464477539063e-8,-5.960464477539063e-8,2.9802322387695312e-8,0.0,2.9802322387695312e-8,2.9802322387695312e-8,2.9802322387695312e-8,0.0,2.9802322387695312e-8,8.940696716308594e-8,2.9802322387695312e-8,2.9802322387695312e-8,0.0,2.9802322387695312e-8,0.0,1.1920928955078125e-7,8.940696716308594e-8,5.960464477539063e-8,2.9802322387695312e-8,2.9802322387695312e-8,2.9802322387695312e-8,2.9802322387695312e-8,8.940696716308594e-8,-5.960464477539063e-8,0.0,8.940696716308594e-8,2.9802322387695312e-8,2.9802322387695312e-8,0.0,2.9802322387695312e-8,2.9802322387695312e-8,0.0,5.960464477539063e-8,1.4901161193847656e-7,8.940696716308594e-8,0.0,5.960464477539063e-8,2.9802322387695312e-8,2.9802322387695312e-8,8.940696716308594e-8,8.940696716308594e-8,5.960464477539063e-8,1.4901161193847656e-7,8.940696716308594e-8,5.960464477539063e-8,1.1920928955078125e-7,1.4901161193847656e-7,1.1920928955078125e-7,2.682209014892578e-7,3.5762786865234375e-7,5.066394805908203e-7,4.172325134277344e-7,7.748603820800781e-7,9.834766387939453e-7,1.2516975402832031e-6,9.5367431640625e-7,2.4437904357910156e-6,3.4570693969726562e-6,4.172325134277344e-6,3.1888484954833984e-6,6.645917892456055e-6,8.225440979003906e-6,8.255243301391602e-6,5.632638931274414e-6,9.98377799987793e-6,0.000010460615158081055,9.626150131225586e-6,6.109476089477539e-6,8.553266525268555e-6,7.62939453125e-6,7.62939453125e-6,6.794929504394531e-6,0.000016510486602783203,0.000023633241653442383,0.000030249357223510742,0.000027000904083251953,0.000051349401473999023,0.0000635981559753418,0.00006505846977233887,0.00005123019218444824,0.00008392333984375,0.00009369850158691406,0.00008550286293029785,0.00006431341171264648,0.00008067488670349121,0.00007909536361694336,0.0000603795051574707,0.0000457763671875,0.0000565648078918457,0.00005361437797546387,0.00003769993782043457,0.000024706125259399414,0.00002434849739074707,0.000017702579498291016,9.298324584960938e-6,5.245208740234375e-6,4.678964614868164e-6,3.069639205932617e-6,1.4603137969970703e-6,7.450580596923828e-7,5.662441253662109e-7,3.5762786865234375e-7,1.7881393432617188e-7,8.940696716308594e-8,1.4901161193847656e-7,5.960464477539063e-8,5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,2.9802322387695312e-8,8.940696716308594e-8,0.0,8.940696716308594e-8,2.086162567138672e-7,2.086162567138672e-7,4.172325134277344e-7,5.364418029785156e-7,1.519918441772461e-6,2.2649765014648438e-6,2.5033950805664062e-6,1.8477439880371094e-6,2.8014183044433594e-6,2.4437904357910156e-6,1.4603137969970703e-6,5.662441253662109e-7,6.556510925292969e-7,3.5762786865234375e-7,2.384185791015625e-7,1.1920928955078125e-7,2.384185791015625e-7,1.4901161193847656e-7,1.4901161193847656e-7,8.940696716308594e-8,2.086162567138672e-7,1.4901161193847656e-7,1.7881393432617188e-7,1.1920928955078125e-7,2.682209014892578e-7,2.384185791015625e-7,2.682209014892578e-7,2.384185791015625e-7,3.5762786865234375e-7,4.172325134277344e-7,5.364418029785156e-7,4.76837158203125e-7,1.0728836059570312e-6,1.6391277313232422e-6,1.996755599975586e-6,2.086162567138672e-6,3.5762786865234375e-6,5.155801773071289e-6,4.76837158203125e-6,3.069639205932617e-6,3.844499588012695e-6,5.781650543212891e-6,7.18235969543457e-6,3.1888484954833984e-6,2.7418136596679688e-6,9.238719940185547e-7,1.8477439880371094e-6,0.000033020973205566406,0.09292972087860107,0.0719636082649231,0.09889593720436096,0.15547260642051697,0.251505047082901,0.2850726246833801,0.3553575873374939,0.4043104946613312,0.4015215039253235,0.40320730209350586,0.3725241422653198,0.3269875645637512,0.29682430624961853,0.2847396731376648,0.3032398819923401,0.2895625829696655,0.3051212728023529,0.340623140335083,0.41593170166015625,0.5069067478179932,0.6188533902168274,0.6992378234863281,0.7581591010093689,0.8004283308982849,0.8509024381637573,0.8778330683708191,0.9015811681747437,0.9149425625801086,0.9359888434410095,0.9366413354873657,0.9416401386260986,0.9273400902748108,0.9184092283248901,0.8786525726318359,0.8442001342773438,0.7230715155601501,0.5250580906867981,0.3090362548828125,0.12365010380744934,0.04682222008705139,0.00854027271270752,0.0019938647747039795,0.00031512975692749023,0.00010401010513305664,4.76837158203125e-6,1.2516975402832031e-6,6.854534149169922e-7,1.1920928955078125e-6,1.519918441772461e-6,4.380941390991211e-6,5.692243576049805e-6,8.940696716308594e-6,6.467103958129883e-6,0.000010013580322265625,2.592802047729492e-6,1.430511474609375e-6,2.086162567138672e-7,1.1920928955078125e-7,8.940696716308594e-8,0.0,2.9802322387695312e-8,5.960464477539063e-8,-5.960464477539063e-8,5.960464477539063e-8,-5.960464477539063e-8,5.960464477539063e-8,0.0,5.960464477539063e-8,-5.960464477539063e-8,2.9802322387695312e-8,-5.960464477539063e-8,0.0,-5.960464477539063e-8,2.9802322387695312e-8,-5.960464477539063e-8,0.0,-5.960464477539063e-8,0.0,-5.960464477539063e-8,0.0,-5.960464477539063e-8,5.960464477539063e-8,-5.960464477539063e-8,0.0,-5.960464477539063e-8,2.9802322387695312e-8,-5.960464477539063e-8,5.960464477539063e-8,-5.960464477539063e-8,5.960464477539063e-8,2.9802322387695312e-8,-5.960464477539063e-8,-5.960464477539063e-8,5.960464477539063e-8,-5.960464477539063e-8,5.960464477539063e-8,-5.960464477539063e-8,2.9802322387695312e-8,2.9802322387695312e-8,0.0,5.960464477539063e-8,8.940696716308594e-8,0.0,2.9802322387695312e-8,8.940696716308594e-8,8.940696716308594e-8,2.9802322387695312e-8,2.9802322387695312e-8,2.9802322387695312e-8,2.9802322387695312e-8,-5.960464477539063e-8,2.9802322387695312e-8,0.0,5.960464477539063e-8,2.9802322387695312e-8,5.960464477539063e-8,8.940696716308594e-8,1.1920928955078125e-7,5.960464477539063e-8,2.086162567138672e-7,2.086162567138672e-7,4.172325134277344e-7,3.8743019104003906e-7,5.960464477539062e-7,4.76837158203125e-7,9.238719940185547e-7,8.344650268554688e-7,1.2516975402832031e-6,1.4901161193847656e-6,3.3676624298095703e-6,3.606081008911133e-6,5.125999450683594e-6,4.470348358154297e-6,7.450580596923828e-6,5.632638931274414e-6,7.212162017822266e-6,5.3942203521728516e-6,8.493661880493164e-6,5.632638931274414e-6,6.973743438720703e-6,4.351139068603516e-6,5.4836273193359375e-6,4.470348358154297e-6,6.884336471557617e-6,8.732080459594727e-6,0.000018894672393798828,0.000022172927856445312,0.000028759241104125977,0.000028967857360839844,0.000047087669372558594,0.00003981590270996094,0.000047534704208374023,0.00003847479820251465,0.00005969405174255371,0.0000451207160949707,0.000053256750106811523,0.00003457069396972656,0.00004494190216064453,0.000028461217880249023,0.000036090612411499023,0.00002396106719970703,0.000033468008041381836,0.000019073486328125,0.00002110004425048828,0.000010222196578979492,0.00001055002212524414,4.410743713378906e-6,4.798173904418945e-6,2.0563602447509766e-6,2.205371856689453e-6,8.642673492431641e-7,8.642673492431641e-7,2.980232238769531e-7,2.086162567138672e-7,5.960464477539063e-8,1.1920928955078125e-7,2.9802322387695312e-8,8.940696716308594e-8,2.9802322387695312e-8,2.9802322387695312e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,2.9802322387695312e-8,8.940696716308594e-8,5.960464477539063e-8,2.384185791015625e-7,2.682209014892578e-7,6.258487701416016e-7,9.834766387939453e-7,2.473592758178711e-6,2.384185791015625e-6,3.516674041748047e-6,2.771615982055664e-6,5.066394805908203e-6,2.384185791015625e-6,2.1457672119140625e-6,7.450580596923828e-7,8.046627044677734e-7,3.5762786865234375e-7,3.5762786865234375e-7,1.7881393432617188e-7,2.384185791015625e-7,1.1920928955078125e-7,2.086162567138672e-7,1.1920928955078125e-7,1.7881393432617188e-7,5.960464477539063e-8,2.384185791015625e-7,5.960464477539063e-8,2.384185791015625e-7,1.1920928955078125e-7,3.5762786865234375e-7,2.682209014892578e-7,3.5762786865234375e-7,3.5762786865234375e-7,6.258487701416016e-7,5.960464477539062e-7,1.3113021850585938e-6,1.4901161193847656e-6,2.3245811462402344e-6,2.115964889526367e-6,4.857778549194336e-6,3.159046173095703e-6,2.9802322387695312e-6,1.3113021850585938e-6,2.3543834686279297e-6,3.1888484954833984e-6,4.410743713378906e-6,1.3113021850585938e-6,6.258487701416016e-7,2.0563602447509766e-6,0.00003001093864440918,0.13954567909240723,0.13062584400177002,0.14472073316574097,0.2321775257587433,0.31755656003952026,0.38611695170402527,0.4569820761680603,0.49895161390304565,0.543854832649231,0.5252128839492798,0.46933746337890625,0.46051040291786194,0.3985392153263092,0.3850809931755066,0.360737144947052,0.3647264540195465,0.3921055793762207,0.4237675666809082,0.4705711901187897,0.5637977719306946,0.662345290184021,0.7291480302810669,0.755407452583313,0.8017715215682983,0.8456039428710938,0.8808686137199402,0.8892415165901184,0.9165198802947998,0.9284542202949524,0.9437968730926514,0.9356693625450134,0.9382047653198242,0.9131374359130859,0.901625394821167,0.8452013731002808,0.7569738626480103,0.5939628481864929,0.35391539335250854,0.1566401720046997,0.05686497688293457,0.011975854635238647,0.002171158790588379,0.00047647953033447266,0.000060051679611206055,5.602836608886719e-6,1.2516975402832031e-6,6.258487701416016e-7,8.344650268554688e-7,2.175569534301758e-6,4.947185516357422e-6,5.543231964111328e-6,8.493661880493164e-6,8.314847946166992e-6,7.867813110351562e-6,2.771615982055664e-6,1.1622905731201172e-6,3.2782554626464844e-7,1.7881393432617188e-7,8.940696716308594e-8,5.960464477539063e-8,5.960464477539063e-8,2.9802322387695312e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,0.0,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,0.0,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,2.9802322387695312e-8,-5.960464477539063e-8,-5.960464477539063e-8,0.0,0.0,1.1920928955078125e-7,0.0,0.0,0.0,2.9802322387695312e-8,0.0,1.4901161193847656e-7,8.940696716308594e-8,2.9802322387695312e-8,2.9802322387695312e-8,0.0,8.940696716308594e-8,0.0,0.0,1.4901161193847656e-7,1.1920928955078125e-7,1.7881393432617188e-7,1.7881393432617188e-7,3.2782554626464844e-7,4.470348358154297e-7,4.172325134277344e-7,5.066394805908203e-7,6.258487701416016e-7,8.642673492431641e-7,8.940696716308594e-7,1.1622905731201172e-6,1.9669532775878906e-6,3.516674041748047e-6,3.5762786865234375e-6,4.500150680541992e-6,5.155801773071289e-6,6.079673767089844e-6,5.036592483520508e-6,5.304813385009766e-6,5.5730342864990234e-6,6.288290023803711e-6,4.500150680541992e-6,4.559755325317383e-6,4.500150680541992e-6,4.5299530029296875e-6,3.5762786865234375e-6,5.245208740234375e-6,9.5367431640625e-6,0.000016301870346069336,0.000017464160919189453,0.000022709369659423828,0.000028133392333984375,0.00003555417060852051,0.00003078579902648926,0.00003451108932495117,0.000035256147384643555,0.00003975629806518555,0.00003159046173095703,0.000034332275390625,0.00003072619438171387,0.0000311732292175293,0.000021189451217651367,0.000024527311325073242,0.000023812055587768555,0.00002345442771911621,0.00001424551010131836,0.000012427568435668945,9.357929229736328e-6,6.973743438720703e-6,3.4868717193603516e-6,2.682209014892578e-6,2.2351741790771484e-6,1.6391277313232422e-6,8.344650268554688e-7,5.066394805908203e-7,3.5762786865234375e-7,2.086162567138672e-7,1.4901161193847656e-7,8.940696716308594e-8,0.0,0.0,2.9802322387695312e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,0.0,2.9802322387695312e-8,8.940696716308594e-8,8.940696716308594e-8,2.086162567138672e-7,1.7881393432617188e-7,4.172325134277344e-7,8.344650268554688e-7,1.6689300537109375e-6,2.086162567138672e-6,2.8014183044433594e-6,3.9637088775634766e-6,5.751848220825195e-6,3.4570693969726562e-6,2.771615982055664e-6,1.6987323760986328e-6,1.2218952178955078e-6,6.258487701416016e-7,4.76837158203125e-7,3.5762786865234375e-7,3.5762786865234375e-7,2.086162567138672e-7,1.7881393432617188e-7,2.384185791015625e-7,2.086162567138672e-7,1.4901161193847656e-7,1.7881393432617188e-7,2.086162567138672e-7,2.980232238769531e-7,2.086162567138672e-7,2.384185791015625e-7,2.682209014892578e-7,3.5762786865234375e-7,3.2782554626464844e-7,4.76837158203125e-7,7.152557373046875e-7,1.3113021850585938e-6,1.2218952178955078e-6,1.9669532775878906e-6,2.473592758178711e-6,3.6954879760742188e-6,2.294778823852539e-6,1.9073486328125e-6,1.0728836059570312e-6,1.519918441772461e-6,1.6093254089355469e-6,1.817941665649414e-6,8.344650268554688e-7,6.854534149169922e-7,1.5795230865478516e-6,0.00002288818359375,0.22211647033691406,0.17703035473823547,0.21134445071220398,0.3195609450340271,0.4138617217540741,0.5041609406471252,0.5611546635627747,0.6223376393318176,0.6556608080863953,0.6305012702941895,0.6333097219467163,0.6011560559272766,0.5711717009544373,0.5462952256202698,0.5315556526184082,0.5188718438148499,0.5204929113388062,0.5508769750595093,0.6057600975036621,0.6396451592445374,0.7045096158981323,0.7355473637580872,0.7650697231292725,0.8002216815948486,0.8392397165298462,0.8625940084457397,0.8863712549209595,0.9034837484359741,0.9274814128875732,0.9325664043426514,0.9381537437438965,0.9305877685546875,0.9266872406005859,0.8927497863769531,0.8635464906692505,0.7553389072418213,0.582815408706665,0.35776692628860474,0.14495208859443665,0.059842467308044434,0.010408401489257812,0.002018362283706665,0.00039896368980407715,0.00007849931716918945,5.5730342864990234e-6,8.344650268554688e-7,3.8743019104003906e-7,8.344650268554688e-7,1.996755599975586e-6,4.887580871582031e-6,5.543231964111328e-6,0.000010907649993896484,7.510185241699219e-6,7.212162017822266e-6,2.115964889526367e-6,1.2218952178955078e-6,1.7881393432617188e-7,5.960464477539063e-8,0.0,2.9802322387695312e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,2.9802322387695312e-8,-5.960464477539063e-8,2.9802322387695312e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,0.0,0.0,2.9802322387695312e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,8.940696716308594e-8,8.940696716308594e-8,2.9802322387695312e-8,2.9802322387695312e-8,1.4901161193847656e-7,8.940696716308594e-8,1.1920928955078125e-7,1.7881393432617188e-7,2.384185791015625e-7,2.682209014892578e-7,5.364418029785156e-7,4.76837158203125e-7,6.854534149169922e-7,6.854534149169922e-7,8.642673492431641e-7,8.046627044677734e-7,1.2814998626708984e-6,1.996755599975586e-6,3.635883331298828e-6,3.3080577850341797e-6,5.245208740234375e-6,4.082918167114258e-6,5.155801773071289e-6,3.3974647521972656e-6,5.245208740234375e-6,3.635883331298828e-6,4.6193599700927734e-6,2.950429916381836e-6,4.857778549194336e-6,3.159046173095703e-6,3.844499588012695e-6,2.8312206268310547e-6,5.662441253662109e-6,7.957220077514648e-6,0.000014990568161010742,0.000013768672943115234,0.000024020671844482422,0.000021725893020629883,0.000029742717742919922,0.000020623207092285156,0.00003316998481750488,0.000024497509002685547,0.000032275915145874023,0.000019043684005737305,0.00003266334533691406,0.000021904706954956055,0.00002664327621459961,0.000015944242477416992,0.000025391578674316406,0.000017076730728149414,0.0000196993350982666,0.00001004338264465332,0.000012725591659545898,5.781650543212891e-6,5.304813385009766e-6,2.3543834686279297e-6,2.8312206268310547e-6,1.430511474609375e-6,1.3113021850585938e-6,5.960464477539062e-7,6.258487701416016e-7,2.384185791015625e-7,1.1920928955078125e-7,8.940696716308594e-8,5.960464477539063e-8,2.9802322387695312e-8,0.0,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,2.9802322387695312e-8,-5.960464477539063e-8,8.940696716308594e-8,2.9802322387695312e-8,1.1920928955078125e-7,2.384185791015625e-7,3.5762786865234375e-7,5.364418029785156e-7,1.2218952178955078e-6,1.2516975402832031e-6,2.2649765014648438e-6,3.069639205932617e-6,5.692243576049805e-6,3.7550926208496094e-6,4.470348358154297e-6,1.996755599975586e-6,1.7881393432617188e-6,8.344650268554688e-7,8.642673492431641e-7,4.76837158203125e-7,4.172325134277344e-7,2.980232238769531e-7,3.5762786865234375e-7,2.384185791015625e-7,1.7881393432617188e-7,1.1920928955078125e-7,2.384185791015625e-7,2.384185791015625e-7,2.682209014892578e-7,2.086162567138672e-7,4.172325134277344e-7,3.8743019104003906e-7,4.172325134277344e-7,2.682209014892578e-7,6.258487701416016e-7,7.748603820800781e-7,1.341104507446289e-6,1.4007091522216797e-6,2.562999725341797e-6,2.473592758178711e-6,3.725290298461914e-6,2.592802047729492e-6,2.2649765014648438e-6,8.046627044677734e-7,4.470348358154297e-7,6.258487701416016e-7,8.940696716308594e-7,9.834766387939453e-7,5.364418029785156e-7,1.5795230865478516e-6,0.000026524066925048828,0.2690492272377014,0.23374539613723755,0.2772783637046814,0.3874201774597168,0.5170611143112183,0.5906795263290405,0.6430391669273376,0.683351993560791,0.7095354795455933,0.7173857688903809,0.7154237031936646,0.7046529054641724,0.6885541081428528,0.6605277061462402,0.6471198797225952,0.6446011066436768,0.6351953744888306,0.6369520425796509,0.6500352621078491,0.6777881383895874,0.7100507020950317,0.7352752685546875,0.7326513528823853,0.7710402011871338,0.8014236688613892,0.8363780975341797,0.8478684425354004,0.8829807639122009,0.8975520133972168,0.9218704104423523,0.9206435680389404,0.9291135668754578,0.9108035564422607,0.8960484266281128,0.8421001434326172,0.7693557143211365,0.5943018198013306,0.3773496150970459,0.15198421478271484,0.051763176918029785,0.010960757732391357,0.002079993486404419,0.0003387033939361572,0.00003844499588012695,4.798173904418945e-6,7.748603820800781e-7,4.470348358154297e-7,4.76837158203125e-7,2.205371856689453e-6,4.589557647705078e-6,7.68899917602539e-6,6.5267086029052734e-6,7.3015689849853516e-6,5.5730342864990234e-6,1.9073486328125e-6,4.172325134277344e-7,2.682209014892578e-7,0.0,2.9802322387695312e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,2.9802322387695312e-8,2.9802322387695312e-8,-5.960464477539063e-8,2.9802322387695312e-8,0.0,2.9802322387695312e-8,-5.960464477539063e-8,5.960464477539063e-8,2.9802322387695312e-8,2.9802322387695312e-8,0.0,0.0,8.940696716308594e-8,5.960464477539063e-8,8.940696716308594e-8,1.1920928955078125e-7,2.086162567138672e-7,2.384185791015625e-7,1.4901161193847656e-7,3.2782554626464844e-7,5.066394805908203e-7,6.854534149169922e-7,4.172325134277344e-7,7.748603820800781e-7,8.642673492431641e-7,1.0132789611816406e-6,8.642673492431641e-7,2.294778823852539e-6,3.129243850708008e-6,3.7848949432373047e-6,2.5331974029541016e-6,3.6656856536865234e-6,3.4570693969726562e-6,3.0100345611572266e-6,2.086162567138672e-6,3.129243850708008e-6,3.129243850708008e-6,3.0100345611572266e-6,2.086162567138672e-6,3.129243850708008e-6,3.129243850708008e-6,3.337860107421875e-6,3.069639205932617e-6,7.987022399902344e-6,0.000011861324310302734,0.00001531839370727539,0.00001245737075805664,0.00002086162567138672,0.000022649765014648438,0.00001996755599975586,0.000015139579772949219,0.000023931264877319336,0.000025540590286254883,0.000020951032638549805,0.000017255544662475586,0.000024497509002685547,0.00002467632293701172,0.000019848346710205078,0.000015288591384887695,0.00002008676528930664,0.0000165402889251709,0.000011861324310302734,6.467103958129883e-6,6.318092346191406e-6,4.172325134277344e-6,2.592802047729492e-6,1.3113021850585938e-6,1.4901161193847656e-6,1.0728836059570312e-6,6.258487701416016e-7,2.980232238769531e-7,2.384185791015625e-7,1.7881393432617188e-7,8.940696716308594e-8,0.0,-5.960464477539063e-8,1.1920928955078125e-7,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,8.940696716308594e-8,0.0,8.940696716308594e-8,5.960464477539063e-8,5.960464477539063e-8,1.1920928955078125e-7,2.980232238769531e-7,5.662441253662109e-7,7.748603820800781e-7,8.642673492431641e-7,2.2351741790771484e-6,3.516674041748047e-6,3.546476364135742e-6,2.2351741790771484e-6,2.7120113372802734e-6,1.9073486328125e-6,1.1026859283447266e-6,5.662441253662109e-7,7.748603820800781e-7,5.960464477539062e-7,4.470348358154297e-7,2.086162567138672e-7,3.2782554626464844e-7,2.682209014892578e-7,2.980232238769531e-7,2.086162567138672e-7,3.5762786865234375e-7,4.172325134277344e-7,4.76837158203125e-7,3.5762786865234375e-7,5.364418029785156e-7,5.662441253662109e-7,6.258487701416016e-7,5.662441253662109e-7,1.1622905731201172e-6,1.6093254089355469e-6,1.996755599975586e-6,1.6987323760986328e-6,2.8014183044433594e-6,4.738569259643555e-6,2.771615982055664e-6,9.238719940185547e-7,5.066394805908203e-7,4.76837158203125e-7,5.364418029785156e-7,2.980232238769531e-7,6.854534149169922e-7,4.172325134277344e-7,1.2814998626708984e-6,0.00002580881118774414,0.30379733443260193,0.27852192521095276,0.33864206075668335,0.47415828704833984,0.5958215594291687,0.651519238948822,0.708922266960144,0.7259365916252136,0.7748622298240662,0.7655673027038574,0.7832032442092896,0.758612871170044,0.7656014561653137,0.7202363014221191,0.7178962230682373,0.689329981803894,0.6896311640739441,0.6678293347358704,0.6697675585746765,0.6855328679084778,0.7137622833251953,0.7070331573486328,0.721856415271759,0.741161584854126,0.7717088460922241,0.7897818684577942,0.8204334378242493,0.8482775688171387,0.8807125091552734,0.9007729887962341,0.9110836982727051,0.9109283685684204,0.9109451770782471,0.881020188331604,0.8465151190757751,0.7300692796707153,0.5864098072052002,0.36048150062561035,0.14890268445014954,0.05585300922393799,0.009035348892211914,0.002219080924987793,0.00027620792388916016,0.00009414553642272949,2.950429916381836e-6,8.642673492431641e-7,4.76837158203125e-7,1.0132789611816406e-6,1.7583370208740234e-6,5.4836273193359375e-6,7.361173629760742e-6,0.000010013580322265625,5.27501106262207e-6,6.407499313354492e-6,1.2516975402832031e-6,6.854534149169922e-7,8.940696716308594e-8,8.940696716308594e-8,0.0,0.0,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,0.0,-5.960464477539063e-8,2.9802322387695312e-8,-5.960464477539063e-8,2.9802322387695312e-8,2.9802322387695312e-8,2.9802322387695312e-8,5.960464477539063e-8,8.940696716308594e-8,8.940696716308594e-8,1.1920928955078125e-7,1.1920928955078125e-7,2.086162567138672e-7,2.384185791015625e-7,3.2782554626464844e-7,3.2782554626464844e-7,7.748603820800781e-7,6.258487701416016e-7,8.344650268554688e-7,6.854534149169922e-7,1.2516975402832031e-6,1.1622905731201172e-6,1.6987323760986328e-6,1.7881393432617188e-6,3.725290298461914e-6,3.069639205932617e-6,3.725290298461914e-6,2.205371856689453e-6,2.9206275939941406e-6,1.7285346984863281e-6,2.3245811462402344e-6,1.6391277313232422e-6,2.6226043701171875e-6,1.9371509552001953e-6,2.7120113372802734e-6,1.9073486328125e-6,3.129243850708008e-6,3.039836883544922e-6,4.32133674621582e-6,5.066394805908203e-6,0.000010848045349121094,0.000011622905731201172,0.000014960765838623047,0.000011920928955078125,0.000018328428268432617,0.000013262033462524414,0.000016570091247558594,0.000013053417205810547,0.000019878149032592773,0.0000152587890625,0.000021010637283325195,0.000016480684280395508,0.000024944543838500977,0.000018835067749023438,0.000023365020751953125,0.000013917684555053711,0.000017076730728149414,9.626150131225586e-6,9.864568710327148e-6,3.904104232788086e-6,4.0531158447265625e-6,1.9073486328125e-6,1.996755599975586e-6,8.344650268554688e-7,9.5367431640625e-7,4.172325134277344e-7,4.76837158203125e-7,1.1920928955078125e-7,1.4901161193847656e-7,1.1920928955078125e-7,5.960464477539063e-8,0.0,2.9802322387695312e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,0.0,0.0,0.0,2.9802322387695312e-8,5.960464477539063e-8,0.0,5.960464477539063e-8,1.7881393432617188e-7,3.5762786865234375e-7,3.5762786865234375e-7,7.450580596923828e-7,1.0132789611816406e-6,2.682209014892578e-6,2.384185791015625e-6,3.159046173095703e-6,2.086162567138672e-6,2.771615982055664e-6,1.2516975402832031e-6,1.2814998626708984e-6,6.854534149169922e-7,8.940696716308594e-7,5.066394805908203e-7,6.258487701416016e-7,3.2782554626464844e-7,5.066394805908203e-7,3.8743019104003906e-7,4.76837158203125e-7,3.5762786865234375e-7,6.854534149169922e-7,5.960464477539062e-7,9.238719940185547e-7,6.258487701416016e-7,1.0132789611816406e-6,8.642673492431641e-7,1.5497207641601562e-6,1.2218952178955078e-6,2.4437904357910156e-6,2.3543834686279297e-6,3.4868717193603516e-6,2.8312206268310547e-6,6.5267086029052734e-6,3.4570693969726562e-6,2.4437904357910156e-6,4.470348358154297e-7,5.364418029785156e-7,6.258487701416016e-7,1.0132789611816406e-6,4.172325134277344e-7,2.980232238769531e-7,1.430511474609375e-6,0.000024944543838500977,0.35255151987075806,0.36558735370635986,0.3762369751930237,0.5226686596870422,0.6185407638549805,0.6756223440170288,0.7133427858352661,0.7642816305160522,0.7922953963279724,0.8143083453178406,0.8095237016677856,0.8140146136283875,0.792137861251831,0.7792961001396179,0.7512761354446411,0.7324877977371216,0.710155189037323,0.6862421035766602,0.6634656190872192,0.6735491752624512,0.6875524520874023,0.6906742453575134,0.6850698590278625,0.7071993350982666,0.7384072542190552,0.7530835866928101,0.7675560712814331,0.8102556467056274,0.8556761741638184,0.8846468925476074,0.8900773525238037,0.9021092057228088,0.8909391760826111,0.8781468868255615,0.8231099247932434,0.7245144248008728,0.5855964422225952,0.3642789125442505,0.1453668773174286,0.049357444047927856,0.006933361291885376,0.0014695227146148682,0.0002333223819732666,0.000037550926208496094,3.039836883544922e-6,9.238719940185547e-7,5.364418029785156e-7,7.152557373046875e-7,2.6226043701171875e-6,6.556510925292969e-6,7.12275505065918e-6,8.761882781982422e-6,6.22868537902832e-6,4.4405460357666016e-6,1.3113021850585938e-6,5.364418029785156e-7,5.960464477539063e-8,8.940696716308594e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,8.940696716308594e-8,0.0,5.960464477539063e-8,-5.960464477539063e-8,8.940696716308594e-8,8.940696716308594e-8,8.940696716308594e-8,1.1920928955078125e-7,1.7881393432617188e-7,2.384185791015625e-7,2.384185791015625e-7,2.682209014892578e-7,5.364418029785156e-7,7.748603820800781e-7,6.258487701416016e-7,7.450580596923828e-7,9.834766387939453e-7,1.2814998626708984e-6,1.1920928955078125e-6,1.6391277313232422e-6,2.4139881134033203e-6,3.4570693969726562e-6,2.8908252716064453e-6,2.8014183044433594e-6,2.473592758178711e-6,2.1457672119140625e-6,1.4007091522216797e-6,1.5497207641601562e-6,1.7881393432617188e-6,2.0563602447509766e-6,1.5795230865478516e-6,1.8775463104248047e-6,2.473592758178711e-6,2.9206275939941406e-6,2.562999725341797e-6,3.427267074584961e-6,6.020069122314453e-6,9.834766387939453e-6,9.685754776000977e-6,0.000011324882507324219,0.000013113021850585938,0.00001424551010131836,0.000011354684829711914,0.000011712312698364258,0.000014871358871459961,0.000016033649444580078,0.0000133514404296875,0.000014781951904296875,0.000020802021026611328,0.00002390146255493164,0.000017911195755004883,0.000016868114471435547,0.00001806020736694336,0.000015228986740112305,8.761882781982422e-6,6.496906280517578e-6,4.947185516357422e-6,3.3080577850341797e-6,1.7285346984863281e-6,1.3113021850585938e-6,1.0132789611816406e-6,7.152557373046875e-7,3.5762786865234375e-7,2.682209014892578e-7,2.086162567138672e-7,1.7881393432617188e-7,2.9802322387695312e-8,0.0,2.9802322387695312e-8,0.0,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,0.0,2.9802322387695312e-8,5.960464477539063e-8,0.0,8.940696716308594e-8,1.1920928955078125e-7,0.0,5.960464477539063e-8,1.4901161193847656e-7,2.384185791015625e-7,2.682209014892578e-7,4.172325134277344e-7,9.238719940185547e-7,1.7285346984863281e-6,1.7881393432617188e-6,2.086162567138672e-6,2.4139881134033203e-6,2.2351741790771484e-6,1.3709068298339844e-6,1.0728836059570312e-6,1.0132789611816406e-6,7.748603820800781e-7,5.662441253662109e-7,5.364418029785156e-7,7.450580596923828e-7,6.258487701416016e-7,5.364418029785156e-7,4.76837158203125e-7,8.046627044677734e-7,8.642673492431641e-7,8.344650268554688e-7,8.344650268554688e-7,1.1324882507324219e-6,1.430511474609375e-6,1.2218952178955078e-6,1.519918441772461e-6,2.3245811462402344e-6,3.248453140258789e-6,2.652406692504883e-6,3.4868717193603516e-6,4.678964614868164e-6,6.616115570068359e-6,3.3080577850341797e-6,2.115964889526367e-6,6.854534149169922e-7,6.258487701416016e-7,5.662441253662109e-7,6.258487701416016e-7,4.172325134277344e-7,4.172325134277344e-7,1.1920928955078125e-6,0.000019073486328125,0.3683054447174072,0.3603994846343994,0.3896309733390808,0.5750856399536133,0.6338010430335999,0.6942617893218994,0.7502413988113403,0.780710756778717,0.8291206359863281,0.8309831023216248,0.849398136138916,0.8314540386199951,0.8394126296043396,0.8055515289306641,0.8050506711006165,0.7650856971740723,0.7585914134979248,0.710221529006958,0.6874451637268066,0.671217679977417,0.6840412616729736,0.6641605496406555,0.6712648868560791,0.6783323287963867,0.7048180103302002,0.7206806540489197,0.7574342489242554,0.782546877861023,0.8267042636871338,0.8615384101867676,0.8837597370147705,0.8900671601295471,0.8913770914077759,0.8658738732337952,0.8247420787811279,0.7073521614074707,0.5463030934333801,0.3241315484046936,0.12169134616851807,0.04176211357116699,0.004598498344421387,0.0007218420505523682,0.00013375282287597656,0.0000388026237487793,3.427267074584961e-6,7.450580596923828e-7,3.5762786865234375e-7,7.450580596923828e-7,2.682209014892578e-6,6.22868537902832e-6,6.3478946685791016e-6,9.447336196899414e-6,4.5299530029296875e-6,3.4570693969726562e-6,8.344650268554688e-7,5.960464477539062e-7,8.940696716308594e-8,2.9802322387695312e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,0.0,8.940696716308594e-8,2.9802322387695312e-8,8.940696716308594e-8,1.7881393432617188e-7,1.7881393432617188e-7,2.086162567138672e-7,1.7881393432617188e-7,3.8743019104003906e-7,4.470348358154297e-7,6.854534149169922e-7,5.960464477539062e-7,9.834766387939453e-7,8.642673492431641e-7,1.2814998626708984e-6,1.0728836059570312e-6,2.0265579223632812e-6,2.2351741790771484e-6,3.069639205932617e-6,2.205371856689453e-6,3.039836883544922e-6,1.6391277313232422e-6,1.6391277313232422e-6,8.642673492431641e-7,1.5795230865478516e-6,1.1622905731201172e-6,1.5795230865478516e-6,1.1622905731201172e-6,1.9669532775878906e-6,1.8477439880371094e-6,2.652406692504883e-6,2.115964889526367e-6,4.023313522338867e-6,4.947185516357422e-6,8.195638656616211e-6,7.12275505065918e-6,0.000011861324310302734,9.59634780883789e-6,0.000011831521987915039,7.718801498413086e-6,0.000012099742889404297,0.000010341405868530273,0.000013530254364013672,9.745359420776367e-6,0.000015348196029663086,0.000015437602996826172,0.000020682811737060547,0.000013917684555053711,0.000018864870071411133,0.000012040138244628906,0.000011473894119262695,5.8710575103759766e-6,6.705522537231445e-6,3.0100345611572266e-6,2.384185791015625e-6,1.0728836059570312e-6,1.3113021850585938e-6,5.662441253662109e-7,4.470348358154297e-7,2.682209014892578e-7,2.980232238769531e-7,1.7881393432617188e-7,5.960464477539063e-8,2.9802322387695312e-8,8.940696716308594e-8,0.0,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,2.9802322387695312e-8,2.9802322387695312e-8,0.0,2.9802322387695312e-8,2.9802322387695312e-8,8.940696716308594e-8,2.9802322387695312e-8,1.1920928955078125e-7,1.4901161193847656e-7,2.384185791015625e-7,1.1920928955078125e-7,4.172325134277344e-7,5.662441253662109e-7,1.1026859283447266e-6,1.0132789611816406e-6,2.2351741790771484e-6,1.519918441772461e-6,1.9073486328125e-6,1.0132789611816406e-6,1.3113021850585938e-6,7.748603820800781e-7,8.642673492431641e-7,5.364418029785156e-7,7.748603820800781e-7,5.960464477539062e-7,6.258487701416016e-7,4.76837158203125e-7,7.748603820800781e-7,6.854534149169922e-7,8.344650268554688e-7,7.748603820800781e-7,1.1920928955078125e-6,1.0132789611816406e-6,1.4901161193847656e-6,1.1920928955078125e-6,2.0563602447509766e-6,2.4437904357910156e-6,3.4570693969726562e-6,2.8908252716064453e-6,4.649162292480469e-6,4.380941390991211e-6,6.22868537902832e-6,3.427267074584961e-6,2.7418136596679688e-6,6.258487701416016e-7,3.5762786865234375e-7,3.8743019104003906e-7,4.76837158203125e-7,5.364418029785156e-7,2.682209014892578e-7,9.834766387939453e-7,0.000019937753677368164,0.39244920015335083,0.40908873081207275,0.4410841464996338,0.5441137552261353,0.6240769624710083,0.6915801167488098,0.7400358319282532,0.7987834215164185,0.8237801194190979,0.8530458807945251,0.8466249704360962,0.8613294959068298,0.8395743370056152,0.8417336940765381,0.8067324757575989,0.8044759035110474,0.7604596018791199,0.7298151254653931,0.6795978546142578,0.6685203909873962,0.6558966040611267,0.6432625651359558,0.6301145553588867,0.6343665719032288,0.6637638807296753,0.6705285906791687,0.6978573203086853,0.7258574366569519,0.7848421335220337,0.8322449922561646,0.8551572561264038,0.8775620460510254,0.8722488284111023,0.851523756980896,0.795972466468811,0.7057950496673584,0.5425378680229187,0.3314547538757324,0.10842204093933105,0.024458497762680054,0.002445042133331299,0.0004557669162750244,0.00007677078247070312,0.000012248754501342773,2.682209014892578e-6,6.258487701416016e-7,3.8743019104003906e-7,5.364418029785156e-7,2.4437904357910156e-6,6.467103958129883e-6,8.046627044677734e-6,4.6193599700927734e-6,4.500150680541992e-6,2.8908252716064453e-6,7.152557373046875e-7,1.4901161193847656e-7,5.960464477539063e-8,2.9802322387695312e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,2.9802322387695312e-8,5.960464477539063e-8,5.960464477539063e-8,8.940696716308594e-8,1.4901161193847656e-7,2.384185791015625e-7,2.980232238769531e-7,2.086162567138672e-7,4.76837158203125e-7,6.258487701416016e-7,6.556510925292969e-7,4.76837158203125e-7,9.238719940185547e-7,1.2218952178955078e-6,1.3113021850585938e-6,1.0728836059570312e-6,2.086162567138672e-6,2.6226043701171875e-6,2.175569534301758e-6,1.1920928955078125e-6,1.4603137969970703e-6,1.1920928955078125e-6,8.046627044677734e-7,6.556510925292969e-7,1.0728836059570312e-6,1.1920928955078125e-6,1.2516975402832031e-6,9.834766387939453e-7,1.7881393432617188e-6,2.115964889526367e-6,2.294778823852539e-6,2.115964889526367e-6,4.798173904418945e-6,6.705522537231445e-6,7.152557373046875e-6,5.841255187988281e-6,9.000301361083984e-6,8.821487426757812e-6,7.68899917602539e-6,5.781650543212891e-6,9.238719940185547e-6,9.566545486450195e-6,9.804964065551758e-6,7.957220077514648e-6,0.000013560056686401367,0.000014573335647583008,0.000012487173080444336,8.13603401184082e-6,9.924173355102539e-6,7.539987564086914e-6,4.649162292480469e-6,2.384185791015625e-6,2.473592758178711e-6,1.6093254089355469e-6,8.940696716308594e-7,4.470348358154297e-7,5.066394805908203e-7,3.5762786865234375e-7,2.384185791015625e-7,1.4901161193847656e-7,1.1920928955078125e-7,5.960464477539063e-8,8.940696716308594e-8,0.0,2.9802322387695312e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,2.9802322387695312e-8,5.960464477539063e-8,-5.960464477539063e-8,0.0,0.0,2.9802322387695312e-8,-5.960464477539063e-8,2.9802322387695312e-8,0.0,5.960464477539063e-8,-5.960464477539063e-8,8.940696716308594e-8,8.940696716308594e-8,1.1920928955078125e-7,2.9802322387695312e-8,8.940696716308594e-8,5.960464477539063e-8,8.940696716308594e-8,5.960464477539063e-8,1.1920928955078125e-7,2.086162567138672e-7,2.980232238769531e-7,2.682209014892578e-7,5.960464477539062e-7,9.834766387939453e-7,1.0728836059570312e-6,9.834766387939453e-7,1.4603137969970703e-6,1.7285346984863281e-6,1.0132789611816406e-6,6.854534149169922e-7,9.238719940185547e-7,8.344650268554688e-7,6.556510925292969e-7,4.76837158203125e-7,7.748603820800781e-7,8.344650268554688e-7,7.450580596923828e-7,5.662441253662109e-7,9.238719940185547e-7,1.0132789611816406e-6,9.5367431640625e-7,8.344650268554688e-7,1.2814998626708984e-6,1.6689300537109375e-6,1.6093254089355469e-6,1.4901161193847656e-6,2.771615982055664e-6,3.5762786865234375e-6,3.4570693969726562e-6,2.682209014892578e-6,4.380941390991211e-6,6.377696990966797e-6,3.3080577850341797e-6,1.0728836059570312e-6,6.556510925292969e-7,3.5762786865234375e-7,3.5762786865234375e-7,1.4901161193847656e-7,3.8743019104003906e-7,2.086162567138672e-7,6.854534149169922e-7,0.000017523765563964844,0.3512496054172516,0.33214497566223145,0.367798388004303,0.4778047800064087,0.6060784459114075,0.6645104289054871,0.7394967079162598,0.7831450700759888,0.8323031663894653,0.843482255935669,0.8580510020256042,0.848390519618988,0.854012668132782,0.8306121826171875,0.824628472328186,0.7933495044708252,0.7813743948936462,0.7354517579078674,0.69806969165802,0.6676637530326843,0.6621381044387817,0.620506227016449,0.6023803353309631,0.6106275320053101,0.6267505288124084,0.6436735391616821,0.6720900535583496,0.69842129945755,0.7504712343215942,0.8004558086395264,0.8399925231933594,0.8594464659690857,0.8639500737190247,0.8379387855529785,0.7896037101745605,0.672376811504364,0.5266550183296204,0.2949671149253845,0.09840160608291626,0.025598585605621338,0.0015425682067871094,0.0003370046615600586,0.00004392862319946289,0.000023126602172851562,1.4603137969970703e-6,6.258487701416016e-7,4.470348358154297e-7,1.0132789611816406e-6,2.205371856689453e-6,7.241964340209961e-6,6.884336471557617e-6,6.884336471557617e-6,2.7120113372802734e-6,2.5331974029541016e-6,4.76837158203125e-7,2.682209014892578e-7,2.9802322387695312e-8,2.9802322387695312e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,2.9802322387695312e-8,2.9802322387695312e-8,5.960464477539063e-8,8.940696716308594e-8,1.1920928955078125e-7,1.1920928955078125e-7,2.682209014892578e-7,2.086162567138672e-7,4.172325134277344e-7,3.5762786865234375e-7,6.258487701416016e-7,5.364418029785156e-7,7.450580596923828e-7,6.854534149169922e-7,1.3113021850585938e-6,1.1026859283447266e-6,1.7583370208740234e-6,1.3709068298339844e-6,2.473592758178711e-6,1.4007091522216797e-6,1.6689300537109375e-6,7.152557373046875e-7,8.940696716308594e-7,5.364418029785156e-7,7.748603820800781e-7,5.960464477539062e-7,1.0728836059570312e-6,9.834766387939453e-7,1.4603137969970703e-6,1.1026859283447266e-6,2.0265579223632812e-6,1.9669532775878906e-6,2.950429916381836e-6,2.8908252716064453e-6,6.139278411865234e-6,5.125999450683594e-6,6.9141387939453125e-6,4.947185516357422e-6,6.884336471557617e-6,5.0067901611328125e-6,6.616115570068359e-6,5.0067901611328125e-6,7.62939453125e-6,6.943941116333008e-6,9.268522262573242e-6,7.539987564086914e-6,0.000011831521987915039,7.569789886474609e-6,8.761882781982422e-6,4.4405460357666016e-6,5.543231964111328e-6,2.5033950805664062e-6,2.682209014892578e-6,1.0132789611816406e-6,1.1026859283447266e-6,4.76837158203125e-7,5.066394805908203e-7,2.086162567138672e-7,2.086162567138672e-7,1.7881393432617188e-7,1.7881393432617188e-7,8.940696716308594e-8,5.960464477539063e-8,0.0,0.0,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,2.9802322387695312e-8,-5.960464477539063e-8,5.960464477539063e-8,2.9802322387695312e-8,-5.960464477539063e-8,-5.960464477539063e-8,5.960464477539063e-8,0.0,1.1920928955078125e-7,2.9802322387695312e-8,8.940696716308594e-8,2.9802322387695312e-8,8.940696716308594e-8,-5.960464477539063e-8,1.4901161193847656e-7,1.1920928955078125e-7,1.4901161193847656e-7,8.940696716308594e-8,1.7881393432617188e-7,1.1920928955078125e-7,1.1920928955078125e-7,1.7881393432617188e-7,2.086162567138672e-7,2.086162567138672e-7,3.2782554626464844e-7,2.980232238769531e-7,7.450580596923828e-7,6.258487701416016e-7,1.0728836059570312e-6,8.344650268554688e-7,1.2218952178955078e-6,6.854534149169922e-7,9.238719940185547e-7,5.662441253662109e-7,8.046627044677734e-7,5.364418029785156e-7,7.748603820800781e-7,5.662441253662109e-7,9.238719940185547e-7,6.258487701416016e-7,8.642673492431641e-7,6.854534149169922e-7,1.2516975402832031e-6,9.238719940185547e-7,1.4007091522216797e-6,9.5367431640625e-7,1.9669532775878906e-6,1.8775463104248047e-6,2.8312206268310547e-6,2.3245811462402344e-6,4.678964614868164e-6,3.4868717193603516e-6,4.26173210144043e-6,3.5762786865234375e-6,7.092952728271484e-6,3.0100345611572266e-6,2.205371856689453e-6,4.172325134277344e-7,4.76837158203125e-7,3.5762786865234375e-7,6.854534149169922e-7,2.086162567138672e-7,1.7881393432617188e-7,6.258487701416016e-7,0.000014603137969970703,0.3253704309463501,0.3065422773361206,0.3189428746700287,0.39194077253341675,0.5433591604232788,0.6278730630874634,0.6826643943786621,0.7624799013137817,0.7954614758491516,0.8370975255966187,0.8281478881835938,0.8476621508598328,0.8221800327301025,0.8286559581756592,0.7903952598571777,0.792516827583313,0.7488198280334473,0.7328171730041504,0.6794796586036682,0.6656177639961243,0.6380319595336914,0.603066086769104,0.5713229179382324,0.5675723552703857,0.5915581583976746,0.6008217930793762,0.6235223412513733,0.6446361541748047,0.7025777697563171,0.7628941535949707,0.801110029220581,0.839432954788208,0.83826744556427,0.8349896669387817,0.7758252620697021,0.6585643887519836,0.5213518738746643,0.2831023037433624,0.0845654308795929,0.01657038927078247,0.0010072588920593262,0.00021770596504211426,0.000035762786865234375,0.000011116266250610352,1.8775463104248047e-6,7.748603820800781e-7,5.960464477539062e-7,1.0728836059570312e-6,3.2782554626464844e-6,8.374452590942383e-6,6.586313247680664e-6,5.424022674560547e-6,2.771615982055664e-6,1.4603137969970703e-6,4.470348358154297e-7,1.4901161193847656e-7,2.9802322387695312e-8,2.9802322387695312e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,5.960464477539063e-8,0.0,5.960464477539063e-8,5.960464477539063e-8,1.4901161193847656e-7,1.1920928955078125e-7,2.682209014892578e-7,2.384185791015625e-7,3.2782554626464844e-7,4.76837158203125e-7,6.258487701416016e-7,5.364418029785156e-7,5.960464477539062e-7,8.344650268554688e-7,1.2516975402832031e-6,1.1622905731201172e-6,1.4007091522216797e-6,1.6093254089355469e-6,1.8477439880371094e-6,1.1920928955078125e-6,1.043081283569336e-6,7.748603820800781e-7,6.258487701416016e-7,4.172325134277344e-7,5.364418029785156e-7,6.258487701416016e-7,9.834766387939453e-7,7.748603820800781e-7,9.5367431640625e-7,1.430511474609375e-6,1.7881393432617188e-6,1.817941665649414e-6,2.3245811462402344e-6,3.516674041748047e-6,5.125999450683594e-6,4.589557647705078e-6,5.245208740234375e-6,5.4836273193359375e-6,5.692243576049805e-6,4.4405460357666016e-6,4.589557647705078e-6,5.841255187988281e-6,6.67572021484375e-6,5.8710575103759766e-6,6.556510925292969e-6,8.344650268554688e-6,8.910894393920898e-6,6.020069122314453e-6,5.602836608886719e-6,4.559755325317383e-6,3.546476364135742e-6,1.9073486328125e-6,1.5795230865478516e-6,1.1622905731201172e-6,6.556510925292969e-7,3.5762786865234375e-7,2.980232238769531e-7,2.682209014892578e-7,1.7881393432617188e-7,1.1920928955078125e-7,8.940696716308594e-8,1.4901161193847656e-7,0.0,2.9802322387695312e-8,5.960464477539063e-8,0.0,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,5.960464477539063e-8,2.9802322387695312e-8,5.960464477539063e-8,1.1920928955078125e-7,2.9802322387695312e-8,1.4901161193847656e-7,2.9802322387695312e-8,2.9802322387695312e-8,2.9802322387695312e-8,8.940696716308594e-8,0.0,2.9802322387695312e-8,8.940696716308594e-8,5.960464477539063e-8,5.960464477539063e-8,5.960464477539063e-8,1.1920928955078125e-7,2.384185791015625e-7,1.4901161193847656e-7,1.4901161193847656e-7,2.384185791015625e-7,2.384185791015625e-7,2.682209014892578e-7,3.2782554626464844e-7,5.364418029785156e-7,6.258487701416016e-7,6.258487701416016e-7,7.748603820800781e-7,9.238719940185547e-7,1.0132789611816406e-6,6.854534149169922e-7,6.258487701416016e-7,7.748603820800781e-7,7.748603820800781e-7,5.662441253662109e-7,5.066394805908203e-7,8.940696716308594e-7,8.642673492431641e-7,8.046627044677734e-7,7.748603820800781e-7,1.1026859283447266e-6,1.2516975402832031e-6,1.1622905731201172e-6,1.1026859283447266e-6,1.6391277313232422e-6,2.175569534301758e-6,2.2649765014648438e-6,2.9206275939941406e-6,3.725290298461914e-6,5.304813385009766e-6,4.023313522338867e-6,4.32133674621582e-6,5.245208740234375e-6,5.751848220825195e-6,3.0994415283203125e-6,2.0265579223632812e-6,5.960464477539062e-7,5.066394805908203e-7,4.76837158203125e-7,4.172325134277344e-7,2.384185791015625e-7,2.384185791015625e-7,5.066394805908203e-7,9.924173355102539e-6,0.23924845457077026,0.18611881136894226,0.20418259501457214,0.30319881439208984,0.39404356479644775,0.49108242988586426,0.6018898487091064,0.697411298751831,0.751301109790802,0.7793291211128235,0.7961525321006775,0.7928764224052429,0.7941360473632812,0.7625105381011963,0.7543957233428955,0.726203441619873,0.7281417846679688,0.6820555925369263,0.6774934530258179,0.6458495855331421,0.6247395873069763,0.5696003437042236,0.5429033637046814,0.5065155029296875,0.54129558801651,0.5533570051193237,0.567153811454773,0.5602930784225464,0.6363493204116821,0.6977713108062744,0.7361246347427368,0.7953976392745972,0.8177493810653687,0.8043531179428101,0.7603299021720886,0.6362639665603638,0.4394442141056061,0.2168107032775879,0.0665154755115509,0.013726025819778442,0.0007919669151306152,0.00008502602577209473,0.000013113021850585938,9.119510650634766e-6,1.6391277313232422e-6,7.748603820800781e-7,4.76837158203125e-7,1.2218952178955078e-6,3.2186508178710938e-6,7.271766662597656e-6,4.947185516357422e-6,5.453824996948242e-6,1.4603137969970703e-6,8.940696716308594e-7,2.086162567138672e-7,1.7881393432617188e-7,0.0,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,2.9802322387695312e-8,2.9802322387695312e-8,5.960464477539063e-8,5.960464477539063e-8,1.7881393432617188e-7,1.7881393432617188e-7,2.980232238769531e-7,2.384185791015625e-7,4.172325134277344e-7,3.5762786865234375e-7,5.066394805908203e-7,3.2782554626464844e-7,7.450580596923828e-7,7.152557373046875e-7,1.2218952178955078e-6,1.043081283569336e-6,1.5795230865478516e-6,1.1622905731201172e-6,1.341104507446289e-6,6.556510925292969e-7,9.834766387939453e-7,5.364418029785156e-7,4.76837158203125e-7,3.5762786865234375e-7,5.066394805908203e-7,4.76837158203125e-7,8.344650268554688e-7,6.258487701416016e-7,1.1026859283447266e-6,1.1026859283447266e-6,1.7285346984863281e-6,1.4603137969970703e-6,2.682209014892578e-6,2.9802322387695312e-6,4.857778549194336e-6,3.516674041748047e-6,5.3942203521728516e-6,4.202127456665039e-6,5.155801773071289e-6,3.4868717193603516e-6,5.0961971282958984e-6,4.708766937255859e-6,6.22868537902832e-6,4.559755325317383e-6,6.794929504394531e-6,6.0498714447021484e-6,7.12275505065918e-6,4.112720489501953e-6,5.125999450683594e-6,2.8014183044433594e-6,2.7120113372802734e-6,1.2218952178955078e-6,1.4007091522216797e-6,6.258487701416016e-7,5.364418029785156e-7,2.086162567138672e-7,2.980232238769531e-7,1.7881393432617188e-7,1.7881393432617188e-7,8.940696716308594e-8,5.960464477539063e-8,0.0,8.940696716308594e-8,0.0,0.0,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,2.9802322387695312e-8,0.0,5.960464477539063e-8,0.0,8.940696716308594e-8,2.9802322387695312e-8,8.940696716308594e-8,2.9802322387695312e-8,8.940696716308594e-8,2.9802322387695312e-8,2.9802322387695312e-8,2.9802322387695312e-8,8.940696716308594e-8,5.960464477539063e-8,8.940696716308594e-8,5.960464477539063e-8,1.4901161193847656e-7,1.1920928955078125e-7,1.7881393432617188e-7,2.384185791015625e-7,2.682209014892578e-7,2.682209014892578e-7,3.2782554626464844e-7,2.980232238769531e-7,4.470348358154297e-7,4.76837158203125e-7,5.364418029785156e-7,4.470348358154297e-7,1.0728836059570312e-6,7.748603820800781e-7,9.238719940185547e-7,5.960464477539062e-7,9.238719940185547e-7,6.258487701416016e-7,8.344650268554688e-7,5.066394805908203e-7,8.046627044677734e-7,6.854534149169922e-7,8.642673492431641e-7,6.258487701416016e-7,9.5367431640625e-7,9.834766387939453e-7,1.1324882507324219e-6,9.238719940185547e-7,1.4603137969970703e-6,1.4901161193847656e-6,2.3245811462402344e-6,2.0563602447509766e-6,3.516674041748047e-6,3.874301910400391e-6,5.602836608886719e-6,3.933906555175781e-6,5.841255187988281e-6,3.993511199951172e-6,4.32133674621582e-6,2.652406692504883e-6,2.4437904357910156e-6,4.470348358154297e-7,2.980232238769531e-7,2.980232238769531e-7,3.8743019104003906e-7,2.682209014892578e-7,1.4901161193847656e-7,4.76837158203125e-7,0.00001049041748046875,0.1496943235397339,0.1373998522758484,0.15548500418663025,0.20682305097579956,0.27849262952804565,0.37131375074386597,0.4706194996833801,0.5968983769416809,0.6516499519348145,0.7060270309448242,0.714706540107727,0.7314364314079285,0.7163108587265015,0.707708477973938,0.68230140209198,0.6684602499008179,0.6592188477516174,0.648577094078064,0.634933590888977,0.6208667755126953,0.5974450707435608,0.5449304580688477,0.4755929708480835,0.46265220642089844,0.4658336639404297,0.4775407910346985,0.45414412021636963,0.48100605607032776,0.5298585891723633,0.6226298213005066,0.6552769541740417,0.7206528186798096,0.7557451725006104,0.760663628578186,0.7093648910522461,0.6343741416931152,0.4083770513534546,0.2282969057559967,0.04906848073005676,0.005679279565811157,0.0002828538417816162,0.000038057565689086914,7.18235969543457e-6,2.086162567138672e-6,1.3113021850585938e-6,6.854534149169922e-7,5.662441253662109e-7,8.046627044677734e-7,2.682209014892578e-6,5.21540641784668e-6,4.470348358154297e-6,1.7881393432617188e-6,1.1026859283447266e-6,4.76837158203125e-7,1.7881393432617188e-7,0.0,1.1920928955078125e-7,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,2.9802322387695312e-8,0.0,-5.960464477539063e-8,8.940696716308594e-8,8.940696716308594e-8,1.1920928955078125e-7,1.1920928955078125e-7,2.384185791015625e-7,2.086162567138672e-7,2.682209014892578e-7,2.086162567138672e-7,4.172325134277344e-7,5.364418029785156e-7,5.364418029785156e-7,4.172325134277344e-7,8.940696716308594e-7,1.2516975402832031e-6,1.2218952178955078e-6,7.152557373046875e-7,1.0132789611816406e-6,8.940696716308594e-7,6.258487701416016e-7,3.5762786865234375e-7,4.76837158203125e-7,3.8743019104003906e-7,2.682209014892578e-7,2.384185791015625e-7,4.76837158203125e-7,5.960464477539062e-7,6.854534149169922e-7,5.960464477539062e-7,1.2814998626708984e-6,1.6093254089355469e-6,1.817941665649414e-6,1.4603137969970703e-6,3.159046173095703e-6,3.933906555175781e-6,3.9637088775634766e-6,2.8014183044433594e-6,4.649162292480469e-6,4.380941390991211e-6,3.9637088775634766e-6,2.7120113372802734e-6,4.827976226806641e-6,4.559755325317383e-6,4.649162292480469e-6,3.3080577850341797e-6,5.662441253662109e-6,5.334615707397461e-6,3.904104232788086e-6,2.2649765014648438e-6,2.9206275939941406e-6,2.086162567138672e-6,1.2516975402832031e-6,6.258487701416016e-7,7.450580596923828e-7,4.76837158203125e-7,2.980232238769531e-7,1.1920928955078125e-7,1.7881393432617188e-7,1.1920928955078125e-7,5.960464477539063e-8,2.9802322387695312e-8,8.940696716308594e-8,2.9802322387695312e-8,2.9802322387695312e-8,-5.960464477539063e-8,2.9802322387695312e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,5.960464477539063e-8,5.960464477539063e-8,-5.960464477539063e-8,2.9802322387695312e-8,5.960464477539063e-8,-5.960464477539063e-8,0.0,2.9802322387695312e-8,2.9802322387695312e-8,2.9802322387695312e-8,2.9802322387695312e-8,8.940696716308594e-8,0.0,8.940696716308594e-8,0.0,1.1920928955078125e-7,1.1920928955078125e-7,2.086162567138672e-7,1.7881393432617188e-7,3.2782554626464844e-7,2.980232238769531e-7,4.172325134277344e-7,2.980232238769531e-7,4.470348358154297e-7,5.364418029785156e-7,5.066394805908203e-7,4.76837158203125e-7,8.046627044677734e-7,8.344650268554688e-7,1.0728836059570312e-6,8.344650268554688e-7,1.1920928955078125e-6,1.1622905731201172e-6,1.0132789611816406e-6,6.854534149169922e-7,9.834766387939453e-7,9.238719940185547e-7,9.5367431640625e-7,6.258487701416016e-7,9.238719940185547e-7,9.834766387939453e-7,9.238719940185547e-7,5.662441253662109e-7,1.0728836059570312e-6,1.1026859283447266e-6,1.1622905731201172e-6,8.046627044677734e-7,1.7285346984863281e-6,2.2351741790771484e-6,2.771615982055664e-6,2.086162567138672e-6,4.082918167114258e-6,4.5299530029296875e-6,4.172325134277344e-6,2.682209014892578e-6,3.5762786865234375e-6,3.725290298461914e-6,2.1457672119140625e-6,7.450580596923828e-7,4.76837158203125e-7,3.5762786865234375e-7,3.8743019104003906e-7,1.4901161193847656e-7,2.086162567138672e-7,1.7881393432617188e-7,3.5762786865234375e-7,9.5367431640625e-6,0.08688396215438843,0.05733785033226013,0.061875492334365845,0.10804659128189087,0.1857822835445404,0.2473316490650177,0.30120208859443665,0.3771430552005768,0.47640255093574524,0.5633902549743652,0.6061535477638245,0.5786828398704529,0.5981191396713257,0.5672441124916077,0.5423229336738586,0.49326321482658386,0.5399947166442871,0.5443355441093445,0.5507999658584595,0.537818968296051,0.5268461108207703,0.46233701705932617,0.4133816957473755,0.40933889150619507,0.3749564290046692,0.38570547103881836,0.36193424463272095,0.35161662101745605,0.363677978515625,0.41347771883010864,0.5166734457015991,0.6336570978164673,0.6693324446678162,0.7035253047943115,0.6579371094703674,0.562803328037262,0.3379243016242981,0.16949373483657837,0.039893269538879395,0.00599321722984314,0.00014203786849975586,0.000021845102310180664,2.950429916381836e-6,2.950429916381836e-6,6.258487701416016e-7,6.258487701416016e-7,5.960464477539062e-7,1.043081283569336e-6,1.7881393432617188e-6,4.6193599700927734e-6,2.6226043701171875e-6,2.0265579223632812e-6,5.066394805908203e-7,4.172325134277344e-7,1.4901161193847656e-7,2.9802322387695312e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,2.9802322387695312e-8,2.9802322387695312e-8,8.940696716308594e-8,1.4901161193847656e-7,8.940696716308594e-8,1.1920928955078125e-7,2.384185791015625e-7,2.086162567138672e-7,3.2782554626464844e-7,2.980232238769531e-7,4.172325134277344e-7,2.980232238769531e-7,5.662441253662109e-7,4.76837158203125e-7,7.748603820800781e-7,7.152557373046875e-7,1.4901161193847656e-6,1.0728836059570312e-6,1.3113021850585938e-6,5.960464477539062e-7,6.854534149169922e-7,3.2782554626464844e-7,4.76837158203125e-7,2.086162567138672e-7,2.980232238769531e-7,1.4901161193847656e-7,2.980232238769531e-7,2.980232238769531e-7,5.364418029785156e-7,5.662441253662109e-7,8.642673492431641e-7,8.046627044677734e-7,1.8477439880371094e-6,1.6093254089355469e-6,2.4437904357910156e-6,2.1457672119140625e-6,4.023313522338867e-6,3.3080577850341797e-6,4.32133674621582e-6,3.0100345611572266e-6,4.857778549194336e-6,3.427267074584961e-6,4.291534423828125e-6,2.652406692504883e-6,4.202127456665039e-6,3.516674041748047e-6,4.380941390991211e-6,2.8908252716064453e-6,5.125999450683594e-6,2.8908252716064453e-6,3.159046173095703e-6,1.4603137969970703e-6,1.817941665649414e-6,9.834766387939453e-7,1.0728836059570312e-6,4.172325134277344e-7,4.76837158203125e-7,2.384185791015625e-7,2.682209014892578e-7,1.4901161193847656e-7,5.960464477539063e-8,2.9802322387695312e-8,8.940696716308594e-8,5.960464477539063e-8,0.0,0.0,0.0,-5.960464477539063e-8,0.0,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,5.960464477539063e-8,-5.960464477539063e-8,0.0,5.960464477539063e-8,8.940696716308594e-8,0.0,8.940696716308594e-8,1.1920928955078125e-7,8.940696716308594e-8,0.0,2.9802322387695312e-8,2.9802322387695312e-8,0.0,8.940696716308594e-8,8.940696716308594e-8,8.940696716308594e-8,1.1920928955078125e-7,1.1920928955078125e-7,2.086162567138672e-7,1.7881393432617188e-7,2.682209014892578e-7,2.682209014892578e-7,4.470348358154297e-7,4.172325134277344e-7,6.854534149169922e-7,5.960464477539062e-7,8.046627044677734e-7,7.450580596923828e-7,1.043081283569336e-6,8.344650268554688e-7,1.3113021850585938e-6,1.2218952178955078e-6,1.7285346984863281e-6,1.1622905731201172e-6,1.6689300537109375e-6,1.1026859283447266e-6,1.5497207641601562e-6,9.238719940185547e-7,1.2814998626708984e-6,9.834766387939453e-7,1.3709068298339844e-6,7.450580596923828e-7,1.3113021850585938e-6,9.238719940185547e-7,1.1026859283447266e-6,7.748603820800781e-7,1.3113021850585938e-6,1.1324882507324219e-6,1.430511474609375e-6,1.2516975402832031e-6,2.771615982055664e-6,2.8312206268310547e-6,3.904104232788086e-6,2.9802322387695312e-6,5.21540641784668e-6,3.3974647521972656e-6,3.904104232788086e-6,2.2351741790771484e-6,3.516674041748047e-6,1.4007091522216797e-6,1.1324882507324219e-6,2.980232238769531e-7,3.5762786865234375e-7,3.5762786865234375e-7,6.258487701416016e-7,2.086162567138672e-7,1.1920928955078125e-7,3.8743019104003906e-7,9.834766387939453e-6,0.054306238889694214,0.0417955219745636,0.037494122982025146,0.06368875503540039,0.13947826623916626,0.18121904134750366,0.22504717111587524,0.26170650124549866,0.3397146463394165,0.3871194124221802,0.4017082154750824,0.4436597228050232,0.4250994026660919,0.38928475975990295,0.34334930777549744,0.3878449499607086,0.41752147674560547,0.4353419542312622,0.43661749362945557,0.4637938439846039,0.45523014664649963,0.42666396498680115,0.3751126527786255,0.3723089396953583,0.34231114387512207,0.3545002341270447,0.31325024366378784,0.3026706874370575,0.30639249086380005,0.31278863549232483,0.3679065704345703,0.5148972868919373,0.6111665964126587,0.6582143902778625,0.6090380549430847,0.48768728971481323,0.2889999747276306,0.14403817057609558,0.023896902799606323,0.00228959321975708,0.0000711977481842041,0.000012069940567016602,2.086162567138672e-6,1.2814998626708984e-6,7.748603820800781e-7,5.662441253662109e-7,5.066394805908203e-7,9.238719940185547e-7,1.9371509552001953e-6,3.6954879760742188e-6,2.115964889526367e-6,1.4901161193847656e-6,5.662441253662109e-7,2.086162567138672e-7,8.940696716308594e-8,8.940696716308594e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,2.9802322387695312e-8,8.940696716308594e-8,2.9802322387695312e-8,8.940696716308594e-8,1.1920928955078125e-7,2.384185791015625e-7,1.7881393432617188e-7,2.086162567138672e-7,3.5762786865234375e-7,3.5762786865234375e-7,2.682209014892578e-7,3.5762786865234375e-7,4.172325134277344e-7,5.960464477539062e-7,5.066394805908203e-7,7.450580596923828e-7,1.1026859283447266e-6,1.430511474609375e-6,1.0132789611816406e-6,9.834766387939453e-7,7.450580596923828e-7,5.066394805908203e-7,2.980232238769531e-7,2.980232238769531e-7,2.086162567138672e-7,2.086162567138672e-7,1.4901161193847656e-7,1.7881393432617188e-7,2.682209014892578e-7,4.172325134277344e-7,5.364418029785156e-7,6.854534149169922e-7,1.1026859283447266e-6,1.5497207641601562e-6,1.5497207641601562e-6,1.9371509552001953e-6,2.7418136596679688e-6,3.635883331298828e-6,3.129243850708008e-6,3.4570693969726562e-6,4.112720489501953e-6,4.380941390991211e-6,3.159046173095703e-6,3.3676624298095703e-6,3.516674041748047e-6,3.6954879760742188e-6,2.9206275939941406e-6,3.3080577850341797e-6,3.606081008911133e-6,4.023313522338867e-6,2.592802047729492e-6,2.5331974029541016e-6,2.115964889526367e-6,1.7285346984863281e-6,1.0132789611816406e-6,7.748603820800781e-7,6.258487701416016e-7,3.2782554626464844e-7,2.086162567138672e-7,8.940696716308594e-8,1.7881393432617188e-7,8.940696716308594e-8,2.9802322387695312e-8,2.9802322387695312e-8,8.940696716308594e-8,0.0,2.9802322387695312e-8,2.9802322387695312e-8,0.0,2.9802322387695312e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,2.9802322387695312e-8,0.0,2.9802322387695312e-8,2.9802322387695312e-8,0.0,0.0,0.0,8.940696716308594e-8,2.9802322387695312e-8,2.9802322387695312e-8,2.9802322387695312e-8,2.9802322387695312e-8,8.940696716308594e-8,5.960464477539063e-8,8.940696716308594e-8,1.4901161193847656e-7,1.4901161193847656e-7,2.980232238769531e-7,2.682209014892578e-7,2.980232238769531e-7,5.066394805908203e-7,6.854534149169922e-7,6.854534149169922e-7,6.854534149169922e-7,1.1622905731201172e-6,1.1920928955078125e-6,1.1622905731201172e-6,1.1622905731201172e-6,1.6689300537109375e-6,1.7285346984863281e-6,1.6391277313232422e-6,1.430511474609375e-6,2.086162567138672e-6,2.0265579223632812e-6,1.4007091522216797e-6,1.341104507446289e-6,1.519918441772461e-6,1.5795230865478516e-6,1.2516975402832031e-6,1.3113021850585938e-6,1.4603137969970703e-6,1.6391277313232422e-6,1.1622905731201172e-6,1.1324882507324219e-6,1.1920928955078125e-6,1.4007091522216797e-6,1.1920928955078125e-6,1.3709068298339844e-6,1.9073486328125e-6,2.8908252716064453e-6,2.9802322387695312e-6,3.635883331298828e-6,4.023313522338867e-6,4.6193599700927734e-6,3.3080577850341797e-6,3.0994415283203125e-6,2.950429916381836e-6,2.384185791015625e-6,1.1920928955078125e-6,9.238719940185547e-7,4.470348358154297e-7,5.364418029785156e-7,5.364418029785156e-7,4.76837158203125e-7,2.980232238769531e-7,1.4901161193847656e-7,4.76837158203125e-7,7.212162017822266e-6,0.032459378242492676,0.011783123016357422,0.013226836919784546,0.03515845537185669,0.07952672243118286,0.09843102097511292,0.11951401829719543,0.14141938090324402,0.2228263020515442,0.2499552071094513,0.27841413021087646,0.28805381059646606,0.2944071292877197,0.2647903263568878,0.24841484427452087,0.2588536739349365,0.3216756582260132,0.3454670310020447,0.35826611518859863,0.3662741780281067,0.40705442428588867,0.38853341341018677,0.3539469838142395,0.3272659480571747,0.31058061122894287,0.2983350157737732,0.25571784377098083,0.23801448941230774,0.23724135756492615,0.22184258699417114,0.26523447036743164,0.35155564546585083,0.4678460955619812,0.5316786170005798,0.5415751338005066,0.4144143760204315,0.2108851969242096,0.09400174021720886,0.01856684684753418,0.002192854881286621,0.00006237626075744629,4.1425228118896484e-6,7.748603820800781e-7,9.5367431640625e-7,4.470348358154297e-7,5.960464477539062e-7,4.172325134277344e-7,1.0132789611816406e-6,1.9371509552001953e-6,3.039836883544922e-6,1.4901161193847656e-6,1.3709068298339844e-6,2.682209014892578e-7,1.4901161193847656e-7,0.0,2.9802322387695312e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,5.960464477539063e-8,2.9802322387695312e-8,5.960464477539063e-8,5.960464477539063e-8,1.1920928955078125e-7,1.7881393432617188e-7,2.384185791015625e-7,2.086162567138672e-7,3.5762786865234375e-7,2.682209014892578e-7,3.5762786865234375e-7,2.980232238769531e-7,4.470348358154297e-7,3.5762786865234375e-7,5.662441253662109e-7,5.364418029785156e-7,9.834766387939453e-7,9.5367431640625e-7,1.4007091522216797e-6,7.748603820800781e-7,1.0728836059570312e-6,3.8743019104003906e-7,3.8743019104003906e-7,2.086162567138672e-7,2.384185791015625e-7,1.1920928955078125e-7,1.1920928955078125e-7,1.4901161193847656e-7,1.4901161193847656e-7,1.7881393432617188e-7,3.5762786865234375e-7,3.2782554626464844e-7,6.258487701416016e-7,7.748603820800781e-7,1.2814998626708984e-6,1.0132789611816406e-6,1.9073486328125e-6,1.996755599975586e-6,3.0100345611572266e-6,2.4139881134033203e-6,3.874301910400391e-6,3.129243850708008e-6,4.231929779052734e-6,2.6226043701171875e-6,3.6954879760742188e-6,2.682209014892578e-6,3.2186508178710938e-6,2.294778823852539e-6,3.4868717193603516e-6,2.9802322387695312e-6,3.725290298461914e-6,2.115964889526367e-6,2.6226043701171875e-6,1.6689300537109375e-6,1.6391277313232422e-6,8.940696716308594e-7,9.834766387939453e-7,4.172325134277344e-7,3.8743019104003906e-7,1.4901161193847656e-7,2.086162567138672e-7,8.940696716308594e-8,5.960464477539063e-8,2.9802322387695312e-8,0.0,2.9802322387695312e-8,2.9802322387695312e-8,-5.960464477539063e-8,2.9802322387695312e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,0.0,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,8.940696716308594e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,0.0,2.9802322387695312e-8,2.9802322387695312e-8,-5.960464477539063e-8,8.940696716308594e-8,2.9802322387695312e-8,2.9802322387695312e-8,2.9802322387695312e-8,2.9802322387695312e-8,2.9802322387695312e-8,2.9802322387695312e-8,8.940696716308594e-8,5.960464477539063e-8,0.0,8.940696716308594e-8,8.940696716308594e-8,1.7881393432617188e-7,1.7881393432617188e-7,2.384185791015625e-7,2.682209014892578e-7,4.172325134277344e-7,4.76837158203125e-7,7.450580596923828e-7,7.450580596923828e-7,1.2218952178955078e-6,1.1026859283447266e-6,1.341104507446289e-6,1.1622905731201172e-6,1.7583370208740234e-6,1.6093254089355469e-6,1.8477439880371094e-6,1.6689300537109375e-6,2.3245811462402344e-6,1.9371509552001953e-6,2.086162567138672e-6,1.430511474609375e-6,1.8775463104248047e-6,1.4603137969970703e-6,1.7285346984863281e-6,1.341104507446289e-6,1.8477439880371094e-6,1.519918441772461e-6,1.9073486328125e-6,1.1324882507324219e-6,1.4603137969970703e-6,1.3113021850585938e-6,1.519918441772461e-6,1.2516975402832031e-6,1.8775463104248047e-6,2.086162567138672e-6,3.248453140258789e-6,2.682209014892578e-6,4.112720489501953e-6,3.7848949432373047e-6,4.380941390991211e-6,2.771615982055664e-6,3.725290298461914e-6,1.9073486328125e-6,1.7285346984863281e-6,9.238719940185547e-7,1.2218952178955078e-6,3.5762786865234375e-7,2.384185791015625e-7,4.470348358154297e-7,4.470348358154297e-7,3.5762786865234375e-7,5.960464477539063e-8,4.76837158203125e-7,8.672475814819336e-6,0.01144513487815857,0.007243305444717407,0.009070605039596558,0.014722347259521484,0.04392537474632263,0.07419285178184509,0.09821340441703796,0.11112728714942932,0.1472300887107849,0.21191543340682983,0.22329702973365784,0.2425118386745453,0.2394380271434784,0.23115521669387817,0.22053641080856323,0.23562189936637878,0.27028894424438477,0.3049547076225281,0.30401504039764404,0.343616247177124,0.3627508878707886,0.36177366971969604,0.3121125400066376,0.3012166917324066,0.28967779874801636,0.27016937732696533,0.2349698841571808,0.2108716070652008,0.20964300632476807,0.20155489444732666,0.25681787729263306,0.31160223484039307,0.35790422558784485,0.4344584345817566,0.4355333149433136,0.3551870882511139,0.18429142236709595,0.09234854578971863,0.011325865983963013,0.0006935596466064453,0.000023752450942993164,2.205371856689453e-6,5.066394805908203e-7,2.980232238769531e-7,5.364418029785156e-7,4.76837158203125e-7,5.364418029785156e-7,6.556510925292969e-7,1.817941665649414e-6,2.682209014892578e-6,1.4901161193847656e-6,5.066394805908203e-7,3.2782554626464844e-7,1.7881393432617188e-7,2.9802322387695312e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,2.9802322387695312e-8,5.960464477539063e-8,0.0,5.960464477539063e-8,1.1920928955078125e-7,1.4901161193847656e-7,1.4901161193847656e-7,3.2782554626464844e-7,2.682209014892578e-7,1.4901161193847656e-7,3.5762786865234375e-7,3.5762786865234375e-7,3.2782554626464844e-7,2.086162567138672e-7,4.172325134277344e-7,5.960464477539062e-7,6.854534149169922e-7,5.066394805908203e-7,1.0132789611816406e-6,1.2218952178955078e-6,7.152557373046875e-7,3.5762786865234375e-7,3.5762786865234375e-7,2.682209014892578e-7,1.1920928955078125e-7,8.940696716308594e-8,5.960464477539063e-8,1.1920928955078125e-7,1.4901161193847656e-7,8.940696716308594e-8,2.086162567138672e-7,2.384185791015625e-7,2.980232238769531e-7,2.384185791015625e-7,5.364418029785156e-7,7.748603820800781e-7,8.344650268554688e-7,6.854534149169922e-7,1.4901161193847656e-6,1.9371509552001953e-6,1.996755599975586e-6,1.5795230865478516e-6,2.7418136596679688e-6,3.129243850708008e-6,2.592802047729492e-6,1.6689300537109375e-6,2.6226043701171875e-6,2.592802047729492e-6,2.3245811462402344e-6,1.6987323760986328e-6,2.950429916381836e-6,3.159046173095703e-6,2.294778823852539e-6,1.4603137969970703e-6,1.996755599975586e-6,1.5497207641601562e-6,1.0132789611816406e-6,5.364418029785156e-7,5.662441253662109e-7,3.5762786865234375e-7,2.384185791015625e-7,8.940696716308594e-8,1.4901161193847656e-7,1.4901161193847656e-7,2.9802322387695312e-8,2.9802322387695312e-8,2.9802322387695312e-8,2.9802322387695312e-8,8.940696716308594e-8,-5.960464477539063e-8,8.940696716308594e-8,0.0,2.9802322387695312e-8,-5.960464477539063e-8,2.9802322387695312e-8,0.0,-5.960464477539063e-8,-5.960464477539063e-8,0.0,2.9802322387695312e-8,2.9802322387695312e-8,0.0,8.940696716308594e-8,2.9802322387695312e-8,8.940696716308594e-8,5.960464477539063e-8,8.940696716308594e-8,-5.960464477539063e-8,2.9802322387695312e-8,5.960464477539063e-8,8.940696716308594e-8,2.9802322387695312e-8,2.9802322387695312e-8,8.940696716308594e-8,8.940696716308594e-8,1.1920928955078125e-7,1.1920928955078125e-7,1.1920928955078125e-7,2.384185791015625e-7,2.980232238769531e-7,3.5762786865234375e-7,3.2782554626464844e-7,6.556510925292969e-7,9.238719940185547e-7,1.2516975402832031e-6,9.238719940185547e-7,1.6391277313232422e-6,1.6391277313232422e-6,1.8477439880371094e-6,1.4603137969970703e-6,2.5033950805664062e-6,2.5331974029541016e-6,2.771615982055664e-6,2.0265579223632812e-6,2.771615982055664e-6,2.592802047729492e-6,2.2649765014648438e-6,1.4007091522216797e-6,1.996755599975586e-6,1.996755599975586e-6,1.996755599975586e-6,1.4007091522216797e-6,2.1457672119140625e-6,2.175569534301758e-6,1.7583370208740234e-6,9.834766387939453e-7,1.6391277313232422e-6,1.6689300537109375e-6,1.6391277313232422e-6,1.1622905731201172e-6,2.473592758178711e-6,3.3080577850341797e-6,3.2782554626464844e-6,2.294778823852539e-6,3.7848949432373047e-6,3.6954879760742188e-6,2.7418136596679688e-6,1.4901161193847656e-6,1.9371509552001953e-6,1.6987323760986328e-6,9.238719940185547e-7,4.76837158203125e-7,4.76837158203125e-7,3.8743019104003906e-7,4.76837158203125e-7,2.980232238769531e-7,3.5762786865234375e-7,1.1920928955078125e-7,4.172325134277344e-7,9.000301361083984e-6,0.005167514085769653,0.0023314058780670166,0.00269240140914917,0.00925937294960022,0.0260830819606781,0.043745726346969604,0.06604984402656555,0.08864033222198486,0.10570105910301208,0.14990198612213135,0.17343011498451233,0.19064536690711975,0.1701749861240387,0.17306265234947205,0.1754685640335083,0.19398155808448792,0.207763671875,0.24151092767715454,0.2472396194934845,0.27501195669174194,0.30340340733528137,0.31170594692230225,0.2740212678909302,0.25986814498901367,0.23916590213775635,0.22403261065483093,0.18886882066726685,0.17024526000022888,0.15054672956466675,0.13963180780410767,0.1761658787727356,0.21869498491287231,0.2853454053401947,0.32768577337265015,0.3237653970718384,0.2635335922241211,0.1163364052772522,0.06284034252166748,0.009360790252685547,0.0011250078678131104,0.00001481175422668457,1.7881393432617188e-6,2.980232238769531e-7,4.76837158203125e-7,2.384185791015625e-7,4.76837158203125e-7,5.662441253662109e-7,1.1622905731201172e-6,1.2814998626708984e-6,2.771615982055664e-6,1.0728836059570312e-6,8.344650268554688e-7,2.086162567138672e-7,1.1920928955078125e-7,2.9802322387695312e-8,0.0,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,0.0,2.9802322387695312e-8,8.940696716308594e-8,2.9802322387695312e-8,5.960464477539063e-8,1.1920928955078125e-7,2.384185791015625e-7,1.4901161193847656e-7,3.2782554626464844e-7,2.384185791015625e-7,3.5762786865234375e-7,2.384185791015625e-7,3.5762786865234375e-7,2.384185791015625e-7,4.470348358154297e-7,2.980232238769531e-7,6.854534149169922e-7,6.258487701416016e-7,9.238719940185547e-7,6.854534149169922e-7,1.043081283569336e-6,4.76837158203125e-7,5.066394805908203e-7,1.7881393432617188e-7,2.384185791015625e-7,1.1920928955078125e-7,1.7881393432617188e-7,5.960464477539063e-8,8.940696716308594e-8,0.0,1.4901161193847656e-7,8.940696716308594e-8,2.086162567138672e-7,1.4901161193847656e-7,2.682209014892578e-7,2.086162567138672e-7,4.76837158203125e-7,4.172325134277344e-7,6.854534149169922e-7,6.854534149169922e-7,1.4901161193847656e-6,1.1026859283447266e-6,1.5497207641601562e-6,1.4603137969970703e-6,2.7418136596679688e-6,1.8477439880371094e-6,2.3543834686279297e-6,1.430511474609375e-6,2.5331974029541016e-6,1.8775463104248047e-6,2.4437904357910156e-6,1.817941665649414e-6,3.248453140258789e-6,2.2649765014648438e-6,2.7120113372802734e-6,1.430511474609375e-6,2.1457672119140625e-6,8.642673492431641e-7,8.344650268554688e-7,3.5762786865234375e-7,3.2782554626464844e-7,1.1920928955078125e-7,1.4901161193847656e-7,5.960464477539063e-8,5.960464477539063e-8,8.940696716308594e-8,8.940696716308594e-8,0.0,0.0,2.9802322387695312e-8,2.9802322387695312e-8,-5.960464477539063e-8,2.9802322387695312e-8,-5.960464477539063e-8,5.960464477539063e-8,-5.960464477539063e-8,0.0,-5.960464477539063e-8,0.0,8.940696716308594e-8,0.0,0.0,8.940696716308594e-8,0.0,0.0,0.0,2.9802322387695312e-8,2.9802322387695312e-8,2.9802322387695312e-8,0.0,8.940696716308594e-8,0.0,2.9802322387695312e-8,0.0,0.0,2.9802322387695312e-8,1.1920928955078125e-7,1.7881393432617188e-7,2.086162567138672e-7,2.086162567138672e-7,3.2782554626464844e-7,4.172325134277344e-7,7.450580596923828e-7,7.450580596923828e-7,1.2516975402832031e-6,1.2516975402832031e-6,1.6391277313232422e-6,1.3709068298339844e-6,2.0265579223632812e-6,1.7881393432617188e-6,2.5331974029541016e-6,2.115964889526367e-6,3.7550926208496094e-6,2.950429916381836e-6,3.933906555175781e-6,2.7418136596679688e-6,4.112720489501953e-6,2.6226043701171875e-6,3.069639205932617e-6,1.996755599975586e-6,3.3676624298095703e-6,2.473592758178711e-6,3.0100345611572266e-6,1.996755599975586e-6,3.3676624298095703e-6,2.086162567138672e-6,2.3543834686279297e-6,1.4603137969970703e-6,2.384185791015625e-6,1.8477439880371094e-6,2.473592758178711e-6,2.0265579223632812e-6,3.874301910400391e-6,3.248453140258789e-6,4.410743713378906e-6,2.562999725341797e-6,3.7848949432373047e-6,1.996755599975586e-6,2.384185791015625e-6,1.1920928955078125e-6,1.6689300537109375e-6,6.854534149169922e-7,8.344650268554688e-7,3.8743019104003906e-7,5.364418029785156e-7,6.854534149169922e-7,9.834766387939453e-7,3.5762786865234375e-7,1.1920928955078125e-7,5.662441253662109e-7,0.000010609626770019531,0.003197014331817627,0.002092599868774414,0.0019668936729431152,0.005275130271911621,0.023141473531723022,0.04020214080810547,0.07318785786628723,0.07504522800445557,0.11141705513000488,0.14357712864875793,0.16566768288612366,0.16604825854301453,0.17135289311408997,0.16744771599769592,0.16852563619613647,0.1740698516368866,0.1875455379486084,0.2242971956729889,0.21397513151168823,0.23437419533729553,0.2702513337135315,0.2893792986869812,0.24548926949501038,0.24473536014556885,0.224656879901886,0.20447084307670593,0.16443678736686707,0.13472867012023926,0.14132270216941833,0.1206381618976593,0.1434764862060547,0.17324614524841309,0.26669201254844666,0.2664107382297516,0.24514758586883545,0.19855058193206787,0.0964532196521759,0.04271981120109558,0.004575878381729126,0.0003801286220550537,9.894371032714844e-6,1.5795230865478516e-6,2.980232238769531e-7,2.682209014892578e-7,4.470348358154297e-7,4.470348358154297e-7,6.258487701416016e-7,1.0728836059570312e-6,1.6093254089355469e-6,2.4139881134033203e-6,1.1622905731201172e-6,6.258487701416016e-7,2.384185791015625e-7,5.960464477539063e-8,0.0,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,0.0,1.1920928955078125e-7,2.9802322387695312e-8,5.960464477539063e-8,1.4901161193847656e-7,2.086162567138672e-7,1.7881393432617188e-7,2.980232238769531e-7,3.8743019104003906e-7,2.086162567138672e-7,2.682209014892578e-7,3.5762786865234375e-7,3.2782554626464844e-7,2.682209014892578e-7,3.5762786865234375e-7,4.76837158203125e-7,6.854534149169922e-7,6.258487701416016e-7,7.152557373046875e-7,8.344650268554688e-7,8.344650268554688e-7,4.470348358154297e-7,3.8743019104003906e-7,2.384185791015625e-7,1.1920928955078125e-7,8.940696716308594e-8,1.4901161193847656e-7,5.960464477539063e-8,8.940696716308594e-8,2.9802322387695312e-8,8.940696716308594e-8,5.960464477539063e-8,8.940696716308594e-8,1.7881393432617188e-7,1.4901161193847656e-7,2.086162567138672e-7,3.2782554626464844e-7,2.682209014892578e-7,4.470348358154297e-7,6.258487701416016e-7,9.238719940185547e-7,7.748603820800781e-7,1.0728836059570312e-6,1.4901161193847656e-6,2.086162567138672e-6,1.4603137969970703e-6,1.6987323760986328e-6,1.8775463104248047e-6,2.205371856689453e-6,1.6987323760986328e-6,2.0563602447509766e-6,2.473592758178711e-6,3.159046173095703e-6,2.384185791015625e-6,2.6226043701171875e-6,2.3245811462402344e-6,2.3245811462402344e-6,1.1622905731201172e-6,8.940696716308594e-7,5.364418029785156e-7,2.980232238769531e-7,1.7881393432617188e-7,5.960464477539063e-8,1.1920928955078125e-7,5.960464477539063e-8,2.9802322387695312e-8,8.940696716308594e-8,1.1920928955078125e-7,0.0,2.9802322387695312e-8,2.9802322387695312e-8,2.9802322387695312e-8,2.9802322387695312e-8,0.0,0.0,5.960464477539063e-8,0.0,0.0,0.0,2.9802322387695312e-8,0.0,2.9802322387695312e-8,2.9802322387695312e-8,2.9802322387695312e-8,0.0,2.9802322387695312e-8,8.940696716308594e-8,8.940696716308594e-8,8.940696716308594e-8,8.940696716308594e-8,5.960464477539063e-8,2.9802322387695312e-8,2.9802322387695312e-8,0.0,0.0,5.960464477539063e-8,1.1920928955078125e-7,1.4901161193847656e-7,1.7881393432617188e-7,3.5762786865234375e-7,4.470348358154297e-7,5.960464477539062e-7,6.854534149169922e-7,1.2516975402832031e-6,1.6391277313232422e-6,1.6987323760986328e-6,1.5497207641601562e-6,2.3245811462402344e-6,2.384185791015625e-6,2.2351741790771484e-6,2.175569534301758e-6,3.635883331298828e-6,4.380941390991211e-6,3.933906555175781e-6,3.844499588012695e-6,4.9173831939697266e-6,5.0961971282958984e-6,3.6954879760742188e-6,3.606081008911133e-6,3.844499588012695e-6,4.4405460357666016e-6,3.3080577850341797e-6,3.516674041748047e-6,3.337860107421875e-6,3.7550926208496094e-6,2.592802047729492e-6,2.384185791015625e-6,2.294778823852539e-6,2.592802047729492e-6,2.1457672119140625e-6,2.4139881134033203e-6,2.8014183044433594e-6,3.635883331298828e-6,3.337860107421875e-6,3.3080577850341797e-6,2.9802322387695312e-6,3.0100345611572266e-6,1.9669532775878906e-6,1.6391277313232422e-6,1.5795230865478516e-6,1.2814998626708984e-6,7.152557373046875e-7,7.152557373046875e-7,6.854534149169922e-7,8.344650268554688e-7,1.0728836059570312e-6,7.748603820800781e-7,4.76837158203125e-7,2.384185791015625e-7,6.258487701416016e-7,9.208917617797852e-6,0.0027971863746643066,0.0006095767021179199,0.0008167922496795654,0.0037051737308502197,0.017500996589660645,0.025568664073944092,0.054812610149383545,0.058448612689971924,0.10554718971252441,0.12471914291381836,0.14812815189361572,0.1341722309589386,0.1575852930545807,0.14070448279380798,0.14519256353378296,0.1347079575061798,0.16896620392799377,0.1735810935497284,0.1830039620399475,0.20379480719566345,0.23786884546279907,0.2443391978740692,0.22138294577598572,0.2196234166622162,0.1859527826309204,0.1667926013469696,0.12829041481018066,0.11423227190971375,0.0901125967502594,0.07791566848754883,0.08183366060256958,0.10225999355316162,0.15922144055366516,0.16873961687088013,0.15220364928245544,0.13371938467025757,0.04856079816818237,0.020014166831970215,0.003541707992553711,0.0004602372646331787,0.000012159347534179688,9.834766387939453e-7,1.7881393432617188e-7,2.980232238769531e-7,2.980232238769531e-7,4.470348358154297e-7,6.258487701416016e-7,1.4007091522216797e-6,1.519918441772461e-6,2.0265579223632812e-6,8.344650268554688e-7,6.854534149169922e-7,5.960464477539063e-8,8.940696716308594e-8,0.0,0.0,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,0.0,0.0,2.9802322387695312e-8,8.940696716308594e-8,1.1920928955078125e-7,1.4901161193847656e-7,1.1920928955078125e-7,2.384185791015625e-7,2.086162567138672e-7,3.5762786865234375e-7,2.384185791015625e-7,3.2782554626464844e-7,2.980232238769531e-7,3.8743019104003906e-7,2.086162567138672e-7,4.76837158203125e-7,5.066394805908203e-7,7.450580596923828e-7,5.364418029785156e-7,9.238719940185547e-7,5.662441253662109e-7,5.662441253662109e-7,2.980232238769531e-7,3.2782554626464844e-7,1.7881393432617188e-7,5.960464477539063e-8,5.960464477539063e-8,5.960464477539063e-8,8.940696716308594e-8,0.0,0.0,5.960464477539063e-8,5.960464477539063e-8,1.7881393432617188e-7,8.940696716308594e-8,1.7881393432617188e-7,1.1920928955078125e-7,1.7881393432617188e-7,1.7881393432617188e-7,2.980232238769531e-7,2.980232238769531e-7,5.960464477539062e-7,3.8743019104003906e-7,8.642673492431641e-7,8.642673492431641e-7,1.5497207641601562e-6,1.1026859283447266e-6,1.7583370208740234e-6,1.4007091522216797e-6,1.996755599975586e-6,1.5497207641601562e-6,2.473592758178711e-6,2.3245811462402344e-6,3.516674041748047e-6,2.294778823852539e-6,3.4570693969726562e-6,2.5331974029541016e-6,2.682209014892578e-6,1.1920928955078125e-6,1.2218952178955078e-6,4.172325134277344e-7,3.5762786865234375e-7,1.7881393432617188e-7,1.7881393432617188e-7,8.940696716308594e-8,5.960464477539063e-8,2.9802322387695312e-8,8.940696716308594e-8,2.9802322387695312e-8,5.960464477539063e-8,0.0,0.0,5.960464477539063e-8,2.9802322387695312e-8,2.9802322387695312e-8,2.9802322387695312e-8,5.960464477539063e-8,2.9802322387695312e-8,2.9802322387695312e-8,8.940696716308594e-8,2.9802322387695312e-8,8.940696716308594e-8,0.0,2.9802322387695312e-8,8.940696716308594e-8,8.940696716308594e-8,2.9802322387695312e-8,8.940696716308594e-8,0.0,0.0,0.0,2.9802322387695312e-8,-5.960464477539063e-8,2.9802322387695312e-8,0.0,8.940696716308594e-8,8.940696716308594e-8,1.4901161193847656e-7,1.7881393432617188e-7,3.2782554626464844e-7,4.172325134277344e-7,6.258487701416016e-7,6.854534149169922e-7,1.1622905731201172e-6,1.5497207641601562e-6,2.2351741790771484e-6,1.9371509552001953e-6,2.4437904357910156e-6,2.175569534301758e-6,2.5331974029541016e-6,2.115964889526367e-6,2.9206275939941406e-6,3.0100345611572266e-6,4.0531158447265625e-6,3.516674041748047e-6,5.304813385009766e-6,4.738569259643555e-6,5.900859832763672e-6,3.7848949432373047e-6,4.798173904418945e-6,4.500150680541992e-6,5.662441253662109e-6,3.993511199951172e-6,4.857778549194336e-6,3.635883331298828e-6,4.32133674621582e-6,2.5331974029541016e-6,3.159046173095703e-6,2.2649765014648438e-6,2.8312206268310547e-6,1.9669532775878906e-6,2.8908252716064453e-6,2.562999725341797e-6,3.516674041748047e-6,2.4139881134033203e-6,3.427267074584961e-6,2.205371856689453e-6,2.2649765014648438e-6,1.2218952178955078e-6,1.8775463104248047e-6,1.0728836059570312e-6,1.0132789611816406e-6,6.556510925292969e-7,1.0132789611816406e-6,5.364418029785156e-7,5.364418029785156e-7,9.5367431640625e-7,1.0728836059570312e-6,7.450580596923828e-7,2.384185791015625e-7,6.854534149169922e-7,0.00001233816146850586,0.0012209415435791016,0.0004748106002807617,0.0008350014686584473,0.0017756521701812744,0.013295948505401611,0.026369839906692505,0.054050952196121216,0.05977872014045715,0.09541842341423035,0.12516695261001587,0.15070778131484985,0.15031778812408447,0.1534690260887146,0.14800015091896057,0.14352205395698547,0.1423986256122589,0.159650981426239,0.16595610976219177,0.17584407329559326,0.18514138460159302,0.22582045197486877,0.24064230918884277,0.21304136514663696,0.19469350576400757,0.1807713508605957,0.15661746263504028,0.11011040210723877,0.085237056016922,0.074410080909729,0.06201756000518799,0.06617170572280884,0.0718909502029419,0.10106223821640015,0.11154159903526306,0.096772700548172,0.07321184873580933,0.02592119574546814,0.012122362852096558,0.0014511644840240479,0.00012168288230895996,6.705522537231445e-6,7.450580596923828e-7,2.086162567138672e-7,1.4901161193847656e-7,3.5762786865234375e-7,6.258487701416016e-7,8.642673492431641e-7,8.940696716308594e-7,1.5795230865478516e-6,1.6987323760986328e-6,7.748603820800781e-7,2.086162567138672e-7,5.960464477539063e-8,2.9802322387695312e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,2.9802322387695312e-8,8.940696716308594e-8,5.960464477539063e-8,8.940696716308594e-8,1.1920928955078125e-7,1.4901161193847656e-7,1.4901161193847656e-7,1.1920928955078125e-7,2.980232238769531e-7,2.980232238769531e-7,2.682209014892578e-7,2.384185791015625e-7,2.682209014892578e-7,3.2782554626464844e-7,2.980232238769531e-7,1.7881393432617188e-7,4.76837158203125e-7,5.662441253662109e-7,6.258487701416016e-7,3.2782554626464844e-7,5.066394805908203e-7,4.172325134277344e-7,2.086162567138672e-7,1.1920928955078125e-7,1.1920928955078125e-7,8.940696716308594e-8,8.940696716308594e-8,2.9802322387695312e-8,8.940696716308594e-8,1.1920928955078125e-7,0.0,5.960464477539063e-8,8.940696716308594e-8,5.960464477539063e-8,8.940696716308594e-8,1.1920928955078125e-7,8.940696716308594e-8,1.1920928955078125e-7,1.1920928955078125e-7,8.940696716308594e-8,2.086162567138672e-7,2.980232238769531e-7,2.980232238769531e-7,2.682209014892578e-7,6.854534149169922e-7,9.238719940185547e-7,9.834766387939453e-7,6.854534149169922e-7,1.2814998626708984e-6,1.430511474609375e-6,1.5497207641601562e-6,1.1622905731201172e-6,2.592802047729492e-6,3.248453140258789e-6,2.9206275939941406e-6,2.0563602447509766e-6,3.3080577850341797e-6,2.8312206268310547e-6,1.6987323760986328e-6,7.450580596923828e-7,6.854534149169922e-7,4.172325134277344e-7,2.384185791015625e-7,1.1920928955078125e-7,1.7881393432617188e-7,8.940696716308594e-8,5.960464477539063e-8,2.9802322387695312e-8,2.9802322387695312e-8,2.9802322387695312e-8,2.9802322387695312e-8,5.960464477539063e-8,2.9802322387695312e-8,0.0,0.0,-5.960464477539063e-8,1.1920928955078125e-7,2.9802322387695312e-8,2.9802322387695312e-8,0.0,-5.960464477539063e-8,0.0,2.9802322387695312e-8,0.0,5.960464477539063e-8,1.4901161193847656e-7,8.940696716308594e-8,8.940696716308594e-8,0.0,2.9802322387695312e-8,2.9802322387695312e-8,0.0,0.0,2.9802322387695312e-8,5.960464477539063e-8,2.9802322387695312e-8,1.1920928955078125e-7,1.7881393432617188e-7,2.980232238769531e-7,2.980232238769531e-7,6.258487701416016e-7,9.834766387939453e-7,1.2516975402832031e-6,1.2218952178955078e-6,2.5033950805664062e-6,2.9802322387695312e-6,3.2186508178710938e-6,2.0563602447509766e-6,3.1888484954833984e-6,2.7120113372802734e-6,2.86102294921875e-6,1.8775463104248047e-6,3.4570693969726562e-6,3.546476364135742e-6,3.904104232788086e-6,3.129243850708008e-6,4.947185516357422e-6,5.4836273193359375e-6,4.5299530029296875e-6,3.069639205932617e-6,4.947185516357422e-6,5.304813385009766e-6,4.589557647705078e-6,2.682209014892578e-6,3.844499588012695e-6,3.516674041748047e-6,2.562999725341797e-6,1.5795230865478516e-6,2.4139881134033203e-6,2.4139881134033203e-6,2.205371856689453e-6,1.5497207641601562e-6,2.6226043701171875e-6,2.7120113372802734e-6,2.2649765014648438e-6,1.4007091522216797e-6,1.8477439880371094e-6,1.519918441772461e-6,1.1622905731201172e-6,6.854534149169922e-7,9.834766387939453e-7,9.834766387939453e-7,6.854534149169922e-7,4.172325134277344e-7,7.748603820800781e-7,1.0728836059570312e-6,1.3113021850585938e-6,6.854534149169922e-7,6.854534149169922e-7,3.5762786865234375e-7,7.450580596923828e-7,0.00001436471939086914,0.0007388889789581299,0.0002849102020263672,0.00036901235580444336,0.00199812650680542,0.010374635457992554,0.020465970039367676,0.05067789554595947,0.06035822629928589,0.09525749087333679,0.1284581422805786,0.1697865128517151,0.17007073760032654,0.15758532285690308,0.1647956669330597,0.1699252724647522,0.16094177961349487,0.17319804430007935,0.18241697549819946,0.17638719081878662,0.19120213389396667,0.19946441054344177,0.21480706334114075,0.1863822340965271,0.1819700300693512,0.1377139389514923,0.11694180965423584,0.07252848148345947,0.06269678473472595,0.03929886221885681,0.03318259119987488,0.0329422652721405,0.04308167099952698,0.05706906318664551,0.06421911716461182,0.04914507269859314,0.036207735538482666,0.011485129594802856,0.006401568651199341,0.0007280707359313965,0.00018787384033203125,4.500150680541992e-6,9.238719940185547e-7,2.086162567138672e-7,4.172325134277344e-7,3.2782554626464844e-7,8.344650268554688e-7,1.1026859283447266e-6,1.7285346984863281e-6,1.1920928955078125e-6,1.8477439880371094e-6,5.364418029785156e-7,3.2782554626464844e-7,2.9802322387695312e-8,0.0,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,2.9802322387695312e-8,0.0,5.960464477539063e-8,1.4901161193847656e-7,1.1920928955078125e-7,5.960464477539063e-8,2.384185791015625e-7,1.7881393432617188e-7,2.086162567138672e-7,1.7881393432617188e-7,2.682209014892578e-7,2.980232238769531e-7,4.172325134277344e-7,2.384185791015625e-7,4.172325134277344e-7,3.5762786865234375e-7,4.76837158203125e-7,3.5762786865234375e-7,6.258487701416016e-7,4.172325134277344e-7,6.258487701416016e-7,2.980232238769531e-7,4.172325134277344e-7,8.940696716308594e-8,1.1920928955078125e-7,1.4901161193847656e-7,8.940696716308594e-8,0.0,5.960464477539063e-8,5.960464477539063e-8,2.9802322387695312e-8,0.0,2.9802322387695312e-8,2.9802322387695312e-8,8.940696716308594e-8,0.0,2.9802322387695312e-8,-5.960464477539063e-8,8.940696716308594e-8,8.940696716308594e-8,5.960464477539063e-8,0.0,1.7881393432617188e-7,5.960464477539063e-8,2.086162567138672e-7,2.980232238769531e-7,7.450580596923828e-7,6.556510925292969e-7,8.344650268554688e-7,6.854534149169922e-7,1.3709068298339844e-6,1.1920928955078125e-6,1.817941665649414e-6,1.7583370208740234e-6,3.7848949432373047e-6,3.159046173095703e-6,4.112720489501953e-6,2.592802047729492e-6,4.023313522338867e-6,1.9073486328125e-6,1.7881393432617188e-6,6.258487701416016e-7,6.258487701416016e-7,2.980232238769531e-7,2.980232238769531e-7,5.960464477539063e-8,1.7881393432617188e-7,8.940696716308594e-8,1.4901161193847656e-7,2.9802322387695312e-8,8.940696716308594e-8,5.960464477539063e-8,2.9802322387695312e-8,2.9802322387695312e-8,8.940696716308594e-8,5.960464477539063e-8,2.9802322387695312e-8,2.9802322387695312e-8,2.9802322387695312e-8,0.0,2.9802322387695312e-8,-5.960464477539063e-8,2.9802322387695312e-8,8.940696716308594e-8,8.940696716308594e-8,2.9802322387695312e-8,0.0,8.940696716308594e-8,5.960464477539063e-8,0.0,8.940696716308594e-8,8.940696716308594e-8,2.9802322387695312e-8,8.940696716308594e-8,8.940696716308594e-8,8.940696716308594e-8,5.960464477539063e-8,1.1920928955078125e-7,2.682209014892578e-7,2.980232238769531e-7,5.662441253662109e-7,6.854534149169922e-7,1.430511474609375e-6,1.6689300537109375e-6,2.592802047729492e-6,2.771615982055664e-6,4.738569259643555e-6,3.9637088775634766e-6,4.500150680541992e-6,2.7120113372802734e-6,3.6656856536865234e-6,2.592802047729492e-6,2.8908252716064453e-6,2.115964889526367e-6,3.516674041748047e-6,3.0100345611572266e-6,3.9637088775634766e-6,3.248453140258789e-6,5.930662155151367e-6,4.4405460357666016e-6,5.364418029785156e-6,4.023313522338867e-6,6.735324859619141e-6,4.32133674621582e-6,4.738569259643555e-6,2.682209014892578e-6,3.7550926208496094e-6,2.0563602447509766e-6,2.652406692504883e-6,1.5795230865478516e-6,2.5033950805664062e-6,1.817941665649414e-6,2.592802047729492e-6,1.6987323760986328e-6,2.682209014892578e-6,1.519918441772461e-6,1.9669532775878906e-6,1.043081283569336e-6,1.2814998626708984e-6,6.258487701416016e-7,9.238719940185547e-7,5.662441253662109e-7,9.5367431640625e-7,4.470348358154297e-7,6.854534149169922e-7,5.066394805908203e-7,1.1920928955078125e-6,1.7583370208740234e-6,2.294778823852539e-6,7.152557373046875e-7,3.2782554626464844e-7,1.1622905731201172e-6,0.000019311904907226562,0.0005426108837127686,0.00036644935607910156,0.0003647804260253906,0.0012454986572265625,0.010226160287857056,0.02243712544441223,0.05807238817214966,0.06383448839187622,0.10475319623947144,0.13722234964370728,0.181919664144516,0.17949974536895752,0.19276222586631775,0.18853148818016052,0.19067558646202087,0.18043917417526245,0.19608396291732788,0.2081766426563263,0.17778542637825012,0.1880229115486145,0.19974422454833984,0.21201086044311523,0.1690548062324524,0.15578338503837585,0.11716225743293762,0.08930006623268127,0.04910722374916077,0.03540375828742981,0.030476927757263184,0.02069723606109619,0.021838665008544922,0.023793041706085205,0.03575444221496582,0.03559368848800659,0.023749977350234985,0.017873674631118774,0.005771011114120483,0.0024712085723876953,0.00031876564025878906,0.00005424022674560547,3.904104232788086e-6,1.1324882507324219e-6,2.980232238769531e-7,3.2782554626464844e-7,6.854534149169922e-7,1.1622905731201172e-6,1.3113021850585938e-6,1.6987323760986328e-6,1.6689300537109375e-6,1.4007091522216797e-6,5.662441253662109e-7,2.682209014892578e-7,5.960464477539063e-8,2.9802322387695312e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,2.9802322387695312e-8,5.960464477539063e-8,-5.960464477539063e-8,0.0,1.1920928955078125e-7,1.1920928955078125e-7,1.1920928955078125e-7,2.086162567138672e-7,2.384185791015625e-7,1.7881393432617188e-7,2.682209014892578e-7,2.682209014892578e-7,2.980232238769531e-7,2.384185791015625e-7,2.980232238769531e-7,3.5762786865234375e-7,4.76837158203125e-7,3.5762786865234375e-7,4.172325134277344e-7,4.76837158203125e-7,5.364418029785156e-7,4.172325134277344e-7,4.172325134277344e-7,3.5762786865234375e-7,2.682209014892578e-7,1.7881393432617188e-7,1.1920928955078125e-7,0.0,5.960464477539063e-8,8.940696716308594e-8,0.0,-5.960464477539063e-8,2.9802322387695312e-8,0.0,2.9802322387695312e-8,2.9802322387695312e-8,2.9802322387695312e-8,0.0,2.9802322387695312e-8,2.9802322387695312e-8,0.0,2.9802322387695312e-8,0.0,0.0,8.940696716308594e-8,5.960464477539063e-8,5.960464477539063e-8,2.980232238769531e-7,4.76837158203125e-7,4.470348358154297e-7,5.364418029785156e-7,7.748603820800781e-7,1.0728836059570312e-6,9.5367431640625e-7,1.430511474609375e-6,2.175569534301758e-6,3.3974647521972656e-6,2.9802322387695312e-6,3.635883331298828e-6,3.993511199951172e-6,4.351139068603516e-6,2.2649765014648438e-6,1.817941665649414e-6,1.1920928955078125e-6,8.344650268554688e-7,4.172325134277344e-7,3.2782554626464844e-7,2.086162567138672e-7,5.960464477539063e-8,1.4901161193847656e-7,5.960464477539063e-8,8.940696716308594e-8,0.0,2.9802322387695312e-8,2.9802322387695312e-8,2.9802322387695312e-8,8.940696716308594e-8,-5.960464477539063e-8,2.9802322387695312e-8,2.9802322387695312e-8,2.9802322387695312e-8,5.960464477539063e-8,2.9802322387695312e-8,0.0,0.0,0.0,8.940696716308594e-8,8.940696716308594e-8,8.940696716308594e-8,8.940696716308594e-8,0.0,1.4901161193847656e-7,0.0,2.9802322387695312e-8,0.0,0.0,5.960464477539063e-8,8.940696716308594e-8,1.1920928955078125e-7,1.7881393432617188e-7,2.980232238769531e-7,3.5762786865234375e-7,4.470348358154297e-7,1.1026859283447266e-6,1.9073486328125e-6,1.996755599975586e-6,2.652406692504883e-6,4.4405460357666016e-6,6.318092346191406e-6,4.887580871582031e-6,4.5299530029296875e-6,4.500150680541992e-6,4.351139068603516e-6,2.8908252716064453e-6,2.562999725341797e-6,2.8908252716064453e-6,3.3080577850341797e-6,2.950429916381836e-6,3.2782554626464844e-6,4.26173210144043e-6,5.602836608886719e-6,4.380941390991211e-6,5.364418029785156e-6,5.602836608886719e-6,6.67572021484375e-6,4.5299530029296875e-6,4.291534423828125e-6,3.2782554626464844e-6,2.9802322387695312e-6,1.9073486328125e-6,1.996755599975586e-6,1.996755599975586e-6,2.115964889526367e-6,1.817941665649414e-6,1.996755599975586e-6,2.1457672119140625e-6,2.0563602447509766e-6,1.519918441772461e-6,1.341104507446289e-6,1.1622905731201172e-6,9.834766387939453e-7,6.556510925292969e-7,6.854534149169922e-7,7.748603820800781e-7,7.152557373046875e-7,5.364418029785156e-7,6.854534149169922e-7,9.238719940185547e-7,1.5497207641601562e-6,2.175569534301758e-6,1.6093254089355469e-6,1.0132789611816406e-6,4.76837158203125e-7,1.3709068298339844e-6,0.000017762184143066406,0.0006853640079498291,0.00012305378913879395,0.00014600157737731934,0.0009061098098754883,0.007059037685394287,0.014329701662063599,0.04862436652183533,0.05514451861381531,0.11016273498535156,0.13079869747161865,0.17569378018379211,0.16901007294654846,0.19787776470184326,0.1802145540714264,0.19045662879943848,0.18276366591453552,0.20440641045570374,0.20716845989227295,0.17911124229431152,0.17844846844673157,0.18880632519721985,0.18449750542640686,0.14656254649162292,0.12331289052963257,0.07823359966278076,0.05669531226158142,0.02809479832649231,0.022180616855621338,0.013764560222625732,0.009908437728881836,0.0077859461307525635,0.010254740715026855,0.015512913465499878,0.015371084213256836,0.010216712951660156,0.007409155368804932,0.0018225312232971191,0.0007388591766357422,0.00013688206672668457,0.00004795193672180176,4.112720489501953e-6,9.834766387939453e-7,2.682209014892578e-7,5.364418029785156e-7,7.152557373046875e-7,1.341104507446289e-6,1.4007091522216797e-6,2.2351741790771484e-6,1.3113021850585938e-6,1.1026859283447266e-6,3.2782554626464844e-7,3.2782554626464844e-7,8.940696716308594e-8,2.9802322387695312e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,2.9802322387695312e-8,0.0,0.0,8.940696716308594e-8,0.0,5.960464477539063e-8,1.7881393432617188e-7,1.7881393432617188e-7,2.384185791015625e-7,1.1920928955078125e-7,2.086162567138672e-7,2.384185791015625e-7,2.682209014892578e-7,2.086162567138672e-7,3.5762786865234375e-7,3.5762786865234375e-7,3.8743019104003906e-7,2.980232238769531e-7,4.76837158203125e-7,3.5762786865234375e-7,4.172325134277344e-7,2.384185791015625e-7,4.172325134277344e-7,2.086162567138672e-7,1.4901161193847656e-7,8.940696716308594e-8,1.7881393432617188e-7,2.9802322387695312e-8,8.940696716308594e-8,0.0,2.9802322387695312e-8,5.960464477539063e-8,2.9802322387695312e-8,0.0,8.940696716308594e-8,-5.960464477539063e-8,0.0,-5.960464477539063e-8,0.0,2.9802322387695312e-8,-5.960464477539063e-8,5.960464477539063e-8,2.9802322387695312e-8,2.9802322387695312e-8,5.960464477539063e-8,0.0,1.1920928955078125e-7,1.1920928955078125e-7,2.682209014892578e-7,2.086162567138672e-7,4.76837158203125e-7,5.364418029785156e-7,8.940696716308594e-7,7.450580596923828e-7,1.4007091522216797e-6,1.7285346984863281e-6,3.129243850708008e-6,2.592802047729492e-6,4.380941390991211e-6,3.904104232788086e-6,4.827976226806641e-6,2.5331974029541016e-6,2.8312206268310547e-6,1.4007091522216797e-6,1.1920928955078125e-6,5.364418029785156e-7,5.066394805908203e-7,2.384185791015625e-7,1.4901161193847656e-7,1.4901161193847656e-7,5.960464477539063e-8,8.940696716308594e-8,5.960464477539063e-8,2.9802322387695312e-8,0.0,2.9802322387695312e-8,2.9802322387695312e-8,5.960464477539063e-8,2.9802322387695312e-8,0.0,2.9802322387695312e-8,5.960464477539063e-8,2.9802322387695312e-8,2.9802322387695312e-8,2.9802322387695312e-8,8.940696716308594e-8,0.0,5.960464477539063e-8,8.940696716308594e-8,2.9802322387695312e-8,5.960464477539063e-8,0.0,2.9802322387695312e-8,0.0,8.940696716308594e-8,8.940696716308594e-8,1.4901161193847656e-7,5.960464477539063e-8,1.7881393432617188e-7,1.4901161193847656e-7,2.980232238769531e-7,3.5762786865234375e-7,7.748603820800781e-7,1.0132789611816406e-6,1.8477439880371094e-6,1.817941665649414e-6,3.725290298461914e-6,4.32133674621582e-6,6.765127182006836e-6,5.036592483520508e-6,6.973743438720703e-6,4.827976226806641e-6,4.976987838745117e-6,2.9206275939941406e-6,3.4570693969726562e-6,2.7418136596679688e-6,3.2186508178710938e-6,2.6226043701171875e-6,3.7848949432373047e-6,4.202127456665039e-6,5.900859832763672e-6,4.32133674621582e-6,6.467103958129883e-6,5.751848220825195e-6,7.033348083496094e-6,4.4405460357666016e-6,5.21540641784668e-6,2.771615982055664e-6,2.6226043701171875e-6,1.519918441772461e-6,2.086162567138672e-6,1.6391277313232422e-6,1.9371509552001953e-6,1.3709068298339844e-6,2.3245811462402344e-6,1.4603137969970703e-6,1.6391277313232422e-6,8.642673492431641e-7,1.341104507446289e-6,7.748603820800781e-7,7.748603820800781e-7,4.172325134277344e-7,8.046627044677734e-7,5.066394805908203e-7,6.258487701416016e-7,4.76837158203125e-7,8.344650268554688e-7,6.854534149169922e-7,8.344650268554688e-7,1.8775463104248047e-6,1.7881393432617188e-6,1.2516975402832031e-6,4.470348358154297e-7,1.430511474609375e-6,0.00002327561378479004,0.00031879544258117676,0.00008794665336608887,0.00013050436973571777,0.0003402531147003174,0.004064232110977173,0.013063520193099976,0.03929293155670166,0.04949948191642761,0.09233197569847107,0.12869346141815186,0.15014824271202087,0.15604671835899353,0.16368073225021362,0.16319623589515686,0.16751867532730103,0.17656677961349487,0.19489619135856628,0.20588147640228271,0.18008297681808472,0.17358705401420593,0.18760323524475098,0.1789727509021759,0.13583004474639893,0.09720927476882935,0.06050899624824524,0.039865702390670776,0.018849849700927734,0.011540114879608154,0.007554441690444946,0.005349338054656982,0.004563719034194946,0.005393087863922119,0.007008343935012817,0.007727265357971191,0.0053996741771698,0.0021774768829345703,0.0006546974182128906,0.0002472996711730957,0.00004887580871582031,9.894371032714844e-6,2.950429916381836e-6,8.344650268554688e-7,3.5762786865234375e-7,3.5762786865234375e-7,9.238719940185547e-7,1.6987323760986328e-6,1.996755599975586e-6,1.1622905731201172e-6,1.341104507446289e-6,8.344650268554688e-7,2.980232238769531e-7,5.960464477539063e-8,2.9802322387695312e-8,2.9802322387695312e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,2.9802322387695312e-8,2.9802322387695312e-8,2.9802322387695312e-8,8.940696716308594e-8,1.1920928955078125e-7,1.1920928955078125e-7,1.4901161193847656e-7,1.7881393432617188e-7,2.086162567138672e-7,1.4901161193847656e-7,1.1920928955078125e-7,2.384185791015625e-7,2.980232238769531e-7,2.980232238769531e-7,1.7881393432617188e-7,3.5762786865234375e-7,3.5762786865234375e-7,2.980232238769531e-7,2.384185791015625e-7,2.980232238769531e-7,3.5762786865234375e-7,2.086162567138672e-7,5.960464477539063e-8,1.7881393432617188e-7,1.7881393432617188e-7,0.0,2.9802322387695312e-8,8.940696716308594e-8,2.9802322387695312e-8,0.0,-5.960464477539063e-8,0.0,0.0,2.9802322387695312e-8,-5.960464477539063e-8,2.9802322387695312e-8,0.0,0.0,-5.960464477539063e-8,0.0,0.0,2.9802322387695312e-8,-5.960464477539063e-8,2.9802322387695312e-8,8.940696716308594e-8,0.0,1.1920928955078125e-7,1.1920928955078125e-7,1.4901161193847656e-7,2.086162567138672e-7,1.7881393432617188e-7,2.980232238769531e-7,5.364418029785156e-7,5.662441253662109e-7,5.364418029785156e-7,1.2814998626708984e-6,2.205371856689453e-6,2.592802047729492e-6,2.2351741790771484e-6,4.26173210144043e-6,5.0067901611328125e-6,3.516674041748047e-6,2.086162567138672e-6,2.473592758178711e-6,1.817941665649414e-6,1.0728836059570312e-6,4.76837158203125e-7,4.76837158203125e-7,3.2782554626464844e-7,1.4901161193847656e-7,5.960464477539063e-8,1.1920928955078125e-7,8.940696716308594e-8,5.960464477539063e-8,0.0,1.4901161193847656e-7,1.4901161193847656e-7,8.940696716308594e-8,2.9802322387695312e-8,0.0,8.940696716308594e-8,2.9802322387695312e-8,5.960464477539063e-8,8.940696716308594e-8,8.940696716308594e-8,5.960464477539063e-8,0.0,8.940696716308594e-8,1.1920928955078125e-7,5.960464477539063e-8,0.0,5.960464477539063e-8,8.940696716308594e-8,5.960464477539063e-8,8.940696716308594e-8,8.940696716308594e-8,1.1920928955078125e-7,1.1920928955078125e-7,5.960464477539063e-8,2.980232238769531e-7,3.5762786865234375e-7,5.066394805908203e-7,5.960464477539062e-7,1.3113021850585938e-6,1.9371509552001953e-6,2.3245811462402344e-6,2.4139881134033203e-6,4.827976226806641e-6,6.794929504394531e-6,6.407499313354492e-6,4.947185516357422e-6,6.0498714447021484e-6,5.692243576049805e-6,3.993511199951172e-6,2.3543834686279297e-6,3.4570693969726562e-6,3.3974647521972656e-6,3.248453140258789e-6,2.294778823852539e-6,4.708766937255859e-6,5.930662155151367e-6,5.334615707397461e-6,3.7848949432373047e-6,6.22868537902832e-6,6.705522537231445e-6,4.649162292480469e-6,2.5331974029541016e-6,2.950429916381836e-6,2.4139881134033203e-6,1.6093254089355469e-6,1.0132789611816406e-6,1.6987323760986328e-6,1.7583370208740234e-6,1.519918441772461e-6,1.0132789611816406e-6,1.430511474609375e-6,1.2218952178955078e-6,9.238719940185547e-7,5.364418029785156e-7,7.450580596923828e-7,6.258487701416016e-7,4.76837158203125e-7,3.5762786865234375e-7,6.258487701416016e-7,6.854534149169922e-7,5.662441253662109e-7,4.76837158203125e-7,1.0728836059570312e-6,1.7583370208740234e-6,2.205371856689453e-6,1.1026859283447266e-6,1.1622905731201172e-6,6.258487701416016e-7,1.4007091522216797e-6,0.00002631545066833496,0.00021499395370483398,0.00006470084190368652,0.000057578086853027344,0.00026741623878479004,0.00250813364982605,0.00645136833190918,0.029465973377227783,0.04030609130859375,0.07649901509284973,0.10139286518096924,0.1245417594909668,0.11980187892913818,0.12473350763320923,0.11679288744926453,0.1408255398273468,0.13733485341072083,0.1729821264743805,0.1842271387577057,0.1937047243118286,0.180231511592865,0.17193886637687683,0.1624523401260376,0.11818394064903259,0.08817979693412781,0.04072517156600952,0.02455878257751465,0.010203152894973755,0.007452070713043213,0.003353685140609741,0.0023274123668670654,0.002332329750061035,0.0026694536209106445,0.0036405622959136963,0.004855334758758545,0.0026716291904449463,0.0013385415077209473,0.00019821524620056152,0.00008940696716308594,0.000015079975128173828,0.000010579824447631836,1.519918441772461e-6,1.0728836059570312e-6,5.662441253662109e-7,8.642673492431641e-7,1.043081283569336e-6,2.3543834686279297e-6,1.7583370208740234e-6,1.8775463104248047e-6,6.258487701416016e-7,6.258487701416016e-7,1.1920928955078125e-7,1.1920928955078125e-7,0.0,2.9802322387695312e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,0.0,2.9802322387695312e-8,8.940696716308594e-8,5.960464477539063e-8,2.086162567138672e-7,1.1920928955078125e-7,1.7881393432617188e-7,5.960464477539063e-8,2.384185791015625e-7,1.1920928955078125e-7,2.384185791015625e-7,1.4901161193847656e-7,2.980232238769531e-7,2.384185791015625e-7,4.172325134277344e-7,2.086162567138672e-7,3.5762786865234375e-7,2.384185791015625e-7,2.980232238769531e-7,1.4901161193847656e-7,2.086162567138672e-7,1.1920928955078125e-7,1.7881393432617188e-7,1.4901161193847656e-7,1.4901161193847656e-7,8.940696716308594e-8,2.9802322387695312e-8,5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,5.960464477539063e-8,-5.960464477539063e-8,2.9802322387695312e-8,-5.960464477539063e-8,2.9802322387695312e-8,-5.960464477539063e-8,0.0,-5.960464477539063e-8,2.9802322387695312e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,5.960464477539063e-8,-5.960464477539063e-8,2.9802322387695312e-8,2.9802322387695312e-8,0.0,8.940696716308594e-8,1.1920928955078125e-7,5.960464477539063e-8,1.1920928955078125e-7,1.7881393432617188e-7,2.980232238769531e-7,2.682209014892578e-7,5.066394805908203e-7,6.258487701416016e-7,1.7285346984863281e-6,1.817941665649414e-6,2.8312206268310547e-6,3.3080577850341797e-6,6.735324859619141e-6,4.380941390991211e-6,5.21540641784668e-6,2.8908252716064453e-6,4.202127456665039e-6,1.8775463104248047e-6,1.7881393432617188e-6,5.662441253662109e-7,6.854534149169922e-7,2.980232238769531e-7,2.980232238769531e-7,1.1920928955078125e-7,8.940696716308594e-8,1.1920928955078125e-7,1.1920928955078125e-7,5.960464477539063e-8,1.1920928955078125e-7,8.940696716308594e-8,1.1920928955078125e-7,8.940696716308594e-8,8.940696716308594e-8,8.940696716308594e-8,8.940696716308594e-8,2.9802322387695312e-8,1.4901161193847656e-7,8.940696716308594e-8,1.1920928955078125e-7,5.960464477539063e-8,2.384185791015625e-7,1.4901161193847656e-7,1.1920928955078125e-7,0.0,8.940696716308594e-8,5.960464477539063e-8,1.1920928955078125e-7,0.0,1.1920928955078125e-7,5.960464477539063e-8,2.384185791015625e-7,2.086162567138672e-7,4.76837158203125e-7,4.76837158203125e-7,7.748603820800781e-7,8.344650268554688e-7,1.817941665649414e-6,1.8477439880371094e-6,2.950429916381836e-6,3.5762786865234375e-6,8.13603401184082e-6,6.318092346191406e-6,8.07642936706543e-6,5.4836273193359375e-6,8.821487426757812e-6,4.947185516357422e-6,5.125999450683594e-6,3.0100345611572266e-6,4.827976226806641e-6,3.6954879760742188e-6,4.678964614868164e-6,3.844499588012695e-6,6.884336471557617e-6,5.543231964111328e-6,7.62939453125e-6,4.887580871582031e-6,7.897615432739258e-6,4.023313522338867e-6,4.708766937255859e-6,2.0563602447509766e-6,2.294778823852539e-6,1.3113021850585938e-6,1.8775463104248047e-6,1.0728836059570312e-6,1.7881393432617188e-6,1.1026859283447266e-6,1.519918441772461e-6,7.748603820800781e-7,1.0132789611816406e-6,5.960464477539062e-7,6.854534149169922e-7,3.5762786865234375e-7,5.960464477539062e-7,3.5762786865234375e-7,5.066394805908203e-7,4.172325134277344e-7,7.152557373046875e-7,4.470348358154297e-7,7.748603820800781e-7,8.940696716308594e-7,1.996755599975586e-6,2.771615982055664e-6,3.4570693969726562e-6,1.1324882507324219e-6,6.258487701416016e-7,2.2649765014648438e-6,0.00003540515899658203,0.00014483928680419922,0.0000883638858795166,0.00005379319190979004,0.0001658797264099121,0.0017140209674835205,0.0053595006465911865,0.02206408977508545,0.029377788305282593,0.06380701065063477,0.0837160050868988,0.09471288323402405,0.0901387631893158,0.09834280610084534,0.09603908658027649,0.11272534728050232,0.12030470371246338,0.15983790159225464,0.17859387397766113,0.17704123258590698,0.1731364130973816,0.18780311942100525,0.16645830869674683,0.10753744840621948,0.0671352744102478,0.032031893730163574,0.015368729829788208,0.007594704627990723,0.004309862852096558,0.00299718976020813,0.001676321029663086,0.0017474889755249023,0.0021272599697113037,0.0030868053436279297,0.0034312903881073,0.0017181634902954102,0.0007024109363555908,0.00010633468627929688,0.00003826618194580078,9.08970832824707e-6,4.5299530029296875e-6,1.8775463104248047e-6,1.0728836059570312e-6,6.258487701416016e-7,8.344650268554688e-7,1.817941665649414e-6,2.7120113372802734e-6,1.9371509552001953e-6,1.519918441772461e-6,7.450580596923828e-7,3.5762786865234375e-7,2.384185791015625e-7,5.960464477539063e-8,5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,2.9802322387695312e-8,8.940696716308594e-8,5.960464477539063e-8,8.940696716308594e-8,1.7881393432617188e-7,1.7881393432617188e-7,1.1920928955078125e-7,1.1920928955078125e-7,2.086162567138672e-7,2.384185791015625e-7,1.7881393432617188e-7,1.7881393432617188e-7,2.086162567138672e-7,3.5762786865234375e-7,3.2782554626464844e-7,2.682209014892578e-7,3.5762786865234375e-7,3.2782554626464844e-7,2.086162567138672e-7,2.086162567138672e-7,2.086162567138672e-7,1.4901161193847656e-7,1.1920928955078125e-7,8.940696716308594e-8,8.940696716308594e-8,8.940696716308594e-8,2.9802322387695312e-8,5.960464477539063e-8,2.9802322387695312e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,2.9802322387695312e-8,2.9802322387695312e-8,2.9802322387695312e-8,2.9802322387695312e-8,8.940696716308594e-8,8.940696716308594e-8,8.940696716308594e-8,8.940696716308594e-8,5.960464477539063e-8,1.4901161193847656e-7,1.4901161193847656e-7,2.682209014892578e-7,5.066394805908203e-7,1.1026859283447266e-6,1.341104507446289e-6,2.1457672119140625e-6,3.635883331298828e-6,6.22868537902832e-6,4.9173831939697266e-6,5.632638931274414e-6,5.841255187988281e-6,5.841255187988281e-6,3.0994415283203125e-6,2.5331974029541016e-6,1.6391277313232422e-6,1.1622905731201172e-6,5.960464477539062e-7,4.172325134277344e-7,3.5762786865234375e-7,2.086162567138672e-7,1.4901161193847656e-7,1.1920928955078125e-7,1.4901161193847656e-7,1.4901161193847656e-7,1.1920928955078125e-7,5.960464477539063e-8,1.7881393432617188e-7,1.7881393432617188e-7,1.1920928955078125e-7,1.1920928955078125e-7,1.1920928955078125e-7,1.7881393432617188e-7,1.7881393432617188e-7,1.7881393432617188e-7,1.4901161193847656e-7,2.086162567138672e-7,1.1920928955078125e-7,5.960464477539063e-8,1.1920928955078125e-7,1.7881393432617188e-7,1.1920928955078125e-7,8.940696716308594e-8,5.960464477539063e-8,1.7881393432617188e-7,1.7881393432617188e-7,1.7881393432617188e-7,3.2782554626464844e-7,4.76837158203125e-7,5.662441253662109e-7,5.066394805908203e-7,1.1622905731201172e-6,1.6093254089355469e-6,1.7881393432617188e-6,2.473592758178711e-6,4.708766937255859e-6,8.07642936706543e-6,6.735324859619141e-6,8.314847946166992e-6,8.374452590942383e-6,9.47713851928711e-6,6.22868537902832e-6,6.0498714447021484e-6,5.245208740234375e-6,5.632638931274414e-6,4.5299530029296875e-6,5.155801773071289e-6,5.692243576049805e-6,6.973743438720703e-6,6.079673767089844e-6,6.645917892456055e-6,6.377696990966797e-6,6.8247318267822266e-6,4.32133674621582e-6,3.725290298461914e-6,2.6226043701171875e-6,1.996755599975586e-6,1.3709068298339844e-6,1.430511474609375e-6,1.519918441772461e-6,1.5795230865478516e-6,1.0728836059570312e-6,1.0132789611816406e-6,9.834766387939453e-7,8.344650268554688e-7,4.76837158203125e-7,4.76837158203125e-7,5.066394805908203e-7,5.066394805908203e-7,4.172325134277344e-7,4.470348358154297e-7,6.258487701416016e-7,7.152557373046875e-7,5.662441253662109e-7,7.748603820800781e-7,1.4901161193847656e-6,2.771615982055664e-6,3.844499588012695e-6,2.682209014892578e-6,1.817941665649414e-6,1.0132789611816406e-6,2.8014183044433594e-6,0.00003358721733093262,0.0002512931823730469,0.000037044286727905273,0.000030517578125,0.00015631318092346191,0.001186668872833252,0.002733588218688965,0.011167347431182861,0.01669687032699585,0.04000505805015564,0.053703516721725464,0.06151828169822693,0.06810185313224792,0.07115212082862854,0.07884156703948975,0.08823615312576294,0.10256308317184448,0.14511027932167053,0.1711496114730835,0.1565350592136383,0.16317793726921082,0.15299540758132935,0.1323663890361786,0.08908775448799133,0.056625038385391235,0.018341094255447388,0.00846680998802185,0.003358185291290283,0.0031760334968566895,0.0017057955265045166,0.0013206303119659424,0.0011173784732818604,0.001812756061553955,0.002172529697418213,0.0022981464862823486,0.000981062650680542,0.00044086575508117676,0.00004598498344421387,0.000011712312698364258,2.8908252716064453e-6,3.2186508178710938e-6,1.1026859283447266e-6,1.0728836059570312e-6,6.258487701416016e-7,1.1026859283447266e-6,1.7285346984863281e-6,2.6226043701171875e-6,1.4603137969970703e-6,1.5795230865478516e-6,3.5762786865234375e-7,2.384185791015625e-7,5.960464477539063e-8,5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,2.9802322387695312e-8,2.9802322387695312e-8,5.960464477539063e-8,8.940696716308594e-8,1.1920928955078125e-7,1.1920928955078125e-7,1.7881393432617188e-7,1.1920928955078125e-7,1.4901161193847656e-7,1.7881393432617188e-7,1.7881393432617188e-7,1.7881393432617188e-7,2.980232238769531e-7,2.086162567138672e-7,2.980232238769531e-7,2.384185791015625e-7,4.172325134277344e-7,2.384185791015625e-7,2.384185791015625e-7,1.7881393432617188e-7,1.7881393432617188e-7,1.1920928955078125e-7,1.1920928955078125e-7,5.960464477539063e-8,1.1920928955078125e-7,2.9802322387695312e-8,8.940696716308594e-8,0.0,2.9802322387695312e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,2.9802322387695312e-8,2.9802322387695312e-8,8.940696716308594e-8,0.0,0.0,0.0,8.940696716308594e-8,8.940696716308594e-8,1.1920928955078125e-7,5.960464477539063e-8,2.980232238769531e-7,2.980232238769531e-7,7.152557373046875e-7,8.344650268554688e-7,1.7285346984863281e-6,2.652406692504883e-6,5.4836273193359375e-6,4.559755325317383e-6,7.450580596923828e-6,5.990266799926758e-6,7.748603820800781e-6,4.112720489501953e-6,4.4405460357666016e-6,2.2649765014648438e-6,1.996755599975586e-6,1.0132789611816406e-6,9.238719940185547e-7,5.066394805908203e-7,4.172325134277344e-7,2.980232238769531e-7,2.682209014892578e-7,2.682209014892578e-7,2.682209014892578e-7,2.682209014892578e-7,2.384185791015625e-7,2.384185791015625e-7,2.682209014892578e-7,1.4901161193847656e-7,2.384185791015625e-7,2.086162567138672e-7,2.086162567138672e-7,2.682209014892578e-7,2.682209014892578e-7,2.086162567138672e-7,2.682209014892578e-7,1.4901161193847656e-7,2.086162567138672e-7,2.086162567138672e-7,1.1920928955078125e-7,1.7881393432617188e-7,8.940696716308594e-8,1.4901161193847656e-7,2.086162567138672e-7,2.384185791015625e-7,2.086162567138672e-7,3.5762786865234375e-7,4.172325134277344e-7,4.76837158203125e-7,7.152557373046875e-7,9.5367431640625e-7,1.3709068298339844e-6,1.4603137969970703e-6,2.682209014892578e-6,4.559755325317383e-6,7.808208465576172e-6,7.12275505065918e-6,9.98377799987793e-6,9.208917617797852e-6,0.000011920928955078125,7.450580596923828e-6,8.374452590942383e-6,6.67572021484375e-6,7.808208465576172e-6,5.751848220825195e-6,7.12275505065918e-6,6.22868537902832e-6,7.778406143188477e-6,5.5730342864990234e-6,7.361173629760742e-6,5.364418029785156e-6,6.079673767089844e-6,3.5762786865234375e-6,4.172325134277344e-6,2.0265579223632812e-6,1.9371509552001953e-6,1.1622905731201172e-6,1.7285346984863281e-6,1.2516975402832031e-6,1.4007091522216797e-6,8.940696716308594e-7,1.2516975402832031e-6,6.854534149169922e-7,6.854534149169922e-7,3.5762786865234375e-7,5.662441253662109e-7,4.172325134277344e-7,4.76837158203125e-7,3.5762786865234375e-7,5.960464477539062e-7,5.960464477539062e-7,6.854534149169922e-7,7.748603820800781e-7,1.341104507446289e-6,1.6093254089355469e-6,2.0563602447509766e-6,4.380941390991211e-6,3.993511199951172e-6,2.8908252716064453e-6,1.3113021850585938e-6,4.0531158447265625e-6,0.000052541494369506836,0.0001926422119140625,0.00004172325134277344,0.00004214048385620117,0.00008532404899597168,0.0006992816925048828,0.002233266830444336,0.006295919418334961,0.010097801685333252,0.02433931827545166,0.03474920988082886,0.03748077154159546,0.046027302742004395,0.05363285541534424,0.05734533071517944,0.06475457549095154,0.08153244853019714,0.11463406682014465,0.12947297096252441,0.12912017107009888,0.12829577922821045,0.1332342028617859,0.11757475137710571,0.06806159019470215,0.030980229377746582,0.010833173990249634,0.004464387893676758,0.0021375715732574463,0.0014128684997558594,0.0012017786502838135,0.0009599924087524414,0.0009607672691345215,0.001109302043914795,0.001552969217300415,0.0014274418354034424,0.0006320476531982422,0.00011906027793884277,0.000018209218978881836,3.7848949432373047e-6,1.3709068298339844e-6,6.258487701416016e-7,8.344650268554688e-7,6.258487701416016e-7,5.662441253662109e-7,6.258487701416016e-7,1.5497207641601562e-6,2.0563602447509766e-6,1.3709068298339844e-6,4.76837158203125e-7,2.980232238769531e-7,1.1920928955078125e-7,8.940696716308594e-8,2.9802322387695312e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,0.0,2.9802322387695312e-8,2.9802322387695312e-8,8.940696716308594e-8,5.960464477539063e-8,5.960464477539063e-8,8.940696716308594e-8,1.4901161193847656e-7,1.4901161193847656e-7,1.1920928955078125e-7,1.7881393432617188e-7,1.4901161193847656e-7,2.384185791015625e-7,1.7881393432617188e-7,2.682209014892578e-7,2.682209014892578e-7,2.980232238769531e-7,2.086162567138672e-7,2.384185791015625e-7,2.384185791015625e-7,1.1920928955078125e-7,5.960464477539063e-8,1.7881393432617188e-7,5.960464477539063e-8,5.960464477539063e-8,2.9802322387695312e-8,8.940696716308594e-8,1.1920928955078125e-7,-5.960464477539063e-8,-5.960464477539063e-8,2.9802322387695312e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,-5.960464477539063e-8,8.940696716308594e-8,2.9802322387695312e-8,5.960464477539063e-8,2.9802322387695312e-8,0.0,8.940696716308594e-8,2.9802322387695312e-8,5.960464477539063e-8,2.9802322387695312e-8,0.0,8.940696716308594e-8,5.960464477539063e-8,2.384185791015625e-7,4.172325134277344e-7,6.854534149169922e-7,6.854534149169922e-7,1.6987323760986328e-6,3.1888484954833984e-6,3.9637088775634766e-6,3.3676624298095703e-6,5.990266799926758e-6,6.735324859619141e-6,4.976987838745117e-6,3.1888484954833984e-6,3.844499588012695e-6,3.069639205932617e-6,1.817941665649414e-6,1.0132789611816406e-6,1.1622905731201172e-6,8.046627044677734e-7,7.450580596923828e-7,3.2782554626464844e-7,6.258487701416016e-7,5.066394805908203e-7,4.76837158203125e-7,2.980232238769531e-7,4.172325134277344e-7,4.172325134277344e-7,3.5762786865234375e-7,2.086162567138672e-7,3.5762786865234375e-7,3.5762786865234375e-7,4.172325134277344e-7,2.086162567138672e-7,3.5762786865234375e-7,3.5762786865234375e-7,2.682209014892578e-7,1.4901161193847656e-7,2.980232238769531e-7,2.384185791015625e-7,2.384185791015625e-7,1.4901161193847656e-7,2.682209014892578e-7,2.980232238769531e-7,3.5762786865234375e-7,1.7881393432617188e-7,5.364418029785156e-7,5.066394805908203e-7,7.748603820800781e-7,5.364418029785156e-7,1.1622905731201172e-6,1.430511474609375e-6,1.9371509552001953e-6,1.6391277313232422e-6,4.857778549194336e-6,7.152557373046875e-6,8.463859558105469e-6,6.139278411865234e-6,0.000010639429092407227,0.0000