CLIP ViT-B/32 Transformer Demonstration with Wallaroo

The CLIP ViT-B/32 Transformer Demonstration with Wallaroo deployment.

This tutorial and the assets can be downloaded as part of the Wallaroo Tutorials repository.

CLIP ViT-B/32 Transformer Demonstration with Wallaroo

The following tutorial demonstrates deploying and performing sample inferences with the Hugging Face CLIP ViT-B/32 Transformer model.

Prerequisites

This tutorial is geared towards the Wallaroo version 2023.2.1 and above. The model clip-vit-base-patch-32.zip must be downloaded and placed into the ./models directory. This is available from the following URL:

https://storage.googleapis.com/wallaroo-public-data/hf-clip-vit-b32/clip-vit-base-patch-32.zip

If performing this tutorial from outside the Wallaroo JupyterHub environment, install the Wallaroo SDK.

Steps

Imports

The first step is to import the libraries used for the example.

import json
import os
import requests

import wallaroo
from wallaroo.pipeline   import Pipeline
from wallaroo.deployment_config import DeploymentConfigBuilder
from wallaroo.framework import Framework
from wallaroo.object import EntityNotFoundError

import pyarrow as pa
import numpy as np
import pandas as pd

from PIL import Image

Connect to the Wallaroo Instance

The first step is to connect to Wallaroo through the Wallaroo client. The Python library is included in the Wallaroo install and available through the Jupyter Hub interface provided with your Wallaroo environment.

This is accomplished using the wallaroo.Client() command, which provides a URL to grant the SDK permission to your specific Wallaroo environment. When displayed, enter the URL into a browser and confirm permissions. Store the connection into a variable that can be referenced later.

If logging into the Wallaroo instance through the internal JupyterHub service, use wl = wallaroo.Client(). For more information on Wallaroo Client settings, see the Client Connection guide.

## blank space to log in 

wl = wallaroo.Client()

## blank space to log in 

wl = wallaroo.Client()

Set Workspace and Pipeline

The next step is to create the Wallaroo workspace and pipeline used for the inference requests.

## convenience functions from the previous notebooks

# return the workspace called <name> through the Wallaroo client.
def get_workspace(name, client):
    workspace = None
    for ws in client.list_workspaces():
        if ws.name() == name:
            workspace= ws
            return workspace
    # if no workspaces were found
    if workspace==None:
        workspace = wl.create_workspace(name)
    return workspace

# get a pipeline by name in the workspace
def get_pipeline(pipeline_name, workspace, client):
    plist = workspace.pipelines()
    pipeline = [p for p in plist if p.name() == pipeline_name]
    if len(pipeline) <= 0:
        pipeline = client.build_pipeline(pipeline_name)
        return pipeline
    return pipeline[0]
# create the workspace and pipeline

workspace_name = 'clip-demo'
pipeline_name = 'clip-demo'

workspace = get_workspace(workspace_name, wl)

wl.set_current_workspace(workspace)
display(wl.get_current_workspace())

pipeline = get_pipeline(pipeline_name, workspace, wl)
pipeline
{'name': 'clip-demo', 'id': 8, 'archived': False, 'created_by': '12ea09d1-0f49-405e-bed1-27eb6d10fde4', 'created_at': '2023-11-20T18:57:53.667873+00:00', 'models': [], 'pipelines': [{'name': 'clip-demo', 'create_time': datetime.datetime(2023, 11, 20, 18, 57, 55, 550690, tzinfo=tzutc()), 'definition': '[]'}]}
nameclip-demo
created2023-11-20 18:57:55.550690+00:00
last_updated2023-11-20 18:57:55.550690+00:00
deployed(none)
archNone
tags
versionsff0e4846-f0f9-4fb0-bc06-d6daa21797bf
steps
publishedFalse

Configure and Upload Model

The 🤗 Hugging Face model is uploaded to Wallaroo by defining the input and output schema, and specifying the model’s framework as wallaroo.framework.Framework.HUGGING_FACE_ZERO_SHOT_IMAGE_CLASSIFICATION.

The data schemas are defined in Apache PyArrow Schema format.

The model is converted to the Wallaroo Containerized runtime after the upload is complete.

input_schema = pa.schema([
    pa.field('inputs', # required, fixed image dimensions
        pa.list_(
            pa.list_(
                pa.list_(
                    pa.int64(),
                    list_size=3
                ),
                list_size=640 
            ),
        list_size=480
    )),
    pa.field('candidate_labels', pa.list_(pa.string(), list_size=4)), # required, equivalent to `options` in the provided demo
]) 

output_schema = pa.schema([
    pa.field('score', pa.list_(pa.float64(), list_size=4)), # has to be same as number of candidate labels
    pa.field('label', pa.list_(pa.string(), list_size=4)), # has to be same as number of candidate labels
])

Upload Model

model = wl.upload_model('clip-vit', './models/clip-vit-base-patch-32.zip', 
                        framework=Framework.HUGGING_FACE_ZERO_SHOT_IMAGE_CLASSIFICATION, 
                        input_schema=input_schema, 
                        output_schema=output_schema)
model
Waiting for model loading - this will take up to 10.0min.
Model is pending loading to a container runtime..
Model is attempting loading to a container runtime............................................................successful

Ready

Nameclip-vit
Version57594cc7-7db1-43c3-b21a-6dcaba846f26
File Nameclip-vit-base-patch-32.zip
SHA4efc24685a14e1682301cc0085b9db931aeb5f3f8247854bedc6863275ed0646
Statusready
Image Pathproxy.replicated.com/proxy/wallaroo/ghcr.io/wallaroolabs/mlflow-deploy:v2023.4.0-4103
ArchitectureNone
Updated At2023-20-Nov 20:51:44

Deploy Pipeline

With the model uploaded and prepared, we add the model as a pipeline step and deploy it. For this example, we will allocate 4 Gi of RAM and 1 CPU to the model’s use through the pipeline deployment configuration.

deployment_config = wallaroo.DeploymentConfigBuilder() \
    .cpus(.25).memory('1Gi') \
    .sidekick_memory(model, '4Gi') \
    .sidekick_cpus(model, 1.0) \
    .build()
pipeline.clear()
pipeline.add_model_step(model)
pipeline.deploy(deployment_config=deployment_config)
nameclip-demo
created2023-11-20 18:57:55.550690+00:00
last_updated2023-11-20 20:51:59.304917+00:00
deployedTrue
archNone
tags
versions7a6168c2-f807-465f-a122-33fe7b5d8a3d, ff0e4846-f0f9-4fb0-bc06-d6daa21797bf
stepsclip-vit
publishedFalse

Run Inference

We verify the pipeline is deployed by checking the status().

The sample images in the ./data directory are converted into numpy arrays, and the candidate labels added as inputs. Both are set as DataFrame arrays where the field inputs are the image values, and candidate_labels the labels.

pipeline.status()
{'status': 'Running',
 'details': [],
 'engines': [{'ip': '10.244.3.140',
   'name': 'engine-bb48bb959-2xctn',
   'status': 'Running',
   'reason': None,
   'details': [],
   'pipeline_statuses': {'pipelines': [{'id': 'clip-demo',
      'status': 'Running'}]},
   'model_statuses': {'models': [{'name': 'clip-vit',
      'version': '57594cc7-7db1-43c3-b21a-6dcaba846f26',
      'sha': '4efc24685a14e1682301cc0085b9db931aeb5f3f8247854bedc6863275ed0646',
      'status': 'Running'}]}}],
 'engine_lbs': [{'ip': '10.244.4.152',
   'name': 'engine-lb-584f54c899-gcbpj',
   'status': 'Running',
   'reason': None,
   'details': []}],
 'sidekicks': [{'ip': '10.244.3.139',
   'name': 'engine-sidekick-clip-vit-11-75c5666b69-tqfbk',
   'status': 'Running',
   'reason': None,
   'details': [],
   'statuses': '\n'}]}
image_paths = [
    "./data/bear-in-tree.jpg",
    "./data/elephant-and-zebras.jpg",
    "./data/horse-and-dogs.jpg",
    "./data/kittens.jpg",
    "./data/remote-monitor.jpg"
]
images = []

for iu in image_paths:
    image = Image.open(iu)
    image = image.resize((640, 480)) # fixed image dimensions
    images.append(np.array(image))

dataframe = pd.DataFrame({"images": images})
input_data = {
        "inputs": images,
        "candidate_labels": [["cat", "dog", "horse", "elephant"]] * 5,
}
dataframe = pd.DataFrame(input_data)
dataframe
inputscandidate_labels
0[[[60, 62, 61], [62, 64, 63], [67, 69, 68], [7...[cat, dog, horse, elephant]
1[[[228, 235, 241], [229, 236, 242], [230, 237,...[cat, dog, horse, elephant]
2[[[177, 177, 177], [177, 177, 177], [177, 177,...[cat, dog, horse, elephant]
3[[[140, 25, 56], [144, 25, 67], [146, 24, 73],...[cat, dog, horse, elephant]
4[[[24, 20, 11], [22, 18, 9], [18, 14, 5], [21,...[cat, dog, horse, elephant]

Inference Outputs

The inference is run, and the labels with their corresponding confidence values for each label are mapped to out.label and out.score for each image.

results = pipeline.infer(dataframe,timeout=600,dataset=["in", "out", "metadata.elapsed", "time", "check_failures"])
pd.set_option('display.max_colwidth', None)
display(results.loc[:, ['out.label', 'out.score']])
out.labelout.score
0[elephant, dog, horse, cat][0.4146825075149536, 0.3483847379684448, 0.1285749077796936, 0.10835790634155273]
1[elephant, horse, dog, cat][0.9981434345245361, 0.001765866531059146, 6.823801231803373e-05, 2.2441448891186155e-05]
2[horse, dog, elephant, cat][0.7596800923347473, 0.217111736536026, 0.020392831414937973, 0.0028152535669505596]
3[cat, dog, elephant, horse][0.9870228171348572, 0.0066468678414821625, 0.0032716328278183937, 0.003058753442019224]
4[dog, horse, cat, elephant][0.5713965892791748, 0.17229433357715607, 0.15523898601531982, 0.1010700911283493]

Undeploy Pipelines

With the tutorial complete, the pipeline is undeployed and the resources returned back to the cluster.

pipeline.undeploy()
nameclip-demo
created2023-11-20 18:57:55.550690+00:00
last_updated2023-11-20 20:51:59.304917+00:00
deployedFalse
archNone
tags
versions7a6168c2-f807-465f-a122-33fe7b5d8a3d, ff0e4846-f0f9-4fb0-bc06-d6daa21797bf
stepsclip-vit
publishedFalse