For details on adding GPU nodepools to a Wallaroo cluster, see Create GPU Nodepools for Kubernetes Clusters.

This tutorial is available on the Wallaroo Tutorials repository.

LLM Summarization GPU Edge Deployment

This tutorial demonstrates how to use the Wallaroo combined with GPU processors to perform inferences with pre-trained computer vision ML models. This demonstration assumes that:

• Wallaroo Version 2023.3 or above instance is installed.
• A nodepools with GPUs part of the Kubernetes cluster. See Create GPU Nodepools for Kubernetes Clusters for more detials.
• The model hf-summarization-bart-large-samsun.zip (1.4 G) has been downloaded to the ./models folder.

Tutorial Goals

For our example, we will perform the following:

• Create a workspace for our work.
• Upload the model.
• Create a pipeline and specify the gpus in the pipeline deployment.
• Perform a sample inference.

Steps

Import Libraries

The first step will be to import our libraries.

import json
import os

import wallaroo
from wallaroo.pipeline   import Pipeline
from wallaroo.deployment_config import DeploymentConfigBuilder
from wallaroo.framework import Framework

import pyarrow as pa
import numpy as np
import pandas as pd

Connect to the Wallaroo Instance

The first step is to connect to Wallaroo through the Wallaroo client. The Python library is included in the Wallaroo install and available through the Jupyter Hub interface provided with your Wallaroo environment.

This is accomplished using the wallaroo.Client() command, which provides a URL to grant the SDK permission to your specific Wallaroo environment. When displayed, enter the URL into a browser and confirm permissions. Store the connection into a variable that can be referenced later.

If logging into the Wallaroo instance through the internal JupyterHub service, use wl = wallaroo.Client(). For more information on Wallaroo Client settings, see the Client Connection guide.

wl = wallaroo.Client()

Configure PyArrow Schema

You can find more info on the available inputs under TextSummarizationInputs or under the official source code from 🤗 Hugging Face.

input_schema = pa.schema([
    pa.field('inputs', pa.string()),
    pa.field('return_text', pa.bool_()),
    pa.field('return_tensors', pa.bool_()),
    pa.field('clean_up_tokenization_spaces', pa.bool_()),
    # pa.field('generate_kwargs', pa.map_(pa.string(), pa.null())), # dictionaries are not currently supported by the engine
])

output_schema = pa.schema([
    pa.field('summary_text', pa.string()),
])

Upload Model

We will now create or connect to our pipeline and upload the model.

import wallaroo.engine_config

model = wl.upload_model('hf-summarization-yns', 
                        'hf-summarisation-bart-large-samsun.zip', 
                        framework=Framework.HUGGING_FACE_SUMMARIZATION, 
                        input_schema=input_schema, 
                        output_schema=output_schema)
model

Waiting for model loading - this will take up to 10.0min.
Model is pending loading to a container runtime..
Model is attempting loading to a container runtime.......................successful
Ready

Deploy Pipeline

With the model uploaded, we can add it is as a step in the pipeline, then deploy it.

For GPU deployment, the pipeline deployment configuration allocated the cpus, ram, gpus, and other settings for the pipeline. For gpus, both the number of GPUs and the nodepool containing the gpus must be specified.

For Wallaroo Native Runtime models (onnx, tensorflow), the method is wallaroo.deployment_config.gpus(int) to allocate the number of gpus to the pipeline. This applies to all Wallaroo Native Runtime models in the pipeline.

For Wallaroo Containerized models (hugging-face, etc), the method is wallaroo.deployment_config.sidekick_gpus(int) to allocate the number of gpus to the model.

The deployment label is set with the wallaroo.deployment_config.deployment_label(string) method.

For more information on allocating resources to a Wallaroo pipeline for deployment, see Wallaroo SDK Essentials Guide: Pipeline Deployment Configuration.

For this example, 1 gpu will be allocated to the pipeline from the nodepool with the deployment label wallaroo.ai/accelerator: a100.

deployment_config = DeploymentConfigBuilder() \
    .cpus(1).memory('1Gi') \
    .sidekick_gpus(model, 1) \
    .sidekick_cpus(model,4) \
    .sidekick_memory(model, '8Gi') \
    .deployment_label('wallaroo.ai/accelerator: a100') \
    .build()

pipeline_name = "hf-summarization-pipeline"

pipeline = wl.build_pipeline(pipeline_name)
pipeline.add_model_step(model)

pipeline.deploy(deployment_config=deployment_config)

Run inference

We can now run a sample inference using the wallaroo.pipeline.infer method and display the results.

input_data = {
        "inputs": ["LinkedIn (/lɪŋktˈɪn/) is a business and employment-focused social media platform that works through websites and mobile apps. It launched on May 5, 2003. It is now owned by Microsoft. The platform is primarily used for professional networking and career development, and allows jobseekers to post their CVs and employers to post jobs. From 2015 most of the company's revenue came from selling access to information about its members to recruiters and sales professionals. Since December 2016, it has been a wholly owned subsidiary of Microsoft. As of March 2023, LinkedIn has more than 900 million registered members from over 200 countries and territories. LinkedIn allows members (both workers and employers) to create profiles and connect with each other in an online social network which may represent real-world professional relationships. Members can invite anyone (whether an existing member or not) to become a connection. LinkedIn can also be used to organize offline events, join groups, write articles, publish job postings, post photos and videos, and more"], # required
        "return_text": [True], # optional: using the defaults, similar to not passing this parameter
        "return_tensors": [False], # optional: using the defaults, similar to not passing this parameter
        "clean_up_tokenization_spaces": [False], # optional: using the defaults, similar to not passing this parameter
}
dataframe = pd.DataFrame(input_data)
dataframe

out = pipeline.infer(dataframe)
out

out["out.summary_text"][0]

'LinkedIn is a business and employment-focused social media platform that works through websites and mobile apps. It launched on May 5, 2003. LinkedIn allows members (both workers and employers) to create profiles and connect with each other in an online social network which may represent real-world professional relationships.'

Model Inferencing with Pipeline Deployment Endpoint

The other option is to use the pipeline’s inference endpoint.

!curl -X POST {pipeline.url()} \
    -H "Content-Type: application/json; format=pandas-records" \
        -d @./data/test_summarization.json

[{"time":1694179270999,"in":{"clean_up_tokenization_spaces":[false],"inputs":["LinkedIn (/lɪŋktˈɪn/) is a business and employment-focused social media platform that works through websites and mobile apps. It launched on May 5, 2003. It is now owned by Microsoft. The platform is primarily used for professional networking and career development, and allows jobseekers to post their CVs and employers to post jobs. From 2015 most of the company's revenue came from selling access to information about its members to recruiters and sales professionals. Since December 2016, it has been a wholly owned subsidiary of Microsoft. As of March 2023, LinkedIn has more than 900 million registered members from over 200 countries and territories. LinkedIn allows members (both workers and employers) to create profiles and connect with each other in an online social network which may represent real-world professional relationships. Members can invite anyone (whether an existing member or not) to become a connection. LinkedIn can also be used to organize offline events, join groups, write articles, publish job postings, post photos and videos, and more"],"return_tensors":[false],"return_text":[true]},"out":{"summary_text":"LinkedIn is a business and employment-focused social media platform that works through websites and mobile apps. It launched on May 5, 2003. LinkedIn allows members (both workers and employers) to create profiles and connect with each other in an online social network which may represent real-world professional relationships."},"check_failures":[],"metadata":{"last_model":"{\"model_name\":\"hf-summarization-yns\",\"model_sha\":\"ee71d066a83708e7ca4a3c07caf33fdc528bb000039b6ca2ef77fa2428dc6268\"}","pipeline_version":"4aeb608f-166b-4b59-bb10-c06f9e49df23","elapsed":[41800,4294967295],"dropped":[]}}]

Undeploy the Pipeline

With the demonstration complete, we can undeploy the pipeline and return the resources back to the Wallaroo instance.

pipeline.undeploy()