The following procedure demonstrates how to upload and deploy a LLM with GPUs. The majority are Hugging Face LLMs packaged as a Wallaroo BYOP framework models.

These upload and deploy instructions have been tested with and apply to the following LLM models:

• Llama
  • Llama v2 7B standard on 1 GPU
  • Llama v2 7B chat on 1 GPU
  • Llama v2 7B instruct on 1 GPU
  • Llama v2 70B quantized on 1 GPU
  • Llama v3 8B standard on 1 GPU
  • Llama v2 8B instruct on 1 GPU
• IBM-Granite

For access to these sample models and a demonstration on using LLMs with Wallaroo:

• Contact your Wallaroo Support Representative OR
• Schedule Your Wallaroo.AI Demo Today

Upload the LLM Model

LLM models are uploaded to Wallaroo via one of two methods:

• The Wallaroo SDK wallaroo.client.Client.upload_model method.
• The Wallaroo MLOps API POST /v1/api/models/upload_and_convert endpoint.

import wallaroo

# connect to Wallaroo

wl = wallaroo.Client()

# upload the model

model = wl.upload_client(
  name = model_name,
  path = file_path,
  input_schema = input_schema,
  output_schema = output.schema,
  framework = framework
)

export WALLAROO_USERNAME="username"
export WALLAROO_PASSWORD="password"
export WALLAROO_AUTH_URL="keycloak.wallaroo.example.com"

TOKEN=$(curl -s -X POST "https://$WALLAROO_AUTH_URL/auth/realms/master/protocol/openid-connect/token" \
                 -d client_id=sdk-client \
                 -d username=${WALLAROO_USERNAME} \
                 -d password=${WALLAROO_PASSWORD} \
                 -d grant_type=password | jq .access_token )
echo $TOKEN

"abc123"

import pyarrow as pa
import base64

input_schema = pa.schema([
    pa.field('input_1', pa.list_(pa.float32(), list_size=10)),
    pa.field('input_2', pa.list_(pa.float32(), list_size=5))
])
output_schema = pa.schema([
    pa.field('output_1', pa.list_(pa.float32(), list_size=3)),
    pa.field('output_2', pa.list_(pa.float32(), list_size=2))
])

encoded_input_schema = base64.b64encode(
                bytes(input_schema.serialize())
            ).decode("utf8")

encoded_output_schema = base64.b64encode(
                bytes(output_schema.serialize())
            ).decode("utf8")

curl --progress-bar -X POST \
  -H "Content-Type: multipart/form-data" \
  -H "Authorization: Bearer $TOKEN" \
  -F 'metadata=\ 
      { \
        "name": $NAME, \
        "visibility": "private", \
        "workspace_id": $WORKSPACE_ID, \
        "conversion": \
          {\
            "framework": $FRAMEWORK, \
            "python_version": "3.8", \
            "requirements": []}, \
            "input_schema": $INPUT_SCHEMA, \
            "output_schema": $OUTPUT_SCHEMA, \
            "arch": 'x86' \
          };type=application/json' \
  -F "file=@$MODEL_PATH;type=application/octet-stream" \
$URL/v1/api/models/upload_and_convert | cat

LLM Deploy

LLM’s are deployed via the Wallaroo SDK through the following process:

1. After the model is uploaded, get the LLM model reference from Wallaroo.
2. Create or use an existing Wallaroo pipeline and assign the LLM as a pipeline model step.
3. Set the deployment configuration to assign the resources including the number of CPUs, amount of RAM, etc for the LLM deployment.
4. Deploy the LLM with the deployment configuration.

Retrieve LLM

LLM’s previously uploaded to Wallaroo can be retrieved without re-uploading the LLM via the Wallaroo SDK method wallaroo.client.Client.get_model(name:String, version:String) which takes the following parameters:

• name: The name of the model.

The method wallaroo.client.get_model(name) retrieves the most recent model version in the current workspace that matches the provided model name unless a specific version is requested. For more details on managing ML models in Wallaroo, see Manage Models.

The following demonstrates retrieving an uploaded LLM and storing it in the variable model_version.

import wallaroo

# connect with the Wallaroo client

wl = wallaroo.Client()

llm_model = wl.get_model(name=model_name)

Create the Wallaroo Pipeline and Add Model Step

LLMs are deployed via Wallaroo pipelines. Wallaroo pipelines are created in the current user’s workspace with the Wallaroo SDK method wallaroo.client.Client.build_pipeline(pipeline_name:String) method. This creates a pipeline in the user’s current workspace with with provided pipeline_name, and returns wallaroo.pipeline.Pipeline, which can be saved to a variable for other commands.

Pipeline names are unique within a workspace; using the build_pipeline method within a workspace where another pipeline with the same name exists will connect to the existing pipeline.

Once the pipeline reference is stored to a variable, LLMs are added to the pipeline as a pipeline step with the method wallaroo.pipeline.Pipeline.add_model_step(model_version: wallaroo.model_version.ModelVersion). We demonstrated retrieving the LLM model version in the step Get Model.

This example demonstrates creating a pipeline and adding a model version as a pipeline step. For more details on managing Wallaroo pipelines for model deployment, see the Model Deploy guide.

# create the pipeline
llm_pipeline = wl.build_pipeline('sample-llm-pipeline')

# add the LLM as a pipeline model step
llm_pipeline.add_model_step(llm_model)

Set the Deployment Configuration and Deploy the Model

Before deploying the LLM, a deployment configuration is created. This sets how the cluster’s resources are allocated for the LLM’s exclusive use.

• Pipeline deployment configurations are created through the wallaroo.deployment_config.DeploymentConfigBuilder() class.
• Various options including the number of cpus, RAM, and other resources are set for the Wallaroo Native Runtime, and the Wallaroo Containerized Runtime.
  • Typically, LLM’s are deployed in the Wallaroo Containerized Runtime, which are referenced in the DeploymentConfigBuilder’s sidekick options.

LLMs deployed with GPUs must include the following parameters:

• sidekick_cpus(core_count: int): Sets the number of GPUs allocated to the LLM.
• deployment_label(label: string): The deployment label that matches the nodepool with the GPU nodes. This ensures that the LLM is deployed in the correct nodepool with the required hardware. For examples on setting up a nodepool with GPUs for LLM deployment, see Large Language Models Infrastructure Requirements.

Once the configuration options are set the deployment configuration is finalized with the wallaroo.deployment_config.DeploymentConfigBuilder().build() method.

The following options are available for deployment configurations for LLM deployments. For more details on deployment configurations, see Deployment Configuration guide.

Once the deployment configuration is set, the LLM is deployed via the wallaroo.pipeline.Pipeline.deploy(deployment_config: Optional[wallaroo.deployment_config.DeploymentConfig]) method. This allocates resources from the cluster for the LLMs deployment based on the DeploymentConfig settings. If the resources set in the deployment configuration are not available at deployment, an error is returned.

The following example shows setting the deployment configuration for a LLM for deployment on x86 architecture with a single GPU, then deploying a pipeline with this deployment configuration.

# set the deployment config with the following:
# Wallaroo Native Runtime:  0.5 cpu, 2 Gi RAM
# Wallaroo Containerized Runtime where the LLM is deployed:  32 CPUs and 40 Gi RAM
deployment_config = DeploymentConfigBuilder() \
    .cpus(0.5).memory('2Gi') \
    .sidekick_cpus(llm_model, 2) \
    .sidekick_memory(llm_model, '40Gi') \
    .sidekick_gpus(llm_model, 1) \
    .deployment_label(deployment_label) \
    .build()

llm_pipeline.deploy(deployment_config)

LLM Upload and Deploy Tutorials

• LLM Deploy on GPU Tutorial