Wallaroo supports LLM models configured for deployment with the SGLang Framework. The Wallaroo SGLang framework provides AI acceleration hardware AMD ROCm.

Requirements

SGLang models always run in the Wallaroo Containerized Runtime.

SGLang Configurations and Compatibility

AI Acceleration Compatibility

The following AI Acceleration hardware frameworks are supported by Wallaroo’s SGLang framework.

AI acceleration is set at model upload.

OpenAI Compatibility

OpenAI compatibility is enabled by default. Additional configuration options are set post model upload as a model configuration setting.

Standard Wallaroo PyArrow schemas are not supported.

Continuous Batching

Continuous batching is configured at the model level post model upload. When enabled, async inference requests are enabled.

Default Framework Configurations

SGLang configuration options are modified through the wallaroo.Framework.SGLangConfig object, and includes the following parameters. These are set at model upload.

The following SGLang parameters are *not configurable.

• attention_backend=flashinfer

The following is an example of uploading a model in the SGLang framework with customized options. Any value not specified in the SGLangConfig parameter is set to the default values listed above.

model_step = wl.upload_model(
    "tinyllamaopenaiyns4",
    "sglang-openai_tinyllama.zip",
    framework=Framework.SGLang,
    framework_config=SGLangConfig(
        context_length=4096, 
        chunked_prefill_size=2048
    ),
    input_schema=pa.schema([]),
    output_schema=pa.schema([]),
    convert_wait=True,
    accel=Acceleration.ROCM
)

vLLM Migration

vLLM models set with the hardware acceleration NVIDIA CUDA are migrated to the SGLang framework with ROCm in Wallaroo by uploading the model and specifying SGLang as the new framework.

For example, a vLLM previously uploaded via the Wallaroo SDK as:

model = wl.upload_model(
  name = model_name,
  path = file_path,
  input_schema = pyarrow.schema([]),
  output_schema = pyarrow.schema([]),
  framework = Framework.VLLM,
  accel=Acceleration.CUDA
)

is migrated in the same workspace using the same settings except the framework is changed to SGLang and the acceleration set to ROCM. For example:

model = wl.upload_model(
  name = model_name,
  path = file_path,
  input_schema = pyarrow.schema([]),
  output_schema = pyarrow.schema([]),
  framework = Framework.SGLang,
  accel=Acceleration.ROCM
)

Upload

Models are uploaded to the Wallaroo Ops instance via either the Wallaroo SDK or the Wallaroo MLOps API. During model upload, the following options are available:

• OpenAI API compatibility is enabled by default.
• AI Hardware Accelerator type is set at model upload; to change the hardware accelerator, the model must be uploaded as a new model version.

import wallaroo
import pyarrow as pa
from wallaroo.framework import Framework
from wallaroo.engine_config import Acceleration

# connect to Wallaroo

wl = wallaroo.Client()

# upload the model

model = wl.upload_model(
  name = model_name,
  path = file_path,
  input_schema = pyarrow.schema([]),
  output_schema = pyarrow.schema([]),
  framework = Framework.SGLang,
  accel=Acceleration.ROCM
)

curl --progress-bar -X POST \
      -H "Content-Type: multipart/form-data" \
      -H "Authorization: Bearer {Bearer Token}"
      -F "metadata={"name": {name}, "visibility": "private", "workspace_id": {Current Workspace['id']}, "conversion": {"arch": "x86", "accel": "none", "framework": "{framework}", "python_version": "3.8", "requirements": []}, \
      "input_schema": "{base64 version of input_schema}", \
      "output_schema": "{base64 version of the output_schema}";type=application/json" \
      -F "file=@{path};type=application/octet-stream" \
      https://{base_url}/v1/api/models/upload_and_convert

import wallaroo
import pyarrow as pa

# set the input and output schemas

input_schema = pa.schema([
    pa.field("text", pa.string())
])

output_schema = pa.schema([
    pa.field("generated_text", pa.string())
])

# use the generate model upload api command

wl.generate_upload_model_api_command(
    base_url='https://example.wallaroo.ai/',
    name='sample_model_name', 
    path='llama_byop.zip',
    framework={insert framework here},
    input_schema=input_schema,
    output_schema=output_schema)

curl --progress-bar -X POST -H "Content-Type: multipart/form-data" -H "Authorization: Bearer abc123" -F "metadata={"name": "sample_model_name", "visibility": "private", "workspace_id": 20, "conversion": {"arch": "x86", "accel": "none", "framework": "{framework type here}", "python_version": "3.8", "requirements": []}, "input_schema": "/////3AAAAAQAAAAAAAKAAwABgAFAAgACgAAAAABBAAMAAAACAAIAAAABAAIAAAABAAAAAEAAAAUAAAAEAAUAAgABgAHAAwAAAAQABAAAAAAAAEFEAAAABwAAAAEAAAAAAAAAAQAAAB0ZXh0AAAAAAQABAAEAAAA", "output_schema": "/////3gAAAAQAAAAAAAKAAwABgAFAAgACgAAAAABBAAMAAAACAAIAAAABAAIAAAABAAAAAEAAAAUAAAAEAAUAAgABgAHAAwAAAAQABAAAAAAAAEFEAAAACQAAAAEAAAAAAAAAA4AAABnZW5lcmF0ZWRfdGV4dAAABAAEAAQAAAA="};type=application/json" -F "file=@llama_byop.zip;type=application/octet-stream"        https://example.wallaroo.ai/v1/api/models/upload_and_convert'

LLM Deploy

LLMs are deployed via the Wallaroo SDK through the following process:

1. After the model is uploaded, get the LLM model reference from Wallaroo.
2. Create or use an existing Wallaroo pipeline and assign the LLM as a pipeline model step.
3. Set the deployment configuration to assign the resources including the number of CPUs, amount of RAM, etc for the LLM deployment.
4. Deploy the LLM with the deployment configuration.

Retrieve LLM

LLMs previously uploaded to Wallaroo can be retrieved without re-uploading the LLM via the Wallaroo SDK method wallaroo.client.Client.get_model(name:String, version:String) which takes the following parameters:

• name: The name of the model.

The method wallaroo.client.get_model(name) retrieves the most recent model version in the current workspace that matches the provided model name unless a specific version is requested. For more details on managing ML models in Wallaroo, see Manage Models.

The following demonstrates retrieving an uploaded LLM and storing it in the variable model_version.

import wallaroo

# connect with the Wallaroo client

wl = wallaroo.Client()

llm_model = wl.get_model(name=model_name)

OpenAI Compatibility Configuration

OpenAI compatibility is enabled via the model configuration from the class wallaroo.openai_config.OpenaiConfig; by default, OpenAI compatibility is enabled for the SGLang framework with the default parameters. Note that if OpenAI compatibility is disabled, all other parameters are ignored.

With the OpenaiConfig object defined, it is when applied to the LLM configuration through the openai_config parameter.

from wallaroo.openai_config import OpenaiConfig

openai_config = OpenaiConfig(chat_completion_config={"temperature": .3, "max_tokens": 200})
llm_model = llm_model.configure(openai_config=openai_config)

Continuous Batching Configuration

Continuous batching allows SGLang to process inference requests asynchronously, improving throughput under concurrent load. It is configured post-upload via the wallaroo.continuous_batching_config.ContinuousBatchingConfig class.

The following example shows updating the ContinuousBatchingConfig values for the uploaded LLM.

from wallaroo.continuous_batching_config import ContinuousBatchingConfig

llm_model = llm_model.configure(
    continuous_batching_config=ContinuousBatchingConfig(max_concurrent_batch_size=5)
)

Create the Wallaroo Pipeline and Add Model Step

LLMs are deployed via Wallaroo pipelines. Wallaroo pipelines are created in the current user’s workspace with the Wallaroo SDK method wallaroo.client.Client.build_pipeline(pipeline_name:String) method. This creates a pipeline in the user’s current workspace with with provided pipeline_name, and returns wallaroo.pipeline.Pipeline, which can be saved to a variable for other commands.

Pipeline names are unique within a workspace; using the build_pipeline method within a workspace where another pipeline with the same name exists will connect to the existing pipeline.

Once the pipeline reference is stored to a variable, LLMs are added to the pipeline as a pipeline step with the method wallaroo.pipeline.Pipeline.add_model_step(model_version: wallaroo.model_version.ModelVersion). We demonstrated retrieving the LLM model version in the step Get Model.

This example demonstrates creating a pipeline and adding a model version as a pipeline step. For more details on managing Wallaroo pipelines for model deployment, see the Model Deploy guide.

# create the pipeline
llm_pipeline = wl.build_pipeline('sample-sglang-pipeline')

# add the LLM as a pipeline model step
llm_pipeline.add_model_step(llm_model)

Set the Deployment Configuration and Deploy the Model

Before deploying the LLM, a deployment configuration is created. This sets how the cluster’s resources are allocated for the LLM’s exclusive use.

• Pipeline deployment configurations are created through the wallaroo.deployment_config.DeploymentConfigBuilder() class.
• Various options including the number of cpus, RAM, and other resources are set for the Wallaroo Native Runtime, and the Wallaroo Containerized Runtime.
  • Typically, LLMs are deployed in the Wallaroo Containerized Runtime, which are referenced in the DeploymentConfigBuilder’s sidekick options.
• Once the configuration options are set the deployment configuration is finalized with the wallaroo.deployment_config.DeploymentConfigBuilder().build() method.

The following options are available for deployment configurations for LLM deployments. For more details on deployment configurations, see Deployment Configuration guide.

Once the deployment configuration is set, the LLM is deployed via the wallaroo.pipeline.Pipeline.deploy(deployment_config: Optional[wallaroo.deployment_config.DeploymentConfig]) method. This allocates resources from the cluster for the LLMs deployment based on the DeploymentConfig settings. If the resources set in the deployment configuration are not available at deployment, an error is returned.

The following example shows setting the deployment configuration for a LLM for deployment on x86 architecture, then deploying a pipeline with this deployment configuration.

from wallaroo.deployment_config import DeploymentConfigBuilder

# set the deployment config with the following:
# Wallaroo Native Runtime:  0.5 cpu, 2 Gi RAM
# Wallaroo Containerized Runtime where the LLM is deployed:  32 CPUs and 40 Gi RAM, 2 GPUS
deployment_config = DeploymentConfigBuilder() \
    .cpus(0.5).memory('2Gi') \
    .sidekick_cpus(llm_model, 32) \
    .sidekick_memory(llm_model, '40Gi') \
    .sidekick_gpus(llm_model, 2) \
    .deployment_label(deployment_label) \
    .build()

llm_pipeline.deploy(deployment_config)

Queue-based Autoscaling Example

Queue-based autoscaling is recommended for LLM deployments since continuous batching makes CPU utilization an unreliable scaling signal. When enabled, queue_depth based settings override CPU based autoscaling deployment configurations.

The following example scales up when queue depth exceeds 4, scales down when it drops below 2, and evaluates metrics over a 60-second window.

from wallaroo.deployment_config import DeploymentConfigBuilder

deployment_config = DeploymentConfigBuilder() \
    .cpus(0.5).memory('2Gi') \
    .sidekick_cpus(llm_model, 32) \
    .sidekick_memory(llm_model, '40Gi') \
    .sidekick_gpus(llm_model, 2) \
    .deployment_label(deployment_label) \
    .replica_autoscale_min_max(maximum=4, minimum=1) \
    .scale_up_queue_depth(4) \
    .scale_down_queue_depth(2) \
    .autoscaling_window(60) \
    .build()
llm_pipeline.deploy(deployment_config)

Once deployed, inferences are available. For details on inference requests for OpenAI Compatibility enabled models, see Inference via OpenAI Compatibility Deployments.

Tutorials

• Llama 8B in SGLang Framework with ROCm AI Acceleration Example