Virtual Large Language Model (vLLMs) uploads and deployment are supported by Wallaroo. vLLMs deployed in Wallaroo provide hardware acceleration support, OpenAI API compatibility, and resources management provided by Wallaroo’s deployment configurations. The following models examples of models that can take advantage of the Wallaroo vLLM framework. Note that not every scenario has been tested; for specific models and options, consult with your Wallaroo support representative.

• Mistral models
  • Codestral: https://huggingface.co/mistralai/Codestral-22B-v0.1/tree/main
  • Devstral: https://huggingface.co/mistralai/Devstral-2-123B-Instruct-2512
• OpenAI
  • gpt-oss: https://huggingface.co/openai/gpt-oss-120b
• Llama
  • Llama-3.1-8B-Instruct: https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct
  • Llama-3.3-70B-Instruct: https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct/tree/main
  • Llama-4-Scout-17B-16E-Instruct: https://huggingface.co/meta-llama/Llama-4-Scout-17B-16E-Instruct
• Deepseek Qwen
  • Qwen-VL: https://huggingface.co/Qwen/Qwen-VL
  • Qwen3-Coder-Next: https://huggingface.co/Qwen/Qwen3-Coder-Next

Requirements

vLLM models always run in the Wallaroo Containerized Runtime.

vLLM Configurations and Compatibility

AI Acceleration Compatibility

The following AI Acceleration hardware frameworks are supported by Wallaroo’s vLLM framework.

AI acceleration is set at model upload.

OpenAI Compatibility

OpenAI compatibility is enabled by default. Additional configuration options are set post model upload as a model configuration setting.

When OpenAI compatibility is disabled, standard Wallaroo PyArrow schemas are supported.

Continuous Batching

Continuous batching is configured at the model level post model upload.

Upload

Models are uploaded to the Wallaroo Ops instance via either the Wallaroo SDK or the Wallaroo MLOps API. During model upload, the following options are available:

• OpenAI API compatibility is enabled by default.
• AI Hardware Accelerator type is set at model upload; to change the hardware accelerator, the model must be uploaded as a new model version.

import wallaroo
import pyarrow as pa
from wallaroo.framework import Framework
from wallaroo.engine_config import Acceleration

# connect to Wallaroo

wl = wallaroo.Client()

# upload the model

model = wl.upload_model(
  name = model_name,
  path = file_path,
  input_schema = pyarrow.schema([]),
  output_schema = pyarrow.schema([]),
  framework = Framework.VLLM,
  accel=Acceleration.CUDA
)

import pyarrow as pa
from wallaroo.framework import Framework, VLLMConfig
from wallaroo.engine_config import Acceleration

model = wl.upload_model(
    name=model_name,
    path=file_path,
    framework=Framework.VLLM,
    framework_config=VLLMConfig(
        gpu_memory_utilization=0.85,
        max_model_len=4096,
        quantization="fp8"
    ),
    input_schema=pa.schema([]),
    output_schema=pa.schema([]),
    accel=Acceleration.CUDA
)

curl --progress-bar -X POST \
      -H "Content-Type: multipart/form-data" \
      -H "Authorization: Bearer {Bearer Token}"
      -F "metadata={"name": {name}, "visibility": "private", "workspace_id": {Current Workspace['id']}, "conversion": {"arch": "x86", "accel": "none", "framework": "{framework}", "python_version": "3.8", "requirements": []}, \
      "input_schema": "{base64 version of input_schema}", \
      "output_schema": "{base64 version of the output_schema}";type=application/json" \
      -F "file=@{path};type=application/octet-stream" \
      https://{base_url}/v1/api/models/upload_and_convert

import wallaroo
import pyarrow as pa

# set the input and output schemas

input_schema = pa.schema([
    pa.field("text", pa.string())
])

output_schema = pa.schema([
    pa.field("generated_text", pa.string())
])

# use the generate model upload api command

wl.generate_upload_model_api_command(
    base_url='https://example.wallaroo.ai/',
    name='sample_model_name', 
    path='llama_byop.zip',
    framework={insert framework here},
    input_schema=input_schema,
    output_schema=output_schema)

curl --progress-bar -X POST -H "Content-Type: multipart/form-data" -H "Authorization: Bearer abc123" -F "metadata={"name": "sample_model_name", "visibility": "private", "workspace_id": 20, "conversion": {"arch": "x86", "accel": "none", "framework": "{framework type here}", "python_version": "3.8", "requirements": []}, "input_schema": "/////3AAAAAQAAAAAAAKAAwABgAFAAgACgAAAAABBAAMAAAACAAIAAAABAAIAAAABAAAAAEAAAAUAAAAEAAUAAgABgAHAAwAAAAQABAAAAAAAAEFEAAAABwAAAAEAAAAAAAAAAQAAAB0ZXh0AAAAAAQABAAEAAAA", "output_schema": "/////3gAAAAQAAAAAAAKAAwABgAFAAgACgAAAAABBAAMAAAACAAIAAAABAAIAAAABAAAAAEAAAAUAAAAEAAUAAgABgAHAAwAAAAQABAAAAAAAAEFEAAAACQAAAAEAAAAAAAAAA4AAABnZW5lcmF0ZWRfdGV4dAAABAAEAAQAAAA="};type=application/json" -F "file=@llama_byop.zip;type=application/octet-stream"        https://example.wallaroo.ai/v1/api/models/upload_and_convert'

LLM Deploy

LLMs are deployed via the Wallaroo SDK through the following process:

1. After the model is uploaded, get the LLM model reference from Wallaroo.
2. Create or use an existing Wallaroo pipeline and assign the LLM as a pipeline model step.
3. Set the deployment configuration to assign the resources including the number of CPUs, amount of RAM, etc for the LLM deployment.
4. Deploy the LLM with the deployment configuration.

Retrieve LLM

LLMs previously uploaded to Wallaroo can be retrieved without re-uploading the LLM via the Wallaroo SDK method wallaroo.client.Client.get_model(name:String, version:String) which takes the following parameters:

• name: The name of the model.

The method wallaroo.client.get_model(name) retrieves the most recent model version in the current workspace that matches the provided model name unless a specific version is requested. For more details on managing ML models in Wallaroo, see Manage Models.

The following demonstrates retrieving an uploaded LLM and storing it in the variable model_version.

import wallaroo

# connect with the Wallaroo client

wl = wallaroo.Client()

llm_model = wl.get_model(name=model_name)

OpenAI Compatibility Configuration

OpenAI compatibility is enabled via the model configuration from the class wallaroo.openai_config.OpenaiConfig; by default, OpenAI compatibility is enabled for the vLLM framework with the default parameters. Note that if OpenAI compatibility is disabled, all other parameters are ignored.

With the OpenaiConfig object defined, it is when applied to the LLM configuration through the openai_config parameter.

from wallaroo.openai_config import OpenaiConfig

openai_config = OpenaiConfig(chat_completion_config={"temperature": .3, "max_tokens": 200})
llm_model = llm_model.configure(openai_config=openai_config)

Continuous Batching Configuration

Continuous batching allows vLLM to process inference requests asynchronously, improving throughput under concurrent load. It is configured post-upload via the wallaroo.continuous_batching_config.ContinuousBatchingConfig class and contains the following parameters.

The following is an example of modifying the ContinuousBatchingConfig to set the max_concurrent_batch_size to 5.

from wallaroo.continuous_batching_config import ContinuousBatchingConfig

llm_model = llm_model.configure(
    continuous_batching_config=ContinuousBatchingConfig(max_concurrent_batch_size=5)
)

Create the Wallaroo Pipeline and Add Model Step

LLMs are deployed via Wallaroo pipelines. Wallaroo pipelines are created in the current user’s workspace with the Wallaroo SDK method wallaroo.client.Client.build_pipeline(pipeline_name:String) method. This creates a pipeline in the user’s current workspace with with provided pipeline_name, and returns wallaroo.pipeline.Pipeline, which can be saved to a variable for other commands.

Pipeline names are unique within a workspace; using the build_pipeline method within a workspace where another pipeline with the same name exists will connect to the existing pipeline.

Once the pipeline reference is stored to a variable, LLMs are added to the pipeline as a pipeline step with the method wallaroo.pipeline.Pipeline.add_model_step(model_version: wallaroo.model_version.ModelVersion). We demonstrated retrieving the LLM model version in the step Get Model.

This example demonstrates creating a pipeline and adding a model version as a pipeline step. For more details on managing Wallaroo pipelines for model deployment, see the Model Deploy guide.

# create the pipeline
llm_pipeline = wl.build_pipeline('sample-llm-pipeline')

# add the LLM as a pipeline model step
llm_pipeline.add_model_step(llm_model)

Set the Deployment Configuration and Deploy the Model

Before deploying the LLM, a deployment configuration is created. This sets how the cluster’s resources are allocated for the LLM’s exclusive use.

• Pipeline deployment configurations are created through the wallaroo.deployment_config.DeploymentConfigBuilder() class.
• Various options including the number of cpus, RAM, and other resources are set for the Wallaroo Native Runtime, and the Wallaroo Containerized Runtime.
  • Typically, LLMs are deployed in the Wallaroo Containerized Runtime, which are referenced in the DeploymentConfigBuilder’s sidekick options.
• Once the configuration options are set the deployment configuration is finalized with the wallaroo.deployment_config.DeploymentConfigBuilder().build() method.

The following options are available for deployment configurations for LLM deployments. For more details on deployment configurations, see Deployment Configuration guide.

Once the deployment configuration is set, the LLM is deployed via the wallaroo.pipeline.Pipeline.deploy(deployment_config: Optional[wallaroo.deployment_config.DeploymentConfig]) method. This allocates resources from the cluster for the LLMs deployment based on the DeploymentConfig settings. If the resources set in the deployment configuration are not available at deployment, an error is returned.

The following example shows setting the deployment configuration for a LLM for deployment on x86 architecture, then deploying a pipeline with this deployment configuration.

from wallaroo.deployment_config import DeploymentConfigBuilder

# set the deployment config with the following:
# Wallaroo Native Runtime:  0.5 cpu, 2 Gi RAM
# Wallaroo Containerized Runtime where the LLM is deployed:  32 CPUs and 40 Gi RAM, 2 GPUS
deployment_config = DeploymentConfigBuilder() \
    .cpus(0.5).memory('2Gi') \
    .sidekick_cpus(llm_model, 32) \
    .sidekick_memory(llm_model, '40Gi') \
    .sidekick_gpus(llm_model, 2) \
    .deployment_label(deployment_label) \
    .build()

llm_pipeline.deploy(deployment_config)

Queue-based Autoscaling Example

Queue-based autoscaling is recommended for LLM deployments since continuous batching makes CPU utilization an unreliable scaling signal. When enabled, queue_depth based settings override CPU based autoscaling deployment configurations.

The following example scales up when queue depth exceeds 4, scales down when it drops below 2, and evaluates metrics over a 60-second window.

from wallaroo.deployment_config import DeploymentConfigBuilder

deployment_config = DeploymentConfigBuilder() \
    .cpus(0.5).memory('2Gi') \
    .sidekick_cpus(llm_model, 32) \
    .sidekick_memory(llm_model, '40Gi') \
    .sidekick_gpus(llm_model, 2) \
    .deployment_label(deployment_label) \
    .replica_autoscale_min_max(maximum=4, minimum=1) \
    .scale_up_queue_depth(4) \
    .scale_down_queue_depth(2) \
    .autoscaling_window(60) \
    .build()
llm_pipeline.deploy(deployment_config)

Once deployed, inferences are available. For details on inference requests for OpenAI Compatibility enabled models, see Inference via OpenAI Compatibility Deployments.