Continuous Batching is applied to LLMs in Wallaroo as a model configuration either at model upload or post model upload via the Wallaroo SDK.

Upload Native vLLM Runtime to Wallaroo

LLMs in the native vLLM framework are uploaded either via the Wallaroo SDK or the Wallaroo MLOps API with framework configuration options.

# (Optional) set the VLLMConfig values

standard_framework_config = wallaroo.framework.VLLMConfig(
    max_num_seqs=max_num_seqs,
    max_model_len=max_model_len,
    max_seq_len_to_capture=max_seq_len_to_capture, 
    quantization=quantization, 
    kv_cache_dtype=kv_cache_dtype,
    gpu_memory_utilization=gpu_memory_utilization,
    block_size=block_size,
    device_group=None
)

# upload the vllm model with the framework configuration values
vllm_model = wl.upload_model(model_name, \
                              model_file_name, \
                              framework=wallaroo.framework.Framework.VLLM, \
                              input_schema=input_schema, \
                              output_schema=output_schema \
                              framework_config=standard_framework_config
                            )

# define the input and output parameters in Apache pyarrow format

input_schema = pa.schema([
    pa.field('prompt', pa.string()),
    pa.field('max_tokens', pa.int64()),
])
output_schema = pa.schema([
    pa.field('generated_text', pa.string()),
    pa.field('num_output_tokens', pa.int64())
])

# convert the input and output values to base64
base64.b64encode(
    bytes(input_schema.serialize())
).decode("utf8")

base64.b64encode(
    bytes(output_schema.serialize())
).decode("utf8")

# upload via the Wallaroo MLOps API endpoint using curl
# framework configuration with gpu_memory_utilization=0.9 and max_model_len=128

curl --progress-bar -X POST \
   -H "Content-Type: multipart/form-data" \
   -H "Authorization: Bearer <your-auth-token-here>" \
   -F 'metadata={"name": "<your-model-name>", "visibility": "private", "workspace_id": <your-workspace-id-here>, "conversion": {"framework": "vllm", "python_version": "3.8", "requirements": [], "framework_config": {"config": {"gpu_memory_utilization": 0.9, "max_model_len": 128}, "framework": "vllm"}}, "input_schema": "/////7AAAAAQAAAAAAAKAAwABgAFAAgACgAAAAABBAAMAAAACAAIAAAABAAIAAAABAAAAAIAAABUAAAABAAAAMT///8AAAECEAAAACQAAAAEAAAAAAAAAAoAAABtYXhfdG9rZW5zAAAIAAwACAAHAAgAAAAAAAABQAAAABAAFAAIAAYABwAMAAAAEAAQAAAAAAABBRAAAAAcAAAABAAAAAAAAAAGAAAAcHJvbXB0AAAEAAQABAAAAA==", "output_schema": "/////8AAAAAQAAAAAAAKAAwABgAFAAgACgAAAAABBAAMAAAACAAIAAAABAAIAAAABAAAAAIAAABcAAAABAAAALz///8AAAECEAAAACwAAAAEAAAAAAAAABEAAABudW1fb3V0cHV0X3Rva2VucwAAAAgADAAIAAcACAAAAAAAAAFAAAAAEAAUAAgABgAHAAwAAAAQABAAAAAAAAEFEAAAACQAAAAEAAAAAAAAAA4AAABnZW5lcmF0ZWRfdGV4dAAABAAEAAQAAAA="};type=application/json' \
   -F "file=@<file path to vllm>" \
   https://<Wallaroo Hostname>/v1/api/models/upload_and_convert | cat

# Retrieve the model
vllm_model = wl.get_model(your-model-name)

Set Continuous Batching Configuration for Native vLLM Runtime

wallaroo.client.Client.upload_model.configure includes the following parameters for continuous batching configurations. If no continuous batching configuration is set, the default values are applied.

The following demonstrates defining the continuous batching configuration. Note example the max_concurrent_batch_size is set to 100 to show how to update the value as needed.

# set the continuous batch config max_concurrent_batch_size
# set to 100; defaults to 256
max_concurrent_batch_size = 100
continuous_batch_config = ContinuousBatchingConfig(max_concurrent_batch_size = max_concurrent_batch_size)

The following demonstrates applying the continuous batching configuring during LLM upload via the Wallaroo SDK.

standard_framework_config = wallaroo.framework.VLLMConfig(
    max_num_seqs=max_num_seqs,
    max_model_len=max_model_len,
    max_seq_len_to_capture=max_seq_len_to_capture, 
    quantization=quantization, 
    kv_cache_dtype=kv_cache_dtype,
    gpu_memory_utilization=gpu_memory_utilization,
    block_size=block_size,
    device_group=None
)

# upload the vllm model with the framework configuration values
vllm_model = wl.upload_model(model_name, \
                              model_file_name, \
                              framework=wallaroo.framework.Framework.VLLM, \
                              input_schema=input_schema, \
                              output_schema=output_schema \
                              framework_config=standard_framework_config
                            ).configure(input_schema=input_schema, \
                                        output_schema=output_schema, \
                                        continuous_batch_config=continuous_batch_config \
                                       )

The following demonstrates applying the continuous batching config after LLM upload.

vllm_model.configure(input_schema=input_schema, \
                     output_schema=output_schema, \
                     continuous_batch_config=continuous_batch_config \
                    )

Deploy LLMs Using the Native Wallaroo vLLM Runtime with Continuous Batch Configuration

Deploying LLMs with a Continuous Batching configuration has the following steps.

• Define the deployment configuration to set the number of CPUs, RAM, and GPUs per model replica.
• Create a Wallaroo pipeline and add the pre-configure LLM with the Continuous Batching configuration as a model step.
• Deploy the Wallaroo pipeline with the deployment configuration.

Deployment Configuration for LLMs Deployed Using the Native vLLM Runtime in Wallaroo

The deployment configuration sets what resources are allocated for the LLM. For this example, the LLM is allocated 8 cpus, 10 Gi RAM, and 1 NVIDIA GPU. The specific AI accelerator is inherited from the upload_model step.

deployment_config = DeploymentConfigBuilder() \
    .cpus(1).memory('2Gi') \
    .sidekick_cpus(vllm_model, 8) \
    .sidekick_memory(vllm_model, '10Gi') \
    .sidekick_gpus(vllm_model, 1) \
    .deployment_label("wallaroo.ai/accelerator:a100") \
    .build()

Create Wallaroo Pipeline and Add the LLM with Native vLLM Runtime

Wallaroo pipelines are created with the wallaroo.client.Client.build_pipeline method. Pipeline steps are used to determine how inference data is provided to the LLM.

The following demonstrates creating a Wallaroo pipeline, and assigning the LLM as a pipeline step.

# create the pipeline
vllm_pipeline = wl.build_pipeline('sample-vllm-pipeline')

# add the LLM as a pipeline model step
vllm_pipeline.add_model_step(vllm_model)

Deploy the LLM pipeline With the Native vLLM Runtime and Continuous Batching Configurations

With the Deployment Configuration defined and the pipeline ready, the pipeline is deployed with the wallaroo.pipeline.Pipeline.deploy(deployment_config: Optional[wallaroo.deployment_config.DeploymentConfig]) method. This allocates resources from the cluster for the deployment based on the DeploymentConfig settings. If the resources requested are not available at deployment, an error is returned.

The following example demonstrates deploying the pipeline with the deployment config.

The following example demonstrates deploying the pipeline with the previously defined deployment configuration.

vllm_pipeline.deploy(deployment_config=deployment_config)

Once the deployment configuration is complete, the pipeline is ready to accept inference requests.

Continuous Batching is applied to Wallaroo Custom Models as a model configuration either at model upload or post model upload via the Wallaroo SDK.

LLMs deployed with the Custom vLLM Runtime leverage the vLLM runtime; using Wallaroo Custom Models provides additional flexibility in customizing the LLM behavior (e.g adding inputs and outputs, defining system prompts or context queries).

# vllm import libraries 
from vllm import AsyncLLMEngine, SamplingParams
from vllm.engine.arg_utils import AsyncEngineArgs

class AsyncVLLMInferenceBuilder(InferenceBuilder):
    """Inference builder class for AsyncVLLMInference."""

    def inference(self) -> AsyncVLLMInference: # extend mac.inference.AsyncInference
        """Returns an Inference subclass instance.
        This specifies the Inference instance to be used
        by create() to build additionally needed components."""
        return AsyncVLLMInference()

    def create(self, config: CustomInferenceConfig) -> AsyncVLLMInference:
        """Creates an Inference subclass and assigns a model to it.
        :param config: Inference configuration
        :return: Inference subclass
        """
        inference = self.inference
        inference.model = AsyncLLMEngine.from_engine_args(
            AsyncEngineArgs(
                model=(config.model_path / "model").as_posix(),
            ),
        )
        return inference

Upload Custom vLLM Runtime to Wallaroo

LLMs in the custom vLLM framework are uploaded either via the Wallaroo SDK or the Wallaroo MLOps API with framework configuration options.

# (Optional) set the framework configuration values
custom_framework_config = wallaroo.framework.CustomConfig(
    max_num_seqs=max_num_seqs,
    max_model_len=max_model_len,
    max_seq_len_to_capture=max_seq_len_to_capture, 
    quantization=quantization, 
    kv_cache_dtype=kv_cache_dtype,
    gpu_memory_utilization=gpu_memory_utilization,
    block_size=block_size,
    device_group=None
)

# upload the vllm with the model configuration values
custom_vllm_model = wl.upload_model(model_name, 
                              model_file_name, 
                              framework=wallaroo.framework.Framework.CUSTOM, 
                              input_schema=input_schema, 
                              output_schema=output_schema 
                              framework_config=custom_framework_config
                              )

# define the input and output parameters in Apache pyarrow format

input_schema = pa.schema([
    pa.field('prompt', pa.string()),
    pa.field('max_tokens', pa.int64()),
])
output_schema = pa.schema([
    pa.field('generated_text', pa.string()),
    pa.field('num_output_tokens', pa.int64())
])

# convert the input and output values to base64
base64.b64encode(
    bytes(input_schema.serialize())
).decode("utf8")

base64.b64encode(
    bytes(output_schema.serialize())
).decode("utf8")

# upload via the Wallaroo MLOps API endpoint using curl
# framework configuration with gpu_memory_utilization=0.9 and max_model_len=128

curl --progress-bar -X POST \
   -H "Content-Type: multipart/form-data" \
   -H "Authorization: Bearer <your-auth-token-here>" \
   -F 'metadata={"name": "<your-model-name>", "visibility": "private", "workspace_id": <your-workspace-id-here>, "conversion": {"framework": "custom", "python_version": "3.8", "requirements": [], "framework_config": {"config": {"gpu_memory_utilization": 0.9, "max_model_len": 128}, "framework": "custom"}}, "input_schema": "/////7AAAAAQAAAAAAAKAAwABgAFAAgACgAAAAABBAAMAAAACAAIAAAABAAIAAAABAAAAAIAAABUAAAABAAAAMT///8AAAECEAAAACQAAAAEAAAAAAAAAAoAAABtYXhfdG9rZW5zAAAIAAwACAAHAAgAAAAAAAABQAAAABAAFAAIAAYABwAMAAAAEAAQAAAAAAABBRAAAAAcAAAABAAAAAAAAAAGAAAAcHJvbXB0AAAEAAQABAAAAA==", "output_schema": "/////8AAAAAQAAAAAAAKAAwABgAFAAgACgAAAAABBAAMAAAACAAIAAAABAAIAAAABAAAAAIAAABcAAAABAAAALz///8AAAECEAAAACwAAAAEAAAAAAAAABEAAABudW1fb3V0cHV0X3Rva2VucwAAAAgADAAIAAcACAAAAAAAAAFAAAAAEAAUAAgABgAHAAwAAAAQABAAAAAAAAEFEAAAACQAAAAEAAAAAAAAAA4AAABnZW5lcmF0ZWRfdGV4dAAABAAEAAQAAAA="};type=application/json' \
   -F "file=@<file path to custom vllm runtime>" \
   https://<Wallaroo Hostname>/v1/api/models/upload_and_convert | cat

# Retrieve the model
custom_vllm_model = wl.get_model(your-model-name)

Set Continuous Batching Configuration for Custom vLLM Runtime

Continuous batching configurations applied to a vLLM with the Custom vLLM Framework through the Wallaroo SDK wallaroo.client.Client.upload_model.configure method. If no continuous batching configuration is set, the default values are applied.

This includes the following parameters.

The following demonstrates defining the continuous batching configuration. Note example the max_concurrent_batch_size is set to 100 to show how to update the value as needed.

# set the continuous batch config max_concurrent_batch_size
# set to 100; defaults to 256
max_concurrent_batch_size = 100
continuous_batch_config = ContinuousBatchingConfig(max_concurrent_batch_size = max_concurrent_batch_size)

The following demonstrates applying the continuous batching configuring during LLM upload via the Wallaroo SDK.

# (Optional) set the framework configuration values
custom_framework_config = wallaroo.framework.CustomConfig(
    max_num_seqs=max_num_seqs,
    max_model_len=max_model_len,
    max_seq_len_to_capture=max_seq_len_to_capture, 
    quantization=quantization, 
    kv_cache_dtype=kv_cache_dtype,
    gpu_memory_utilization=gpu_memory_utilization,
    block_size=block_size,
    device_group=None
)

# upload the vllm with the model configuration values
custom_vllm_model = wl.upload_model(model_name, 
                              model_file_name, 
                              framework=wallaroo.framework.Framework.CUSTOM, 
                              input_schema=input_schema, 
                              output_schema=output_schema 
                              framework_config=custom_framework_config
                              ).configure(input_schema=input_schema, 
                                         output_schema=output_schema, 
                                         continuous_batch_config=continuous_batch_config 
                            )

The following demonstrates applying the continuous batching config after LLM upload.

custom_vllm_model.configure(input_schema=input_schema, 
                            output_schema=output_schema, 
                            continuous_batch_config=continuous_batch_config 
                           )

Deploy LLMs Using the Custom Wallaroo vLLM Runtime with Continuous Batch Configuration

Deploying a Custom LLM with a Continuous Batching configuration has the following steps.

• Define the deployment configuration to set the number of CPUs, RAM, and GPUs per replica.
• Create a Wallaroo pipeline and add the Custom Model with the Continuous Batch configuration as a model step.
• Deploy the Wallaroo pipeline with the deployment configuration.

Deployment Configuration for LLMs Deployed Using the Custom vLLM Runtime in Wallaroo

The deployment configuration sets what resources are allocated for the Custom Model’s use. For this example, the LLM is allocated 8 cpus, 10 Gi RAM, and 1 NVIDIA GPU. The specific AI accelerator is inherited from the upload_model step.

deployment_config = DeploymentConfigBuilder() \
    .cpus(1).memory('2Gi') \
    .sidekick_cpus(custom_vllm_model, 8) \
    .sidekick_memory(custom_vllm_model, '10Gi') \
    .sidekick_gpus(custom_vllm_model, 1) \
    .deployment_label("wallaroo.ai/accelerator:a100") \
    .build()

Create Wallaroo Pipeline and Add the LLM with Custom vLLM Runtime

Wallaroo pipelines are created with the wallaroo.client.Client.build_pipeline method. Pipeline steps are used to determine how inference data is provided to the LLM.

The following demonstrates creating a Wallaroo pipeline, and adding the LLM as a pipeline step.

# create the pipeline
custom_vllm_pipeline = wl.build_pipeline('sample-vllm-pipeline')

# add the LLM as a pipeline model step
custom_vllm_pipeline.add_model_step(custom_vllm_model)

Deploy the LLM pipeline With the Custom vLLM Runtime and Continuous Batching Configurations

With the Deployment Configuration assigned to the model and the pipeline ready, the pipeline is deployed with the wallaroo.pipeline.Pipeline.deploy(deployment_config: Optional[wallaroo.deployment_config.DeploymentConfig]) method. This allocates resources from the cluster for the deployment based on the DeploymentConfig settings. If the resources set in the deployment configuration are not available at deployment, an error is returned.

The following example demonstrates deploying the pipeline with the previously defined deployment configuration:

vllm_pipeline.deploy(deployment_config=deployment_config)

Once the deployment configuration is complete, the pipeline is ready to accept inference requests.

Wallaroo pipelines using either native or custom vLLM runtime with continuous batching are published to Open Container Initiative (OCI) Registries for remote/edge deployments via the wallaroo.pipeline.Pipeline.publish(deployment_config) command. This uploads the following artifacts to the OCI registry:

• The native or custom LLM with:
  • vLLM Framework configurations
  • Continuous Batching Config
• If specified, the deployment configuration
• The Wallaroo engine for the architecture and AI accelerator, both inherited from the model settings at model upload.

Once the publish process is complete, the pipeline can be deployed to one or more edge/remote environments.

For more details, see Edge and Multi-cloud Pipeline Publish.

The following example demonstrates publishing a Wallaroo pipeline with a native vLLM runtime.

pipeline.publish(deployment_config=deployment_config)

The following error messages may appear if the Framework Configuration Exceptions are triggered.

Continuous Batching Not Supported with Single Batch Mode

If the LLM’s configuration sets batch_config="single", the following error message will appear during model upload:

"Continuous batching is not supported with single batch mode. Please update the model configuration or contact wallaroo for support at support@wallaroo.ai”

Continuous Batching Not Supported with Multiple Batches

Inference requests to a LLM deployed with continuous batching must only include a single batch per user request. If a multiple batch inference request is made (aka - an inference request with multiple rows), the following error is returned.

"Continuous batching is not supported with more than one batch at a time, please send requests with a single batch size or contact Wallaroo for support at support@wallaroo.ai"