The Wallaroo SDK provides QAIC support for models uploaded for Wallaroo Ops.

Upload Models for QAIC via the Wallaroo SDK

Models are uploaded to Wallaroo via the wallaroo.client.upload_model method. For QAIC support, the following architecture and acceleration settings are used:

• The AI acceleration is set with the accel parameter. For QAIC, this accepts the wallaroo.engine_config.Acceleration.QAIC.
  • (Optional) Set the acceleration configuration options to fine tune hardware performance.

Note that QAIC processors are x86 compatible, so no changes are needed to the model upload default architecture of X86.

The method wallaroo.client.Client.upload_model takes the following parameters:

QAIC hardware performance is configurable at model upload with wallaroo.engine_config.Acceleration.QAIC.with_config(wallaroo.engine_config.QaicConfig). This provides additional hardware fine tuning. If no acceleration parameters are defined, the default values are applied.

wallaroo.engine_config.QaicConfig takes the following parameters.

Upload Model for QAIC AI Acceleration via the Wallaroo SDK Example

The following demonstrates uploading a model for deployment on with QAIC AI acceleration. The input and output schemas are optional depending on the model runtime. For more details, see Model Upload

The following shows uploading the LLM with QAIC AI acceleration enabled without the acceleration configuration options.

import wallaroo

# set the Wallaroo client
wl = wallaroo.Client()

model = wl.upload_model(
    model_name, 
    model_file_name, 
    framework=framework,
    input_schema=input_schema, 
    output_schema=output_schema, 
    accel=Acceleration.QAIC
)

The following demonstrates uploading the LLM with QAIC AI acceleration enabled with the acceleration configuration options.

import wallaroo

# set the Wallaroo client
wl = wallaroo.Client()

# Set the QAIC acceleration parameters.  This is an **optional** step
qaic_config = wallaroo.engine_config.QaicConfig(
    num_devices=4, 
    full_batch_size=16, 
    ctx_len=256, 
    prefill_seq_len=128, 
    mxfp6_matmul=True, 
    mxint8_kv_cache=True
)

model = wl.upload_model(
    "sample-model-name", 
    "sample-model-file.zip", 
    framework=framework
    input_schema=input_schema, 
    output_schema=output_schema, 
    accel=Acceleration.QAIC.with_config(qaic_config)
)

Deploy Models for QAIC AI Acceleration via the Wallaroo SDK

Models are added to pipeline as pipeline steps. Models are then deployed through the wallaroo.pipeline.Pipeline.deploy(deployment_config: Optional[wallaroo.deployment_config.DeploymentConfig] = None) method.

When deploying a model in a Wallaroo Ops instance, the deployment configurations inherits the model acceleration setting.Other settings, such as the number of CPUs, etc can be changed without modifying the acceleration setting.

The deployment configuration sets what resources are allocated for the model. For this example, the model is allocated the following:

• cpus: 4
• RAM: 12 Gi
• gpus: 4
  • For Wallaroo deployment configurations for QAIC, the gpu parameter specifies the number of SoCs allocated.
• Deployment label: Specifies the node with the QAIC SoCs.

from wallaroo.deployment_config import DeploymentConfigBuilder

deployment_config = DeploymentConfigBuilder() \
    .replica_autoscale_min_max(minimum=1, maximum=2) \
    .cpus(1).memory('1Gi') \
    .sidekick_cpus(model, 4) \
    .sidekick_memory(model, '12Gi') \
    .sidekick_gpus(model, 4) \
    .deployment_label("kubernetes.io/os:linux") \
    .build()

To change the acceleration settings for model deployment, models should be re-uploaded as either a new model or a new model version for maximum compatibility with the hardware infrastructure.

The following demonstrates deploying a generic AI/ML model with the architecture set to QAIC. For this example, the model is deployed with a pre-determined deployment configuration saved to deployment_config.

# create the pipeline
pipeline = wl.build_pipeline("sample_pipeline")

# set the pipeline model step as the model set to the Power10 architecture

pipeline.add_model_step(model)

# deploy the pipeline with the deployment configuration

pipeline.deploy(deployment_configuration)

Wallaroo supports deploying models on edge and multi-cloud environments through publishing the Wallaroo pipeline with the model and deployment configuration to an Open Container Initiative (OCI) registry. These pipeline publishes are deployed on devices with QAIC hardware.

Publish to OCI Registry via the Wallaroo SDK

Wallaroo pipelines are published to an OCI registry with the following elements:

• The ML models added as pipeline steps.
• The Wallaroo inference engine inherited from the model’s acceleration settings - in this case, QAIC.
• The deployment configuration specified in the publish command.

Pipelines are published as images to the edge registry set in the Enable Wallaroo Edge Registry with the wallaroo.pipeline.Pipeline.publish method.

When a pipeline is published, the containerized pipeline with its models, and the inference engine for the architecture and acceleration uploaded to the OCI registry. Once published, the publish is deployed on edge locations with Docker, Podman, or helm based deployments. For more details, see Edge and Multi-cloud Pipeline Publish.

Publish a Pipeline Parameters

The wallaroo.pipeline.Pipeline.publish method takes the following parameters. The containerized pipeline will be pushed to the Edge registry service with the model, pipeline configurations, and other artifacts needed to deploy the pipeline.

Publish a Pipeline Example

The following example shows how to publish a pipeline to the edge registry service associated with the Wallaroo instance for an Edge device with QAIC AI accelerator chips. In this example, the deployment configuration is set with:

• Cpus: 4
• RAM: 12 Gi
• gpus: 4
  • For Wallaroo deployment configurations for QAIC, the gpu parameter specifies the number of System-on-Chips (SoCs) allocated.
• Deployment label: Specifies the node with the gpus.

deployment_config = DeploymentConfigBuilder() \
    .cpus(1).memory('1Gi') \
    .sidekick_cpus(model, 4) \
    .sidekick_memory(model, '12Gi') \
    .sidekick_gpus(model, 4) \
    .deployment_label("kubernetes.io/os:linux") \
    .build()

The pipeline is then published with the `wallaroo.pipeline.Pipeline.publish(deployment_config) command.

pipeline.publish(deployment_config=deployment_config)

Waiting for pipeline publish... It may take up to 600 sec.
....................................................................................... Published.

docker run \
    -p $EDGE_PORT:8080 \
    -e OCI_USERNAME=$OCI_USERNAME \
    -e OCI_PASSWORD=$OCI_PASSWORD \
    -e PIPELINE_URL=us-west1-docker.pkg.dev/wallaroo-dev-253816/testqaic/pipelines/llamaqaicopenaiedge:a0db44db-cc58-437e-9e73-2da3e3ae45e9 \
    -e CONFIG_CPUS=1.0 --cpus=5.0 --memory=13g \
    us-west1-docker.pkg.dev/wallaroo-dev-253816/testqaic/engines/proxy/wallaroo/ghcr.io/wallaroolabs/fitzroy-mini-qaic-vllm:v2025.1.0-6261

podman run \
    -p $EDGE_PORT:8080 \
    -e OCI_USERNAME=$OCI_USERNAME \
    -e OCI_PASSWORD=$OCI_PASSWORD \
    -e PIPELINE_URL=us-west1-docker.pkg.dev/wallaroo-dev-253816/testqaic/pipelines/llamaqaicopenaiedge:a0db44db-cc58-437e-9e73-2da3e3ae45e9 \
    -e CONFIG_CPUS=1.0 --cpus=5.0 --memory=13g \
    us-west1-docker.pkg.dev/wallaroo-dev-253816/testqaic/engines/proxy/wallaroo/ghcr.io/wallaroolabs/fitzroy-mini-qaic-vllm:v2025.1.0-6261

helm install --atomic $HELM_INSTALL_NAME \
    oci://us-west1-docker.pkg.dev/wallaroo-dev-253816/testqaic/charts/llamaqaicopenaiedge \
    --namespace $HELM_INSTALL_NAMESPACE \
    --version 0.0.1-a0db44db-cc58-437e-9e73-2da3e3ae45e9 \
    --set ociRegistry.username=$OCI_USERNAME \
    --set ociRegistry.password=$OCI_PASSWORD

Deploy on Edge Devices

By default, the pipeline publishes include deployment commands for Docker, Podman, or Helm based deployments. The following example cover Docker based deployments. For more details, see Edge and Multi-cloud Deployment and Inference.

Deploying ML Models with Qualcomm QAIC hardware with Intel GPUs in edge and multi-cloud environments via docker run require additional parameters.

For QAIC deployments via docker or podman, additional parameters are required depending on the devices used.

• For All Devices: For all devices on the edge deployment, the parameter --privileged is required. The following example is a sample command deploying a Wallaroo pipeline published in an OCI registry on an edge device with QAIC AI accelerators. For example:

  • Docker example:

    docker run -it -p 8080:8080 \
        -e DEBUG=true \
        -e CONFIG_CPUS=16 \
        -e OCI_USERNAME=oauth2accesstoken \
        -e OCI_PASSWORD=$tok \
        -e PIPELINE_URL=us-west1-docker.pkg.dev/wallaroo-dev-253816/testqaic/pipelines/llamaqaicopenaiedge:a0db44db-cc58-437e-9e73-2da3e3ae45e9 \
        --privileged \
        --cpuset-cpus 0-15 --cpus 16 \
        us-west1-docker.pkg.dev/wallaroo-dev-253816/testqaic/engines/proxy/wallaroo/ghcr.io/wallaroolabs/fitzroy-mini-qaic-vllm:v2025.1.0-6261
    

• For specific devices, each device is specified via the --device parameter. The following example specifies devices accel4 through accel7.

  • Docker Example

    docker run -p 8080:8080 \
    -e CONFIG_CPUS=16 \
    -e OCI_USERNAME=oauth2accesstoken \
    -e OCI_PASSWORD=$tok \
    -e PIPELINE_URL=us-west1-docker.pkg.dev/wallaroo-dev-253816/testqaic/pipelines/llamaqaicopenaiedge:a0db44db-cc58-437e-9e73-2da3e3ae45e9 \
    --device=/dev/accel/accel4 \
    --device=/dev/accel/accel5 \
    --device=/dev/accel/accel6 \
    --device=/dev/accel/accel7 \
    --cpuset-cpus 0-15 \
    --cpus 16 \
    us-west1-docker.pkg.dev/wallaroo-dev-253816/testqaic/engines/proxy/wallaroo/ghcr.io/wallaroolabs/fitzroy-mini-qaic-vllm:v2025.1.0-6261
    

• Deploy Llama with Continuous Batching Using Native vLLM Framework and QAIC AI Acceleration
• Deploy RAG Llama with QAIC
• Deploy Llama with Continuous Batching Using Native vLLM Framework with QAIC using OpenAI Inference
• Deploy RAG Llama with QAIC and OpenAI Inference

The specified model optimization configuration is not available

• If the model acceleration option is set to QAIC, but the architecture is set to an incompatible architecture (aka anything other than X86):
  • The upload, deployment and publish operations fail with the following error message: “The specified model optimization configuration is not available. Please try this operation again using a different configuration or contact Wallaroo at support@wallaroo.ai for questions or help.”