AI/ML models can be deployed in centralized Wallaroo OPs instances and Edge devices on a variety of infrastructures and processors. The CPU infrastructure is set during the model upload and packaging stage.

Models specified with the Power10 architecture during the upload and automated model packaging can be deployed on Wallaroo Ops instances or multicloud deployments.

Power10 Support

For details on using Power10 with Wallaroo and setting up a demonstration:

• Contact your Wallaroo Support Representative OR
• Schedule Your Wallaroo.AI Demo Today

Model Packaging and Deployments Prerequisites for Power10

To upload and package a model for Wallaroo Ops or multicloud edge deployments, the following prerequisites must be met.

• Wallaroo Ops
  • At least one Power10 node deployed in the cluster.
• Edge Devices
  • Enable Edge Registry Services in the Wallaroo instance to publish the pipeline to an OCI (Open Container Initiative) registry for edge deployments.
  • Power10 processor support for the edge device.

AI Workloads for Power10 via the Wallaroo SDK

The Wallaroo SDK provides Power10 support for models uploaded for Wallaroo Ops or multicloud edge deployments.

Upload Models for Power10 via the Wallaroo SDK

Models are uploaded to Wallaroo via the wallaroo.client.upload_model method. The infrastructure is set with the optional arch parameter, which accepts the wallaroo.engine_config.Architecture object.

wallaroo.client.upload_model has the following parameters. For more details on model uploads, see Automated Model Packaging.

Upload Model for Power10 Architecture Example

The following demonstrates uploading a model for deployment on the Power10 Architecture.

import wallaroo

# set the Wallaroo client
wl = wallaroo.Client()

# upload the model and save the reference to a variable
power10_model = wl.upload_model(
    name="sample_model",
    path=model_file_path,
    framework=framework, # the wallaroo.framework.Framework
    input_schema = input_schema, # input schema in PyArrow Schema format
    output_schema = output_schema, # input schema in PyArrow Schema format
    arch = wallaroo.engine_config.Architecture.Power10
)

Deploy Models for Power10 via the Wallaroo SDK

Models are added to pipeline as pipeline steps. Models are then deployed through the wallaroo.pipeline.Pipeline.deploy(deployment_config: Optional[wallaroo.deployment_config.DeploymentConfig] = None) method.

For full details, see Pipeline Deployment Configuration.

When deploying a model in a Wallaroo Ops instance, the deployment configurations inherits the model architecture setting. No additional changes are needed to set the architecture when deploying the model. Other settings, such as the number of CPUs, etc can be changed without modifying the architecture setting.

To change the architecture settings for model deployment, models should be re-uploaded as either a new model or a new model version for maximum compatibility with the hardware infrastructure. For more information on uploading models or new model versions, see Upload Models for Power10 via the Wallaroo SDK.

The following demonstrates deploying a generic AI/ML model with the architecture set to Power10. For this example, the model is deployed with a pre-determined deployment configuration saved to deployment_config.

# create the pipeline
pipeline = wl.build_pipeline("sample_pipeline")

# set the pipeline model step as the model set to the Power10 architecture

pipeline.add_model_step(power10_model)

# deploy the pipeline with the deployment configuration

pipeline.deploy(deployment_configuration)

Publish Pipeline for Power10 via the Wallaroo SDK

Publishing the pipeline to uses the method wallaroo.pipeline.publish(deployment_config: Optional[wallaroo.deployment_config.DeploymentConfig]).

This requires that the Wallaroo Ops instance have Edge Registry Services enabled.

A deployment configuration must be included with the pipeline publish, even if no changes to the cpus, memory, etc are made. For more detail on deployment configurations, see Pipeline Deployment Configuration.

The deployment configuration for the pipeline publish inherits the model’s architecture. Options such as the number of cpus, amount of memory, etc can be adjusted without impacting the model’s architecture settings.

Pipelines do not need to be deployed in the Wallaroo Ops instance before publishing the pipeline.

For more information, see Wallaroo SDK Essentials Guide: Pipeline Edge Publication.

The following demonstrates deploying the generic model uploaded earlier.

# default deployment configuration
publish = pipeline.publish(deployment_config=wallaroo.DeploymentConfigBuilder().build())
display(publish)

Waiting for pipeline publish... It may take up to 600 sec.
Pipeline is publishing................ Published.

docker run \
    -p $EDGE_PORT:8080 \
    -e OCI_USERNAME=$OCI_USERNAME \
    -e OCI_PASSWORD=$OCI_PASSWORD \
    -e PIPELINE_URL=[registry.example.com/uat/pipelines/architecture-demonstration-Power10:fd5e3d64-9eea-492d-92b2-8bdb5b20ec83](https://registry.example.com/uat/pipelines/sample_pipeline:d033152c-494c-44a6-8981-627c6b6ad72e) \
    -e CONFIG_CPUS=1 registry.example.com/uat/engines/proxy/wallaroo/ghcr.io/wallaroolabs/fitzroy-mini-ppc64le:v2024.4.0-main

helm install --atomic $HELM_INSTALL_NAME \
    oci://registry.example.com/uat/charts/architecture-demonstration-Power10 \
    --namespace $HELM_INSTALL_NAMESPACE \
    --version 0.0.1-fd5e3d64-9eea-492d-92b2-8bdb5b20ec83 \
    --set ociRegistry.username=$OCI_USERNAME \
    --set ociRegistry.password=$OCI_PASSWORD

Tutorials

The following tutorials demonstrate deploying models on the Power10 architecture.

• Llamacpp Deploy on IBM Power10 Tutorial
• Run Anywhere: Deploy and Publish Computer Vision Model Resnet50 for IBM Power10
• Run Anywhere: Deploy and Publish Computer Vision Model Resnet50 with Post Processing for IBM Power10