LLM Deploy

The following tutorials demonstrate deploying different LLMs with Wallaroo.

Llamacpp Deploy on IBM Power10 Tutorial

How to deploy and publish Llamacpp LLMs on the IBM Power10 Architecture

IBM Granite 8B Code Instruct Large Language Model (LLM) with GPU Tutorial

Deploy Llama with OpenAI Compatibility

Deploy RAG LLM with OpenAI Compatibility

LLM Deploy on ARM Tutorial

LLM Deploy on GPU Tutorial

LLM Deploy on x86 Tutorial

Managed Inference Endpoints

Deploy with QAIC AI Acceleration

Inference Endpoint API Spec for OpenAI Compatibility Enabled vLLM Tutorial

How to use the Wallaroo SDK to create the inference endpoint API spec.