We are pleased to announce the following product improvements in our 2025.1 release:

• LLM deployment on QAIC: Wallaroo supports Qualcomm QAIC, providing GenAI/LLM deployment on Qualcomm’s AI 100 accelerator cards for optimized performance and lower costs.
  • Feature Documentation:
    • LLM Inference with Qualcomm QAIC
    • Inference on Qualcomm QAIC AI Acceleration
  • Tutorials:
    • Deploy Llama with Continuous Batching Using Native vLLM Framework and QAIC Acceleration
    • Deploy RAG Llama with QAIC
    • Deploy Llama with Continuous Batching Using Native vLLM Framework with QAIC using OpenAI Inference
    • Deploy RAG Llama with OpenAI compatibility on QAIC
• LLM token streaming with OpenAI compatibility deployment: Wallaroo provides OpenAI compatibility for improved interactive token streaming user experiences with LLM-based applications while taking advantage of Wallaroo’s ability to maximize throughput and optimizing latency. Additionally with OpenAI compatibility, AI developers can seamlessly migrate their applications from OpenAI endpoints to Wallaroo on-prem endpoints, in connected and air-gapped environments, without losing any functionality.
  • Feature Documentation:
    • Deploy LLMs with OpenAI Compatibility
  • Tutorials:
    • Deploy Llama with OpenAI Compatibility
    • Deploy RAG LLM with OpenAI Compatibility