We are pleased to announce the following product improvements in our 2024.4 release:

• Air-gapped Kubernetes Cluster Support: Wallaroo is installable on air-gapped Kubernetes clusters. This gives organizations the ability to run AI workloads in a fully isolated environment without external connections (including to the internet), and ensure compliance with the highest levels of privacy & security protocoles for their data and infrastrucutre.
  • Air-Gapped Cluster Installation
  • Guides:
    • Wallaroo Install for Air-Gapped General Guide
    • Wallaroo Install for Air-Gapped AWS EKS Procedure
• LLMs optimizations with Continuous Batching: Continuous batching provides LLM deployments with NVIDIA GPUs additional optimizations by dynamically grouping incoming requests in real time. This allows LLM-based and agentic AI applications increased flexibility to maximize performance and hardware utilization.
  • Continuous Batching for LLMs
  • Tutorials:
    • Continuous Batching for Llama 3.1 8B with vLLM
    • Continuous Batching for Custom Llama with vLLM
• Edge and Multi-cloud Pipeline Publish Updates: The Wallaroo SDK provides enhancements to edge and multi-cloud publishing methods. These provide developers increased information on the pipeline publish sources, edges associated with the publish, version details, and role based filters.
  • Edge and Multi-cloud Pipeline Publish
  • Edge and Multi-cloud Tutorials