Wallaroo Inference Server Free Edition

Table of Contents

Wallaroo Inference Server Free Edition provides a fast, low cost way to inference on pre-trained and deployed models on the Wallaroo engine. The Wallaroo Inference Server for these models is provided at no charge; users pay for their infrastructure costs.

The following models are supported.

• Computer Vision Faster R-CNN
• Computer Vision Resnet
• Computer Vision YOLOv8n
• Computer Vision U-Net
• Hugging Face Summarization
• Llama 2

Wallaroo Inference Server Free Edition provides a fast, low cost way to inference on pre-trained and deployed models on the Wallaroo engine. The Wallaroo Inference Server for these models is provided at no charge; users pay for their infrastructure costs.

Installation Procedure

To install a Wallaroo Inference Server from the Azure Marketplace, complete the following steps. The installation steps are broken up by each Azure installation section.

The settings below are the minimum viable options for setting up a Wallaroo Inference Server for Azure Marketplace on an Azure Virtual Machine (VM) in the least amount of steps and time. See full Microsoft Azure VM settings, see the Virtual machines in Azure documentation.

Prerequisites

Have a Microsoft Azure account and access to the Microsoft Azure marketplace.

Overview

From the Overview page, select the following:

• Plan: Select which Wallaroo Inference Server Free Edition plan. These include:
  • Wallaroo.AI HF Summarizer Standard - x64
  • Wallaroo.AI HF Summarizer Standard - GPU x64
  • Wallaroo.AI ResNet50 Inference Server - x64
  • Wallaroo.AI ResNet50 Inference Server - Arm64
  • Wallaroo.AI Yolo v8 Inference Server - x64
  • Wallaroo.AI Yolo v8 Inference Server - GPU
  • Wallaroo.AI Faster R-CNN - x64
  • Wallaroo.AI Faster R-CNN - GPU
  • Wallaroo.AI Llama Inference Server - GPU

For most installations, select Start with a pre-set configuration. To fully customize, select Create. The steps below assume that Start with a pre-set configuration was selected.

Note that GPU inference server require a VM with Nvidia GPU cuda support.

Pre-set Configuration

Most installations of the Wallaroo Inference Server Free Edition performs on minimum hardware. Use te settings Dev/Test with Compute optimized (F-Series) for a low cost option.

Select Continue to create a VM.

Basic

The Basics tab sets the name, region, and other settings.

The following options are the minimum recommended options. Modify as per your organization’s requirements.

• Project Details: Specify how the resource and costs are managed.
  • Subscription: Select the Azure subscription that manages the deployed resources.
  • Resource group: Select the resource group that will manage the resource.
• Instance Details: Specify the VM resources used.
  • Virtual Machine Name: Set the name of the virtual machine.
  • Region: Set the region.
  • Availability Options: Set to No infrastructure redundancy required.
  • Security type: Set to Standard.
  • Image: Do not change - this is auto-selected for the appropriate image for the Wallaroo Inference Server Free Edition.
  • VM architecture: Leave as x64.
  • Size: Select the best size for your organization. A recommended minimum size is Standard_F4s_v2 which provides 4 vcpus with 8 GiB RAM.
• Administrator account: Set the administrative options for this VM.
  • Authentication type: Select the best option for your organization. The default option SSH public key is sufficient.
  • Username: Set a username. The default is azureuser.
  • SSH public key source: Select whether to use an existing key pair or generate a new one. NOTE: If generated, the SSH key will be available for downloading and authenticating via ssh. Store this key in a secure location.

Disks

Keep the default settings and select Next: Networking.

Networking

Keep the default settings and select Next: Management.

Management

If Basic->Availability Options is set to No infrastructure redundancy required, set Enable Automanage to Disabled.

Select Next: Monitoring.

Monitoring

Keep the default settings and select Next: Advanced.

Advanced

Keep the default settings and select Next: Tags.

Tags

Keep the default settings and select Next: Review + create.

Review + create

Verify the details, and select Create.

The process will take 5-10 minutes to complete the process.

Post Installation Steps

The following steps are optional, but recommended to make accessing the new Wallaroo Inference Server.

Host Name Addition

To specify a host name for the Wallaroo Inference Server:

1. From the Microsoft Azure dashboard, access the Virtual machines dashboard. Select the virtual machine hosting the Wallaroo Inference Server.
2. From the Overview page, select DNS Name.
3. From the Settings->Configuration screen, set the DNS name label (optional) field. By default the host name of the virtual machine will be the label plus the region. For example, if the label is sample-hf-wallaroo and the region is westus2, then the host name will be sample-hf-wallaroo.westus2.cloudapp.azure.com.
   1. If the host name is not unique, an error will be displayed.
4. Select Save to store the settings.

The Wallaroo Inference Server will be available from the specified host name.

Supplemental Material

• Wallaroo Inference Server Free Edition Tutorials for tutorials on performing inferences and other requests.