How to Monitor LLMs

LLMs deployed through Wallaroo are deployed through the following methods:

  • In-Line Validation: Implemented as an in-line model step in the same Wallaroo Pipeline as the deployed LLM.
  • LLM Monitoring: A set of LLM listeners that score the LLM against a set of standard and user customized metrics.

For access to these sample models and a demonstration on using LLMs with Wallaroo:

LLM Monitoring Overview
LLM Validation Listeners

LLM Monitoring Listeners