wallaroo.dynamic_batching_config
class
DynamicBatchingConfig:
Model configuration for dynamic batching
DynamicBatchingConfig( max_batch_delay_ms: int = 10, batch_size_target: int = 4, batch_size_limit: Optional[int] = None)
Initialize the DynamicBatchingConfig object.
Attributes: max_batch_delay_ms (int): Maximum amount of time we will wait before sending a batch to the model for inference batch_size_target (int): Minimum size of a batch we will send to the model. batch_size_limit (int, optional): Maximum size of a batch that the model can process. Defaults to None.