wallaroo.dynamic_batching_config

class DynamicBatchingConfig:

Model configuration for dynamic batching

DynamicBatchingConfig( max_batch_delay_ms: int = 10, batch_size_target: int = 4, batch_size_limit: Optional[int] = None)

Initialize the DynamicBatchingConfig object.

Attributes: max_batch_delay_ms (int): Maximum amount of time we will wait before sending a batch to the model for inference batch_size_target (int): Minimum size of a batch we will send to the model. batch_size_limit (int, optional): Maximum size of a batch that the model can process. Defaults to None.

max_batch_delay_ms

Set the maximum batch delay in milliseconds.

Parameters
  • max_batch_delay_ms: Maximum batch delay in milliseconds.
Raises
  • ValueError: If value is not an integer or is less than or equal to 0.
batch_size_target

Set the target batch size.

Parameters
  • batch_size_target: Target batch size.
Raises
  • ValueError: If value is not an integer or is less than or equal to 0.
batch_size_limit

Set the batch size limit.

Parameters
  • batch_size_limit: Batch size limit.
Raises
  • ValueError: If value is not an integer or is less than or equal to 0.
def to_json(self):

Convert the DynamicBatchingConfig object to a JSON object.

Returns

JSON representation of the DynamicBatchingConfig object.

@classmethod
def from_dict(cls, config_dict):

Create a DynamicBatchingConfig object from a dictionary.

Parameters
  • config_dict: Dictionary containing the configuration values.
Returns

DynamicBatchingConfig: DynamicBatchingConfig object created from the dictionary.