wallaroo.dynamic_batching_config

class DynamicBatchingConfig:

Model configuration for dynamic batching

DynamicBatchingConfig( max_batch_delay_ms: int = 10, batch_size_target: int = 4, batch_size_limit: Optional[int] = None)

Initialize the DynamicBatchingConfig object.

Attributes: max_batch_delay_ms (int): Maximum amount of time we will wait before sending a batch to the model for inference batch_size_target (int): Minimum size of a batch we will send to the model. batch_size_limit (int, optional): Maximum size of a batch that the model can process. Defaults to None.

max_batch_delay_ms

Get the maximum batch delay in milliseconds.

batch_size_target

Get the target batch size.

batch_size_limit

Get the batch size limit.

def to_json(self):

Convert the DynamicBatchingConfig object to a JSON object.

Returns

JSON representation of the DynamicBatchingConfig object.

@classmethod
def from_dict(cls, config_dict):

Create a DynamicBatchingConfig object from a dictionary.

Parameters
  • config_dict: Dictionary containing the configuration values.
Returns

DynamicBatchingConfig: DynamicBatchingConfig object created from the dictionary.