Inference requests sent to Wallaroo are submitted under the inference data schema definitions. The following details those requirements for:
Wallaroo pipelines accept either Apache Arrow tables natively or pandas DataFrames.
wallaroo.pipeline.infer
or wallaroo.pipeline.infer_from_file
, it is a “What you give is what you get” situation: If an Apache Arrow table is submitted for an inference request with the , then an Apache Arrow table is returned. If a pandas DataFrame is submitted, then a pandas DataFrame is returned.Content-Type
and Accept
determine what data type is submitted and what data type is received:application/json; format=pandas-records
application/vnd.apache.arrow.file
Data types for inputs and outputs to inference requests through Wallaroo is based on the Apache Arrow Data Types, with the following exceptions:
null
: Not allowed. All fields must have submitted values that match their data type. For example, if the schema expects a float value, then some value of type float such as 0.0
must be submitted and cannot be None
or Null
. If a schema expects a string value, then some value of type string must be submitted, etc. The exception are BYOP models, which can accept optional inputs.time32
and time64
: Datetime data types must be converted to string.Data submitted to Wallaroo for inference requests have the following data constraints.
Each input row for an inference is related directly to the inference row output.
For example, the INPUT and OUTPUT rows match, with each input row directly corresponding to an output row.
tensor | |
---|---|
0 | [-1.0603297501, 2.3544967095000002, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192000001, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526000001, 1.9870535692, 0.7005485718000001, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756, -0.1466244739, -1.4463212439] |
1 | [-1.0603297501, 2.3544967095000002, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192000001, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526000001, 1.9870535692, 0.7005485718000001, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756, -0.1466244739, -1.4463212439] |
2 | [-1.0603297501, 2.3544967095000002, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192000001, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526000001, 1.9870535692, 0.7005485718000001, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756, -0.1466244739, -1.4463212439] |
3 | [-1.0603297501, 2.3544967095000002, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192000001, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526000001, 1.9870535692, 0.7005485718000001, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756, -0.1466244739, -1.4463212439] |
4 | [0.5817662108, 0.09788155100000001, 0.1546819424, 0.4754101949, -0.19788623060000002, -0.45043448540000003, 0.016654044700000002, -0.0256070551, 0.0920561602, -0.2783917153, 0.059329944100000004, -0.0196585416, -0.4225083157, -0.12175388770000001, 1.5473094894000001, 0.2391622864, 0.3553974881, -0.7685165301, -0.7000849355000001, -0.1190043285, -0.3450517133, -1.1065114108, 0.2523411195, 0.0209441826, 0.2199267436, 0.2540689265, -0.0450225094, 0.10867738980000001, 0.2547179311] |
time | in.tensor | out.dense_1 | anomaly.count | |
---|---|---|---|---|
0 | 2023-11-17 20:34:17.005 | [-1.0603297501, 2.3544967095, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526, 1.9870535692, 0.7005485718, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756, -0.1466244739, -1.4463212439] | [0.99300325] | 0 |
1 | 2023-11-17 20:34:17.005 | [-1.0603297501, 2.3544967095, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526, 1.9870535692, 0.7005485718, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756, -0.1466244739, -1.4463212439] | [0.99300325] | 0 |
2 | 2023-11-17 20:34:17.005 | [-1.0603297501, 2.3544967095, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526, 1.9870535692, 0.7005485718, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756, -0.1466244739, -1.4463212439] | [0.99300325] | 0 |
3 | 2023-11-17 20:34:17.005 | [-1.0603297501, 2.3544967095, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526, 1.9870535692, 0.7005485718, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756, -0.1466244739, -1.4463212439] | [0.99300325] | 0 |
4 | 2023-11-17 20:34:17.005 | [0.5817662108, 0.097881551, 0.1546819424, 0.4754101949, -0.1978862306, -0.4504344854, 0.0166540447, -0.0256070551, 0.0920561602, -0.2783917153, 0.0593299441, -0.0196585416, -0.4225083157, -0.1217538877, 1.5473094894, 0.2391622864, 0.3553974881, -0.7685165301, -0.7000849355, -0.1190043285, -0.3450517133, -1.1065114108, 0.2523411195, 0.0209441826, 0.2199267436, 0.2540689265, -0.0450225094, 0.1086773898, 0.2547179311] | [0.0010916889] | 0 |
Each element must have the same internal data type. For example, the following is valid because all of the data types within each element are float32
.
t = [
[2.35, 5.75],
[3.72, 8.55],
[5.55, 97.2]
]
The following is invalid, as it mixes floats and strings in each element:
t = [
[2.35, "Bob"],
[3.72, "Nancy"],
[5.55, "Wani"]
]
The following inputs are valid, as each data type is consistent within the elements.
df = pd.DataFrame({
"t": [
[2.35, 5.75, 19.2],
[5.55, 7.2, 15.7],
],
"s": [
["Bob", "Nancy", "Wani"],
["Jason", "Rita", "Phoebe"]
]
})
df
t | s | |
---|---|---|
0 | [2.35, 5.75, 19.2] | [Bob, Nancy, Wani] |
1 | [5.55, 7.2, 15.7] | [Jason, Rita, Phoebe] |
Data types for inputs and outputs to inference requests through Wallaroo is based on the Apache Arrow Data Types, with the following exceptions:
null
: Not allowed. All fields must have submitted values that match their data type. For example, if the schema expects a float value, then some value of type float such as 0.0
must be submitted and cannot be None
or Null
. If a schema expects a string value, then some value of type string must be submitted, etc. The exception are BYOP models, which can accept optional inputs.time32
and time64
: Datetime data types must be converted to string.Data submitted to Wallaroo for inference requests have the following data constraints.
Each input row for an inference is related directly to the inference row output.
For example, the INPUT and OUTPUT rows match, with each input row directly corresponding to an output row.
tensor | |
---|---|
0 | [-1.0603297501, 2.3544967095000002, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192000001, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526000001, 1.9870535692, 0.7005485718000001, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756, -0.1466244739, -1.4463212439] |
1 | [-1.0603297501, 2.3544967095000002, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192000001, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526000001, 1.9870535692, 0.7005485718000001, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756, -0.1466244739, -1.4463212439] |
2 | [-1.0603297501, 2.3544967095000002, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192000001, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526000001, 1.9870535692, 0.7005485718000001, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756, -0.1466244739, -1.4463212439] |
3 | [-1.0603297501, 2.3544967095000002, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192000001, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526000001, 1.9870535692, 0.7005485718000001, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756, -0.1466244739, -1.4463212439] |
4 | [0.5817662108, 0.09788155100000001, 0.1546819424, 0.4754101949, -0.19788623060000002, -0.45043448540000003, 0.016654044700000002, -0.0256070551, 0.0920561602, -0.2783917153, 0.059329944100000004, -0.0196585416, -0.4225083157, -0.12175388770000001, 1.5473094894000001, 0.2391622864, 0.3553974881, -0.7685165301, -0.7000849355000001, -0.1190043285, -0.3450517133, -1.1065114108, 0.2523411195, 0.0209441826, 0.2199267436, 0.2540689265, -0.0450225094, 0.10867738980000001, 0.2547179311] |
time | in.tensor | out.dense_1 | anomaly.count | |
---|---|---|---|---|
0 | 2023-11-17 20:34:17.005 | [-1.0603297501, 2.3544967095, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526, 1.9870535692, 0.7005485718, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756, -0.1466244739, -1.4463212439] | [0.99300325] | 0 |
1 | 2023-11-17 20:34:17.005 | [-1.0603297501, 2.3544967095, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526, 1.9870535692, 0.7005485718, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756, -0.1466244739, -1.4463212439] | [0.99300325] | 0 |
2 | 2023-11-17 20:34:17.005 | [-1.0603297501, 2.3544967095, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526, 1.9870535692, 0.7005485718, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756, -0.1466244739, -1.4463212439] | [0.99300325] | 0 |
3 | 2023-11-17 20:34:17.005 | [-1.0603297501, 2.3544967095, -3.5638788326, 5.1387348926, -1.2308457019, -0.7687824608, -3.5881228109, 1.8880837663, -3.2789674274, -3.9563254554, 4.0993439118, -5.6539176395, -0.8775733373, -9.131571192, -0.6093537873, -3.7480276773, -5.0309125017, -0.8748149526, 1.9870535692, 0.7005485718, 0.9204422758, -0.1041491809, 0.3229564351, -0.7418141657, 0.0384120159, 1.0993439146, 1.2603409756, -0.1466244739, -1.4463212439] | [0.99300325] | 0 |
4 | 2023-11-17 20:34:17.005 | [0.5817662108, 0.097881551, 0.1546819424, 0.4754101949, -0.1978862306, -0.4504344854, 0.0166540447, -0.0256070551, 0.0920561602, -0.2783917153, 0.0593299441, -0.0196585416, -0.4225083157, -0.1217538877, 1.5473094894, 0.2391622864, 0.3553974881, -0.7685165301, -0.7000849355, -0.1190043285, -0.3450517133, -1.1065114108, 0.2523411195, 0.0209441826, 0.2199267436, 0.2540689265, -0.0450225094, 0.1086773898, 0.2547179311] | [0.0010916889] | 0 |
Each element must have the same internal data type. For example, the following is valid because all of the data types within each element are float32
.
t = [
[2.35, 5.75],
[3.72, 8.55],
[5.55, 97.2]
]
The following is invalid, as it mixes floats and strings in each element:
t = [
[2.35, "Bob"],
[3.72, "Nancy"],
[5.55, "Wani"]
]
The following inputs are valid, as each data type is consistent within the elements.
df = pd.DataFrame({
"t": [
[2.35, 5.75, 19.2],
[5.55, 7.2, 15.7],
],
"s": [
["Bob", "Nancy", "Wani"],
["Jason", "Rita", "Phoebe"]
]
})
df
t | s | |
---|---|---|
0 | [2.35, 5.75, 19.2] | [Bob, Nancy, Wani] |
1 | [5.55, 7.2, 15.7] | [Jason, Rita, Phoebe] |
The following examples show different inference requests with data sets.
For standard batch inputs, each column and row corresponds to one data input. For example, a set of house data fed into a house price model has the following inputs.
number_of_bedrooms | year_built | number_of_bathrooms | |
---|---|---|---|
0 | 3 | 1990-06-01 | 1 |
1 | 4 | 2023-07-02 | 2.5 |
2 | 3 | 2000-08-15 | 1.5 |
As the data types meet all three data requirements, the model is set with the configuration batch_config=None
, allowing for batch inference submissions. The following represents a inference results from the data above for a house price model, with each output row corresponding to the input row.
time | in.number_of_bedrooms | in.year_built | in.number_of_bathrooms | out.variable | |
---|---|---|---|---|---|
0 | 2023-10-31 16:57:28.974 | 3 | 1990-06-01 | 1 | [137079.4] |
1 | 2023-10-31 16:57:28.974 | 4 | 2023-07-02 | 2.5 | [2002393.6] |
2 | 2023-10-31 16:57:28.974 | 3 | 2000-08-15 | 1.5 | [1967344.1] |
For batch inputs, each inference request is contained in a single row of data. For example, a bike rental model predicts rentals based on previous dates for bikes sold, whether it was a week day, etc. These are submitted as a group of dates that are then used by the model to predict bike rentals on figure dates. For example, this 5 day period is submitted for the inference request.
dteday | site_id | cnt | season | holiday | weekday | workingday | |
---|---|---|---|---|---|---|---|
0 | 2011-02-02 | site0001 | 1240 | 1 | 0 | 3 | 1 |
1 | 2011-02-03 | site0001 | 1551 | 1 | 0 | 4 | 1 |
2 | 2011-02-04 | site0001 | 2324 | 1 | 0 | 5 | 1 |
3 | 2011-02-05 | site0001 | 805 | 1 | 0 | 6 | 0 |
4 | 2011-02-06 | site0001 | 1948 | 1 | 0 | 0 | 0 |
To submit to the Wallaroo deployed pipeline as a batch, each range of dates and values is set into a single row of data. This is reflected in the input and output schemas defined for the model. The pyarrow schema pa.list_
is used to define the data being received as a List per field.
import pyarrow as pa
import wallaroo
wl = wallaroo.Client()
input_schema = pa.schema([
pa.field('dteday', pa.list_(pa.string()) ),
pa.field('site_id', pa.list_(pa.string()) ),
pa.field('cnt', pa.list_(pa.int64()) ),
pa.field('season', pa.list_(pa.int64()) ),
pa.field('holiday', pa.list_(pa.int64()) ),
pa.field('weekday', pa.list_(pa.int64()) ),
pa.field('workingday', pa.list_(pa.int64()) ),
])
output_schema = pa.schema([
pa.field('dteday', pa.list_(pa.string()) ),
pa.field('site_id', pa.list_(pa.string()) ),
pa.field('forecast', pa.list_(pa.int64()) ),
])
arima_model = (wl.upload_model('bikeforecast-arima',
'./models/forecast.py',
framework=Framework.PYTHON)
.configure(runtime="python",
input_schema = input_schema,
output_schema = output_schema
)
)
Here, a month of data for single site is prepared for inference through a Wallaroo deployed model. Each inference request contains in a single row per site. Additional sites would be in their own row, with their month’s worth of data set as lists within the cells.
display(input_frame)
dteday | site_id | cnt | season | holiday | weekday | workingday | |
---|---|---|---|---|---|---|---|
0 | [2011-02-02, 2011-02-03, 2011-02-04, 2011-02-05, 2011-02-06, 2011-02-07, 2011-02-08, 2011-02-09, 2011-02-10, 2011-02-11, 2011-02-12, 2011-02-13, 2011-02-14, 2011-02-15, 2011-02-16, 2011-02-17, 2011-02-18, 2011-02-19, 2011-02-20, 2011-02-21, 2011-02-22, 2011-02-23, 2011-02-24, 2011-02-25, 2011-02-26, 2011-02-27, 2011-02-28, 2011-03-01, 2011-03-02, 2011-03-03, 2011-03-04, 2011-03-05, 2011-03-06, 2011-03-07, 2011-03-08] | [site0001, site0001, site0001, site0001, site0001, site0001, site0001, site0001, site0001, site0001, site0001, site0001, site0001, site0001, site0001, site0001, site0001, site0001, site0001, site0001, site0001, site0001, site0001, site0001, site0001, site0001, site0001, site0001, site0001, site0001, site0001, site0001, site0001, site0001, site0001] | [1240, 1551, 2324, 805, 1948, 1650, 913, 931, 1256, 1614, 1000, 1883, 1964, 2036, 2586, 3219, 3947, 1826, 1418, 723, 1281, 2564, 2181, 1539, 2059, 2428, 836, 1235, -1, -1, -1, -1, -1, -1, -1] | [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] | [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] | [3, 4, 5, 6, 0, 1, 2, 3, 4, 5, 6, 0, 1, 2, 3, 4, 5, 6, 0, 1, 2, 3, 4, 5, 6, 0, 1, 2, 3, 4, 5, 6, 0, 1, 2] | [1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 0, 0, 1, 1] |
The inference results are then converted to whatever format the data scientist and ML engineers require.
time | out.dteday | out.forecast | out.site_id | anomaly.count | |
---|---|---|---|---|---|
0 | 2023-09-22 00:15:20.214 | [2011-03-02, 2011-03-03, 2011-03-04, 2011-03-05, 2011-03-06, 2011-03-07, 2011-03-08] | [2269, 1712, 1795, 1371, 1819, 2045, 1974] | [site0001, site0001, site0001, site0001, site0001, site0001, site0001] | 0 |
Wallaroo inference requests support either pandas DataFrame or Apache Arrows as inputs. When using a numpy array with multi-dimensional shapes, the Wallaroo SDK provides helper functions for data conversion to support the Wallaroo data requirements.
Note that when using the Wallaroo MLOps with multi-dimensional numpy arrays, the helper functions are contained in the Wallaroo SDK to assist with data conversion.
Inference requests made through the Wallaroo SDK accept Numpy arrays as input values within a pandas DataFrame. For Apache Arrow tables, the numpy array must be flattened.
The Wallaroo SDK method wallaroo.utils.flatten_np_array_columns(df, col)
converts a pandas DataFrame column specified with the name col
. The pandas DataFrame is then converted to an Apache Arrow table for inferencing.
The following example deploys a Computer Vision Mobilenet model. A multi-dimensional numpy array is generated from an image, then used it directly for inference request in a DataFrame.
import wallaroo
import torch
import cv2
wl = wallaroo.Client()
# upload the model
mobilenet_model = wl.upload_model("mobilenet-demo",
'./models/mobilenet.pt.onnx',
framework=Framework.ONNX).configure(batch_config="single")
# build the pipeline with the model and deploy
pipeline = wl.build_pipeline("cv-sample-pipeline")
pipeline.add_model_step(mobilenet_model)
# deployed the pipeline
pipeline.deploy()
# convert the image to a tensor
image = cv2.imread("./data/images/input/example/dairy_bottles.png")
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
tensor = torch.FloatTensor(image)
# get the numpy value and use in a dataframe
npArray = tensor.cpu().numpy()
df_with_numpy = pd.DataFrame({"tensor":[npArray]})
The following is an example of the the first row and the first 4 columns from the image converted into a multi-dimensional numpy array.
tensors | |
---|---|
0 | [[0.9372549, 0.9529412, 0.9490196], [0.93333334, 0.9490196, 0.9490196], [0.9254902, 0.9529412, 0.94509804], [0.91764706, 0.9490196, 0.9490196]] |
The DataFrame with the multi-dimensional numpy array is submitted to the Wallaroo pipeline in an inference request. The following isolates the first 4 objects detected and overall confidence value of all detected objects.
infResults = pipeline.infer(df_with_numpy, timeout=300)
df = pd.DataFrame([
{"time": infResults.loc[0, ['time']][0],
'out.avg_conf': [infResults.loc[0, ['out.avg_conf']][0]],
'boxes': [infResults.loc[0, ['out.boxes']][0][0:4][0:4]],
'classes': [infResults.loc[0, ['out.classes']][0][0:4]],
'confidences': [infResults.loc[0, ['out.confidences']][0][0:4]]
}
])
display(df)
time | out.avg_conf | boxes | classes | confidences | |
---|---|---|---|---|---|
0 | 2024-02-07 15:22:21.497 | [0.2895055] | [[[0.0, 210.2901, 85.26464, 479.07495], [72.037796, 197.3227, 151.44221, 468.43225], [211.28014, 184.72838, 277.21924, 420.42746], [143.23906, 203.83008, 216.85551, 448.8881]]] | [[44, 44, 44, 44]] | [[0.98649, 0.9011532, 0.60778517, 0.59223205]] |
Apache Arrow tables submitted for inference request must have multi-dimensional arrays flattened before they are submitted. The following example shows flattening the numpy array with the wallaroo.utils.flatten_np_array_columns(df, col)
method, which returns the column’s values as flattened arrays.
import wallaroo
import torch
import cv2
wl = wallaroo.Client()
# upload the model
mobilenet_model = wl.upload_model("mobilenet-demo",
'./models/mobilenet.pt.onnx',
framework=Framework.ONNX).configure(batch_config="single")
# build the pipeline with the model and deploy
pipeline = wl.build_pipeline("cv-sample-pipeline")
pipeline.add_model_step(mobilenet_model)
# deployed the pipeline
pipeline.deploy()
# convert the image to a tensor
image = cv2.imread("./data/images/input/example/dairy_bottles.png")
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
tensor = torch.FloatTensor(image)
# get the numpy value and use in a dataframe
npArray = tensor.cpu().numpy()
df_with_numpy = pd.DataFrame({"tensor":[npArray]})
The following shows the the first 3 columns of the first 4 rows of the image file converted to tensors as a multi-dimensional numpy array in a pandas DataFrame.
tensors | |
---|---|
0 | [[0.9372549, 0.9529412, 0.9490196], [0.93333334, 0.9490196, 0.9490196], [0.9254902, 0.9529412, 0.94509804], [0.91764706, 0.9490196, 0.9490196]] |
Before submitting as an Apache Arrow table, the multi-dimensional arrays are flattened.
The following demonstrates using the wallaroo.utils.flatten_np_array_columns(df, col)
takes a pandas DataFrame, and the column name of the arrays to flatten method. For this example, we will flatten the column and retrieve a new array from it.
wallaroo.utils.flatten_np_array_columns(df_with_numpy, 'tensor')
0 [0.9372549, 0.9529412, 0.9490196, 0.94509804, 0.94509804, 0.94509804 ...]
Name: tensor, dtype: object
We create an Apache Arrow table from the flattened numpy array and submit it for the inference request.
table = pa.table([
pa.array(wallaroo.utils.flatten_np_array_columns(df_with_numpy, 'tensor'))
], names=["tensor"])
display(table)
pyarrow.Table
tensor: list<item: float>
child 0, item: float
----
tensor: [[[0.9372549,0.9529412,0.9490196,0.94509804,...]]]
result = pipeline.infer(table)
display(result['out.avg_confidence'])
<pyarrow.lib.ChunkedArray object at 0x2a69b4b30>
[
[
[
0.3588039
]
]
]
The Wallaroo MLOps API supports the following Content-Type
and Accept
headers values:
application/json; format=pandas-records
: JSON in pandas Record format.application/vnd.apache.arrow.file
: A binary in Apache Arrow format.In both cases, multi-dimensional numpy array must be flattened before submitting it as part of an API Inference Request.
The following examples demonstrates using the Wallaroo SDK wallaroo.utils.flatten_np_array_columns(df, col)
method to flatten a column in a pandas DataFrame that contains multi-dimensional numpy arrays.
The following is an example of a DataFrame with an image converted to a multi-dimensional numpy array of tensors.
tensors | |
---|---|
0 | [[0.9372549, 0.9529412, 0.9490196], [0.93333334, 0.9490196, 0.9490196], [0.9254902, 0.9529412, 0.94509804], [0.91764706, 0.9490196, 0.9490196]] |
Before submitting as an MLOps API request as a pandas DataFrame, the multi-dimensional arrays are flattened. The following demonstrates using the wallaroo.utils.flatten_np_array_columns(df, col)
method. The flattened array is inserted as the tensor
column for a new pandas DataFrame.
df_with_flattened_array = pd.DataFrame({'tensor': wallaroo.utils.flatten_np_array_columns(df_with_numpy, 'tensor')})
# show only the first 5 tensors
df = df_with_flattened_array.apply(lambda x: [i[0:5] for i in x])
display(df)
tensor | |
---|---|
0 | [0.9372549, 0.9529412, 0.9490196, 0.94509804, 0.94509804] |
With the multi-dimensional arrays flattened, it is submitted as an inference request.
headers = wl.auth.auth_header()
headers['Content-Type'] = 'application/json; format=pandas-records'
deploy_url = pipeline._deployment._url()
response = requests.post(
deploy_url,
headers=headers,
data=df_with_flattened_array.to_json(orient="records")
)
display(pd.DataFrame(response.json()).loc[:, ['time', 'out.avg_confidence']])
time | out | |
---|---|---|
0 | 1705687869292 | {‘avg_confidence’: [0.3588039]} |
The following is an example of a DataFrame with an image converted to a multi-dimensional numpy array of tensors.
tensors | |
---|---|
0 | [[0.9372549, 0.9529412, 0.9490196], [0.93333334, 0.9490196, 0.9490196], [0.9254902, 0.9529412, 0.94509804], [0.91764706, 0.9490196, 0.9490196]] |
Before submitting as an MLOps API request as an Apache Arrow table, the multi-dimensional arrays are flattened. The following demonstrates using the wallaroo.utils.flatten_np_array_columns(df, col)
method.
wallaroo.utils.flatten_np_array_columns(df_with_numpy, 'tensor')
0 [0.9372549, 0.9529412, 0.9490196, 0.94509804, 0.94509804, 0.94509804 ...]
Name: tensor, dtype: object
With the multi-dimensional arrays flattened, it is inserted into an Apache Arrow table and submitted in an inference request.
table = pa.table([
pa.array(wallaroo.utils.flatten_np_array_columns(df_with_numpy, 'tensor'))
], names=["tensor"])
display(table)
pyarrow.Table
tensor: list<item: float>
child 0, item: float
----
tensor: [[[0.9372549,0.9529412,0.9490196,0.94509804,...]]]
headers = wl.auth.auth_header()
# set Content-Type type
headers['Content-Type']='application/vnd.apache.arrow.file'
# set accept as apache arrow table
headers['Accept']="application/vnd.apache.arrow.file"
deploy_url = pipeline._deployment._url()
response = requests.post(
deploy_url,
headers=headers,
data=table,
verify=True
)
# Arrow table is retrieved
with pa.ipc.open_file(response.content) as reader:
arrow_table = reader.read_all()
display(result['out.avg_confidence'])
<pyarrow.lib.ChunkedArray object at 0x2a69b4b30>
[
[
[
0.3588039
]
]
]
Other examples come from the Wallaroo Tutorials as listed below.