Wallaroo Inference Server Tutorial: Computer Vision Faster R-CNN

A demonstration of using the Computer Vision Faster R-CNN model with Wallaroo Inference Server.

The following tutorial is available on the Wallaroo Github Repository.

Wallaroo Inference Server: Faster R-CNN

This notebook is used in conjunction with the Wallaroo Inference Server Free Edition for the Computer Vision resnet 50 machine learning model. This provides a free license for performing inferences through the Computer Vision Faster R-CNN machine learning model. For full demonstrations of this model, see Wallaroo Use Case Tutorials.

Prerequisites

A deployed Wallaroo Inference Server Free Edition with one of the following options:
- Wallaroo.AI Faster R-CNN - x64
- Wallaroo.AI Faster R-CNN - GPU
Access via port 8080 to the Wallaroo Inference Server Free Edition.

Note that GPU inference server require a VM with Nvidia GPU cuda support.

Computer Vision Faster R-CNN Model Schemas

Inputs

The Faster R-CNN Model takes the following inputs.

Field	Type	Description
`tensor`	Float	Tensor in the shape (n, 3, 480, 640) float. This is the normalized pixel values of the 640x480 color image.

Outputs

Field	Type	Description
`boxes`	Variable length List[Float]	The bounding boxes of detected objects with each 4 number sequence representing (x_coordinate, y_coordinate, width, height). List length is `4*n` where n is the number of detected objects.
`classes`	Variable length List[Int]	Integer values representing the categorical classes that are predicted by the model. List length is `n` where `n` is the number of detected objects.
`confidences`	Variable length List[Float]	The confidence of detected classes. List length is `n` where `n` is the number of detected objects.

Wallaroo Inference Server API Endpoints

The following HTTPS API endpoints are available for Wallaroo Inference Server.

Pipelines Endpoint

Endpoint: HTTPS GET /pipelines
Returns:
- List of pipelines with the following fields.
  - id (String): The name of the pipeline.
  - status (String): The pipeline status. Running indicates the pipeline is available for inferences.

Pipeline Endpoint Example

The following demonstrates using curl to retrieve the Pipelines endpoint. Replace the HOSTNAME with the address of your Wallaroo Inference Server.

!curl HOSTNAME:8080/pipelines

{"pipelines":[{"id":"frcnn","status":"Running"}]}

Models Endpoint

Endpoint: GET /models
Returns:
- List of models with the following fields.
  - name (String): The name of the model.
  - sha (String): The sha hash of the model.
  - status (String): The model status. Running indicates the models is available for inferences.
  - version (String): The model version in UUID format.

Models Endpoint Example

The following demonstrates using curl to retrieve the Models endpoint. Replace the HOSTNAME with the address of your Wallaroo Inference Server.

!curl HOSTNAME:8080/models

{"models":[{"name":"frcnn","sha":"ee606dc9776a1029420b3adf59b6d29395c89d1d9460d75045a1f2f152d288e7","status":"Running","version":"0762d591-7d31-4738-8394-2a148d00fbdc"}]}

Inference Endpoint

Endpoint: HTTPS POST /pipelines/frcnn
Headers:
- Content-Type: application/vnd.apache.arrow.file: For Apache Arrow tables.
- Content-Type: application/json; format=pandas-records: For pandas DataFrame in record format.

Input Parameters: DataFrame in /pipelines/hf-summarizer-standard OR Apache Arrow table in application/vnd.apache.arrow.file with the following inputs:

tensor (Float Required): The tensor shape is a variable array in the shape (3, {picture width}, {picture height}) float of the the normalized pixel values of the 640x480 color image. For example, a 1x1 image renders:

[
  {
    "tensor": [
        [
            [
                [0.9372549057]
            ], 
            [
                [0.9372549057]
            ], 
            [
                [0.8666666746]
            ]
        ]
    ]
  }
]

The following code is used to convert an image into a 640x480 DataFrame with the appropriate shape for the model:

import cv2
import torch
import numpy as np

def imageResize(image, width, height):

    im_pillow = np.array(image)
    image = cv2.cvtColor(im_pillow, cv2.COLOR_RGB2BGR)

    self.debug("Resizing to w:"+str(width) + " height:"+str(height))
    image = cv2.resize(image, (width, height)) 
    resizedImage = image.copy()

    # convert the image from BGR to RGB channel ordering and change the
    # image from channels last to channels first ordering
    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
    image = image.transpose((2, 0, 1))

    # add the batch dimension, scale the raw pixel intensities to the
    # range [0, 1], and convert the image to a floating point tensor
    image = np.expand_dims(image, axis=0)
    image = image / 255.0
    tensor = torch.FloatTensor(image)
    return tensor, resizedImage

Returns:
- Headers
  - Content-Type: application/json; format=pandas-records: pandas DataFrame in record format.
- Data
  - check_failures (List[Integer]): Whether any validation checks were triggered. For more information, see Wallaroo SDK Essentials Guide: Pipeline Management: Anomaly Testing.
  - elapsed (List[Integer]): A list of time in nanoseconds for:
    - [0] The time to serialize the input.
    - [1…n] How long each step took.
  - model_name (String): The name of the model used.
  - model_version (String): The version of the model in UUID format.
  - original_data: The original input data. Returns null if the input may be too long for a proper return.
  - outputs (List): The outputs of the inference result separated by data type. The number of arrays for each field is determined by the number of detected objects.
    - Float: The bounding boxes for each detected object.
      - data (List[Float]): The bounding boxes data in the shape returned in the dim field.
      - dim (List[Integer]): The dimension shape returned in the format [number of objects, 4].
      - v (Integer): The vector shape of the data.
    - Int64: The class of each detected object.
      - data (List[Integer]): The class results in the shape of the dim field.
      - dim (List[Integer]): The dimension shape returned in the format [number of objects].
      - v (Integer): The vector shape of the data.
    - Float: The confidences of each detected object.
      - data (List[Float]): The confidence values in the shape of the dim field.
      - dim (List[Integer]): The dimension shape returned in the format [number of objects].
      - v (Integer): The vector shape of the data.
  - pipeline_name (String): The name of the pipeline.
  - shadow_data: Any shadow deployed data inferences in the same format as outputs.
  - time (Integer): The time since UNIX epoch.

Inference Endpoint Example

The following example performs an inference using the Apache Arrow table input ./data/image_224x224.arrow from an image converted into a tensor for inferencing.

!curl -X POST HOSTNAME:8080/pipelines/frcnn \
    -H "Content-Type:application/vnd.apache.arrow.file" \
    --data-binary @./data/test_table.arrow

[{"check_failures":[],"elapsed":[62382994,3894096061],"model_name":"frcnn","model_version":"0762d591-7d31-4738-8394-2a148d00fbdc","original_data":null,"outputs":[{"Float":{"data":[2.1511011123657227,193.98316955566406,76.26535034179688,475.4029846191406,610.822509765625,98.60633087158203,639.8867797851562,232.27053833007812,544.2866821289062,98.7265396118164,581.2883911132812,230.20494079589844,454.9934387207031,113.0856704711914,484.7846374511719,210.12820434570312,502.5888671875,331.87664794921875,551.2268676757812,476.4918212890625,538.5425415039062,292.1205139160156,587.4655151367188,468.1288146972656,578.5416870117188,99.70755767822266,617.2246704101562,233.57081604003906,548.552001953125,191.84564208984375,577.3058471679688,238.4773712158203,459.8332824707031,344.297119140625,505.42633056640625,456.7117919921875,483.4716796875,110.56584930419922,514.0936279296875,205.00155639648438,262.1221923828125,190.36659240722656,323.49029541015625,405.2057800292969,511.6675109863281,104.53833770751953,547.0171508789062,228.23663330078125,75.39196014404297,205.6231231689453,168.49893188476562,453.44085693359375,362.5065612792969,173.1685791015625,398.6695556640625,371.8243103027344,490.4246826171875,337.62701416015625,534.1234130859375,461.0242004394531,351.3855895996094,169.14898681640625,390.75830078125,244.0699005126953,525.1982421875,291.7389831542969,570.5552978515625,417.6438903808594,563.4224243164062,285.3888854980469,609.3085327148438,452.2594299316406,425.579345703125,366.2491455078125,480.6353454589844,474.5400085449219,154.53799438476562,198.03770446777344,227.64283752441406,439.8441162109375,597.0289306640625,273.6045837402344,637.2067260742188,439.0321350097656,473.88763427734375,293.419921875,519.7537231445312,349.23040771484375,262.7759704589844,192.0358123779297,313.3096008300781,258.3465881347656,521.1492919921875,152.8902587890625,534.859619140625,246.52365112304688,389.8963317871094,178.07867431640625,431.87554931640625,360.5932312011719,215.99900817871094,179.52967834472656,280.2846984863281,421.9092102050781,523.6453857421875,310.7387390136719,560.3648681640625,473.5797119140625,151.71310424804688,191.4107666015625,228.7101287841797,443.3218688964844,0.507830798625946,14.856098175048828,504.5198059082031,405.7276916503906,443.83685302734375,340.1248779296875,532.83740234375,475.77716064453125,472.37847900390625,329.13092041015625,494.0364685058594,352.5906066894531,572.41455078125,286.2613220214844,601.86767578125,384.58990478515625,532.7720947265625,189.8910369873047,551.902587890625,241.760498046875,564.0308837890625,105.75121307373047,597.0350952148438,225.32579040527344,551.2584838867188,287.16033935546875,590.9205932617188,405.7154846191406,70.46804809570312,0.39822694659233093,92.78654479980469,84.401123046875,349.4453430175781,3.618438959121704,392.6148376464844,98.43362426757812,64.40483856201172,215.1493377685547,104.09456634521484,436.5079650878906,615.121826171875,269.4668273925781,633.3085327148438,306.0345153808594,238.31851196289062,0.7395721673965454,290.289794921875,91.30622863769531,449.37347412109375,337.3955383300781,480.132080078125,369.35125732421875,74.95623016357422,191.84234619140625,164.2128448486328,457.0014343261719,391.9664611816406,6.255006790161133,429.2305603027344,100.72328186035156,597.4866333007812,276.6980895996094,618.0615234375,298.6277770996094,384.5116882324219,171.95826721191406,407.0126953125,205.28720092773438,341.5733947753906,179.80580139160156,365.8834533691406,208.57888793945312,555.0277709960938,288.626953125,582.6162109375,358.0912780761719,615.9203491210938,264.926513671875,632.3316040039062,280.25518798828125,297.9515380859375,0.5227981805801392,347.18743896484375,95.13105773925781,311.648681640625,203.67933654785156,369.6169128417969,392.58062744140625,163.1035614013672,0.0,227.6746826171875,86.4968490600586,68.51898956298828,1.870926022529602,161.25877380371094,82.89816284179688,593.6093139648438,103.26359558105469,617.1240234375,200.96546936035156,263.3114929199219,200.12203979492188,275.6990051269531,234.26516723632812,592.228515625,279.66064453125,619.7049560546875,379.147705078125,597.7548828125,269.654296875,618.5473022460938,286.25213623046875,478.0430603027344,204.3616485595703,530.0074462890625,239.35195922851562,501.34527587890625,289.280029296875,525.7659912109375,333.00262451171875,462.4776916503906,336.9205627441406,491.3201599121094,358.18914794921875,254.40383911132812,203.89566040039062,273.6617736816406,237.1314239501953,307.9604187011719,154.70947265625,440.4544982910156,386.8805847167969,195.53915405273438,187.13592529296875,367.65179443359375,404.9109802246094,77.52113342285156,2.93235182762146,160.15235900878906,81.59642028808594,577.6480102539062,283.400390625,601.4359130859375,307.4188537597656,516.6387329101562,129.7450408935547,540.4093627929688,242.17572021484375,543.2536010742188,253.3868865966797,631.3576049804688,466.62567138671875,271.13580322265625,45.97062683105469,640.0,456.8823547363281,568.7720336914062,188.66448974609375,595.8660278320312,235.05397033691406,400.99700927734375,169.88531494140625,419.3564453125,185.80076599121094,473.9580993652344,328.0773620605469,493.2315979003906,339.41436767578125,602.985107421875,376.3950500488281,633.742431640625,438.6175842285156,480.9596252441406,117.62994384765625,525.0882568359375,242.7487030029297,215.71177673339844,194.8744659423828,276.59906005859375,414.3832092285156,462.2966003417969,331.9463806152344,493.1417541503906,347.9764709472656,543.9497680664062,196.90623474121094,556.0431518554688,238.9914093017578,386.82049560546875,446.6134033203125,428.8106384277344,480.0000305175781,152.22515869140625,0.0,226.9856414794922,85.44817352294922,616.9181518554688,246.15548706054688,630.8656005859375,273.4844665527344,576.3336181640625,100.67465209960938,601.962890625,183.6100616455078,260.9930419921875,202.7347869873047,319.4052429199219,304.1719970703125,333.5813903808594,169.7592315673828,390.9659118652344,278.0940246582031,286.2676696777344,3.7005159854888916,319.3962097167969,88.7715835571289,532.5545043945312,209.36183166503906,552.5841674804688,240.0896453857422,572.9991455078125,207.65208435058594,595.6241455078125,235.4663543701172,6.274436950683594,0.5994095206260681,96.0499267578125,81.27410888671875,203.32298278808594,188.94265747070312,227.27435302734375,254.29417419433594,513.7689208984375,154.48574829101562,531.3786010742188,245.21615600585938,547.5381469726562,369.9145202636719,583.13916015625,470.3020935058594,316.64532470703125,252.806640625,336.654541015625,272.8283386230469,234.79965209960938,0.0003524780331645161,298.05706787109375,92.01449584960938,523.0662841796875,195.2969207763672,534.4849853515625,242.2367401123047,467.7620849609375,202.35621643066406,536.2362060546875,245.5334014892578,146.07208251953125,0.9752746820449829,178.93894958496094,83.21017456054688,18.780662536621094,1.0261597633361816,249.98724365234375,88.49466705322266,600.9063720703125,270.4198303222656,625.3632202148438,297.9408264160156,556.4094848632812,287.8470153808594,580.2647094726562,322.64727783203125,80.67057037353516,161.22784423828125,638.6259765625,480.0000305175781,455.983154296875,334.66290283203125,480.167236328125,349.97784423828125,613.990478515625,267.98065185546875,635.9580688476562,375.177490234375,345.7596435546875,2.459097385406494,395.9118347167969,99.68698120117188],"dim":[100,4],"v":1}},{"Int64":{"data":[44,44,44,44,44,44,44,44,44,44,44,44,44,44,44,44,44,44,44,44,44,44,44,44,44,44,44,86,82,44,44,44,44,44,44,84,84,44,44,44,44,86,84,44,44,44,44,44,84,44,44,84,44,44,44,44,51,44,44,44,44,44,44,44,44,44,82,44,44,44,44,44,86,44,44,1,84,44,44,44,44,84,47,47,84,14,44,44,53,84,47,47,44,84,44,44,82,44,44,44],"dim":[100],"v":1}},{"Float":{"data":[0.9965357780456543,0.9883402585983276,0.9700246453285217,0.9696425795555115,0.9647804498672485,0.9603756666183472,0.9542890191078186,0.9467539191246033,0.946523904800415,0.9448496699333191,0.9361184239387512,0.9165345430374146,0.9133635759353638,0.8874814510345459,0.8440590500831604,0.8255265355110168,0.8232693076133728,0.8174006938934326,0.7956525087356567,0.7866907715797424,0.7731487154960632,0.7519370317459106,0.7360916137695312,0.7009188532829285,0.6932355165481567,0.650771975517273,0.6324360966682434,0.5787761807441711,0.5023477673530579,0.5016375780105591,0.44628554582595825,0.4280437231063843,0.4253791868686676,0.39086252450942993,0.3683646619319916,0.34732428193092346,0.32950618863105774,0.3105373680591583,0.290763258934021,0.28558316826820374,0.2668001651763916,0.26302841305732727,0.2544434070587158,0.2456863522529602,0.23536619544029236,0.2332199662923813,0.22612977027893066,0.22483190894126892,0.22332394123077393,0.2144296020269394,0.20122256875038147,0.1975485235452652,0.19439265131950378,0.1908392459154129,0.18713772296905518,0.17646068334579468,0.1662893146276474,0.16326230764389038,0.14825181663036346,0.13694529235363007,0.12920643389225006,0.1281532198190689,0.12235746532678604,0.12128996849060059,0.11628169566392899,0.11498632282018661,0.11184822022914886,0.110161192715168,0.10950638353824615,0.10391511023044586,0.10385661572217941,0.09757373481988907,0.09632062166929245,0.0955762192606926,0.09159914404153824,0.09062063694000244,0.08262358605861664,0.08223513513803482,0.07993929833173752,0.07989177107810974,0.0787586122751236,0.0782012939453125,0.07737930119037628,0.07690244168043137,0.07593464851379395,0.07503405958414078,0.07482589781284332,0.06898099929094315,0.0684114620089531,0.06764157116413116,0.06575051695108414,0.06490862369537354,0.061884183436632156,0.060101211071014404,0.05788714438676834,0.05717653036117554,0.05661648511886597,0.05601711571216583,0.054582688957452774,0.053669560700654984],"dim":[100],"v":1}}],"pipeline_name":"frcnn","shadow_data":{},"time":1696526041661}]

Next: Wallaroo Inference Server Tutorial: Computer Vision Resnet