Computer Vision Tutorials

This tutorial and the assets can be downloaded as part of the Wallaroo Tutorials repository.

Step 00: Introduction and Setup

This tutorial demonstrates how to use the Wallaroo to detect objects in images through the following models:

• rnn mobilenet: A single stage object detector that performs fast inferences. Mobilenet is typically good at identifying objects at a distance.
• resnet50: A dual stage object detector with slower inferencing but but is able to detect objects that are closer to each other.

This tutorial series will demonstrate the following:

• How to deploy a Wallaroo pipeline with trained rnn mobilenet model and perform sample inferences to detect objects in pictures, then display those objects.
• How to deploy a Wallaroo pipeline with a trained resnet50 model and perform sample inferences to detect objects in pictures, then display those objects.
• Use the Wallaroo feature shadow deploy to have both models perform inferences, then select the inference result with the higher confidence and show the objects detected.

This tutorial assumes that users have installed the Wallaroo SDK or are running these tutorials from within their Wallaroo instance’s JupyterHub service.

This demonstration should be run within a Wallaroo JupyterHub instance for best results.

Prerequisites

The included OpenCV class is included in this demonstration as CVDemoUtils.py, and requires the following dependencies:

• ffmpeg
• libsm
• libxext

Internal JupyterHub Service

To install these dependencies in the Wallaroo JupyterHub service, use the following commands from a terminal shell via the following procedure:

1. Launch the JupyterHub Service within the Wallaroo install.

2. Select File->New->Terminal.

3. Enter the following:

   sudo apt-get update
   

   sudo apt-get install ffmpeg libsm6 libxext6  -y
   

External SDK Users

For users using the Wallaroo SDK to connect with a remote Wallaroo instance, the following commands will install the required dependancies:

For Linux users, this can be installed with:

sudo apt-get update
sudo apt-get install ffmpeg libsm6 libxext6  -y

MacOS users can prepare their environments using a package manager such as Brew with the following:

brew install ffmpeg libsm libxext

Libraries and Dependencies

1. This repository may use large file sizes for the models. Use the Wallaroo Tutorials Releases to download a .zip file of the most recent computer vision tutorial that includes the models.
2. Import the following Python libraries into your environment:
   1. torch
   2. wallaroo
   3. torchvision
   4. opencv-python
   5. onnx
   6. onnxruntime
   7. imutils
   8. pytz
   9. ipywidgets

These can be installed by running the command below in the Wallaroo JupyterHub service. Note the use of pip install torch --no-cache-dir for low memory environments.

!pip install torchvision
!pip install torch --no-cache-dir
!pip install opencv-python
!pip install onnx
!pip install onnxruntime
!pip install imutils
!pip install pytz
!pip install ipywidgets

Collecting torchvision
  Downloading torchvision-0.18.1-cp38-cp38-macosx_11_0_arm64.whl.metadata (6.6 kB)
Requirement already satisfied: numpy in /Users/johnhansarick/.virtualenvs/wallaroosdk2024.2/lib/python3.8/site-packages (from torchvision) (1.22.3)
Collecting torch==2.3.1 (from torchvision)
  Downloading torch-2.3.1-cp38-none-macosx_11_0_arm64.whl.metadata (26 kB)
Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /Users/johnhansarick/.virtualenvs/wallaroosdk2024.2/lib/python3.8/site-packages (from torchvision) (10.4.0)
Collecting filelock (from torch==2.3.1->torchvision)
  Downloading filelock-3.15.4-py3-none-any.whl.metadata (2.9 kB)
Requirement already satisfied: typing-extensions>=4.8.0 in /Users/johnhansarick/.virtualenvs/wallaroosdk2024.2/lib/python3.8/site-packages (from torch==2.3.1->torchvision) (4.12.2)
Collecting sympy (from torch==2.3.1->torchvision)
  Downloading sympy-1.13.1-py3-none-any.whl.metadata (12 kB)
Collecting networkx (from torch==2.3.1->torchvision)
  Downloading networkx-3.1-py3-none-any.whl.metadata (5.3 kB)
Collecting jinja2 (from torch==2.3.1->torchvision)
  Downloading jinja2-3.1.4-py3-none-any.whl.metadata (2.6 kB)
Collecting fsspec (from torch==2.3.1->torchvision)
  Downloading fsspec-2024.6.1-py3-none-any.whl.metadata (11 kB)
Collecting MarkupSafe>=2.0 (from jinja2->torch==2.3.1->torchvision)
  Using cached MarkupSafe-2.1.5-cp38-cp38-macosx_10_9_universal2.whl.metadata (3.0 kB)
Collecting mpmath<1.4,>=1.1.0 (from sympy->torch==2.3.1->torchvision)
  Downloading mpmath-1.3.0-py3-none-any.whl.metadata (8.6 kB)
Downloading torchvision-0.18.1-cp38-cp38-macosx_11_0_arm64.whl (1.6 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.6/1.6 MB 8.5 MB/s eta 0:00:00a 0:00:01
[?25hDownloading torch-2.3.1-cp38-none-macosx_11_0_arm64.whl (61.0 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.0/61.0 MB 24.5 MB/s eta 0:00:0000:0100:01
[?25hDownloading filelock-3.15.4-py3-none-any.whl (16 kB)
Downloading fsspec-2024.6.1-py3-none-any.whl (177 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 177.6/177.6 kB 12.7 MB/s eta 0:00:00
[?25hDownloading jinja2-3.1.4-py3-none-any.whl (133 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 133.3/133.3 kB 8.2 MB/s eta 0:00:00
[?25hDownloading networkx-3.1-py3-none-any.whl (2.1 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 25.9 MB/s eta 0:00:0000:0100:01
[?25hDownloading sympy-1.13.1-py3-none-any.whl (6.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.2/6.2 MB 26.3 MB/s eta 0:00:0000:0100:01
[?25hUsing cached MarkupSafe-2.1.5-cp38-cp38-macosx_10_9_universal2.whl (18 kB)
Downloading mpmath-1.3.0-py3-none-any.whl (536 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 536.2/536.2 kB 19.3 MB/s eta 0:00:00
[?25hInstalling collected packages: mpmath, sympy, networkx, MarkupSafe, fsspec, filelock, jinja2, torch, torchvision
Successfully installed MarkupSafe-2.1.5 filelock-3.15.4 fsspec-2024.6.1 jinja2-3.1.4 mpmath-1.3.0 networkx-3.1 sympy-1.13.1 torch-2.3.1 torchvision-0.18.1
Requirement already satisfied: torch in /Users/johnhansarick/.virtualenvs/wallaroosdk2024.2/lib/python3.8/site-packages (2.3.1)
Requirement already satisfied: filelock in /Users/johnhansarick/.virtualenvs/wallaroosdk2024.2/lib/python3.8/site-packages (from torch) (3.15.4)
Requirement already satisfied: typing-extensions>=4.8.0 in /Users/johnhansarick/.virtualenvs/wallaroosdk2024.2/lib/python3.8/site-packages (from torch) (4.12.2)
Requirement already satisfied: sympy in /Users/johnhansarick/.virtualenvs/wallaroosdk2024.2/lib/python3.8/site-packages (from torch) (1.13.1)
Requirement already satisfied: networkx in /Users/johnhansarick/.virtualenvs/wallaroosdk2024.2/lib/python3.8/site-packages (from torch) (3.1)
Requirement already satisfied: jinja2 in /Users/johnhansarick/.virtualenvs/wallaroosdk2024.2/lib/python3.8/site-packages (from torch) (3.1.4)
Requirement already satisfied: fsspec in /Users/johnhansarick/.virtualenvs/wallaroosdk2024.2/lib/python3.8/site-packages (from torch) (2024.6.1)
Requirement already satisfied: MarkupSafe>=2.0 in /Users/johnhansarick/.virtualenvs/wallaroosdk2024.2/lib/python3.8/site-packages (from jinja2->torch) (2.1.5)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in /Users/johnhansarick/.virtualenvs/wallaroosdk2024.2/lib/python3.8/site-packages (from sympy->torch) (1.3.0)
Requirement already satisfied: opencv-python in /Users/johnhansarick/.virtualenvs/wallaroosdk2024.2/lib/python3.8/site-packages (4.10.0.84)
Requirement already satisfied: numpy>=1.21.0 in /Users/johnhansarick/.virtualenvs/wallaroosdk2024.2/lib/python3.8/site-packages (from opencv-python) (1.22.3)
Collecting onnx
  Downloading onnx-1.16.1-cp38-cp38-macosx_11_0_universal2.whl.metadata (16 kB)
Requirement already satisfied: numpy>=1.20 in /Users/johnhansarick/.virtualenvs/wallaroosdk2024.2/lib/python3.8/site-packages (from onnx) (1.22.3)
Collecting protobuf>=3.20.2 (from onnx)
  Downloading protobuf-5.27.2-cp38-abi3-macosx_10_9_universal2.whl.metadata (592 bytes)
Downloading onnx-1.16.1-cp38-cp38-macosx_11_0_universal2.whl (16.5 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 16.5/16.5 MB 20.6 MB/s eta 0:00:0000:0100:01
[?25hDownloading protobuf-5.27.2-cp38-abi3-macosx_10_9_universal2.whl (412 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 412.2/412.2 kB 15.6 MB/s eta 0:00:00
[?25hInstalling collected packages: protobuf, onnx
Successfully installed onnx-1.16.1 protobuf-5.27.2
Collecting onnxruntime
  Downloading onnxruntime-1.18.1-cp38-cp38-macosx_11_0_universal2.whl.metadata (4.3 kB)
Collecting coloredlogs (from onnxruntime)
  Downloading coloredlogs-15.0.1-py2.py3-none-any.whl.metadata (12 kB)
Collecting flatbuffers (from onnxruntime)
  Downloading flatbuffers-24.3.25-py2.py3-none-any.whl.metadata (850 bytes)
Requirement already satisfied: numpy<2.0,>=1.21.6 in /Users/johnhansarick/.virtualenvs/wallaroosdk2024.2/lib/python3.8/site-packages (from onnxruntime) (1.22.3)
Requirement already satisfied: packaging in /Users/johnhansarick/.virtualenvs/wallaroosdk2024.2/lib/python3.8/site-packages (from onnxruntime) (24.1)
Requirement already satisfied: protobuf in /Users/johnhansarick/.virtualenvs/wallaroosdk2024.2/lib/python3.8/site-packages (from onnxruntime) (5.27.2)
Requirement already satisfied: sympy in /Users/johnhansarick/.virtualenvs/wallaroosdk2024.2/lib/python3.8/site-packages (from onnxruntime) (1.13.1)
Collecting humanfriendly>=9.1 (from coloredlogs->onnxruntime)
  Downloading humanfriendly-10.0-py2.py3-none-any.whl.metadata (9.2 kB)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in /Users/johnhansarick/.virtualenvs/wallaroosdk2024.2/lib/python3.8/site-packages (from sympy->onnxruntime) (1.3.0)
Downloading onnxruntime-1.18.1-cp38-cp38-macosx_11_0_universal2.whl (15.9 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 15.9/15.9 MB 21.4 MB/s eta 0:00:0000:0100:01
[?25hDownloading coloredlogs-15.0.1-py2.py3-none-any.whl (46 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 46.0/46.0 kB 4.3 MB/s eta 0:00:00
[?25hDownloading flatbuffers-24.3.25-py2.py3-none-any.whl (26 kB)
Downloading humanfriendly-10.0-py2.py3-none-any.whl (86 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 86.8/86.8 kB 12.0 MB/s eta 0:00:00
[?25hInstalling collected packages: flatbuffers, humanfriendly, coloredlogs, onnxruntime
Successfully installed coloredlogs-15.0.1 flatbuffers-24.3.25 humanfriendly-10.0 onnxruntime-1.18.1
Collecting imutils
  Downloading imutils-0.5.4.tar.gz (17 kB)
  Preparing metadata (setup.py) ... [?25ldone
[?25hBuilding wheels for collected packages: imutils
  Building wheel for imutils (setup.py) ... [?25ldone
[?25h  Created wheel for imutils: filename=imutils-0.5.4-py3-none-any.whl size=25837 sha256=111c266b79db0d15e154d4625eebb04cb07a278486c927ad22ec40e4b4e2745f
  Stored in directory: /Users/johnhansarick/Library/Caches/pip/wheels/59/1b/52/0dea905f8278d5514dc4d0be5e251967f8681670cadd3dca89
Successfully built imutils
Installing collected packages: imutils
Successfully installed imutils-0.5.4
Requirement already satisfied: pytz in /Users/johnhansarick/.virtualenvs/wallaroosdk2024.2/lib/python3.8/site-packages (2024.1)
Collecting ipywidgets
  Downloading ipywidgets-8.1.3-py3-none-any.whl.metadata (2.4 kB)
Requirement already satisfied: comm>=0.1.3 in /Users/johnhansarick/.virtualenvs/wallaroosdk2024.2/lib/python3.8/site-packages (from ipywidgets) (0.2.2)
Requirement already satisfied: ipython>=6.1.0 in /Users/johnhansarick/.virtualenvs/wallaroosdk2024.2/lib/python3.8/site-packages (from ipywidgets) (7.24.1)
Requirement already satisfied: traitlets>=4.3.1 in /Users/johnhansarick/.virtualenvs/wallaroosdk2024.2/lib/python3.8/site-packages (from ipywidgets) (5.14.3)
Collecting widgetsnbextension~=4.0.11 (from ipywidgets)
  Downloading widgetsnbextension-4.0.11-py3-none-any.whl.metadata (1.6 kB)
Collecting jupyterlab-widgets~=3.0.11 (from ipywidgets)
  Downloading jupyterlab_widgets-3.0.11-py3-none-any.whl.metadata (4.1 kB)
Requirement already satisfied: setuptools>=18.5 in /Users/johnhansarick/.virtualenvs/wallaroosdk2024.2/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (69.3.0)
Requirement already satisfied: jedi>=0.16 in /Users/johnhansarick/.virtualenvs/wallaroosdk2024.2/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (0.19.1)
Requirement already satisfied: decorator in /Users/johnhansarick/.virtualenvs/wallaroosdk2024.2/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (5.1.1)
Requirement already satisfied: pickleshare in /Users/johnhansarick/.virtualenvs/wallaroosdk2024.2/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (0.7.5)
Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /Users/johnhansarick/.virtualenvs/wallaroosdk2024.2/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (3.0.47)
Requirement already satisfied: pygments in /Users/johnhansarick/.virtualenvs/wallaroosdk2024.2/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (2.18.0)
Requirement already satisfied: backcall in /Users/johnhansarick/.virtualenvs/wallaroosdk2024.2/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (0.2.0)
Requirement already satisfied: matplotlib-inline in /Users/johnhansarick/.virtualenvs/wallaroosdk2024.2/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (0.1.7)
Requirement already satisfied: pexpect>4.3 in /Users/johnhansarick/.virtualenvs/wallaroosdk2024.2/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (4.9.0)
Requirement already satisfied: appnope in /Users/johnhansarick/.virtualenvs/wallaroosdk2024.2/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets) (0.1.4)
Requirement already satisfied: parso<0.9.0,>=0.8.3 in /Users/johnhansarick/.virtualenvs/wallaroosdk2024.2/lib/python3.8/site-packages (from jedi>=0.16->ipython>=6.1.0->ipywidgets) (0.8.4)
Requirement already satisfied: ptyprocess>=0.5 in /Users/johnhansarick/.virtualenvs/wallaroosdk2024.2/lib/python3.8/site-packages (from pexpect>4.3->ipython>=6.1.0->ipywidgets) (0.7.0)
Requirement already satisfied: wcwidth in /Users/johnhansarick/.virtualenvs/wallaroosdk2024.2/lib/python3.8/site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython>=6.1.0->ipywidgets) (0.2.13)
Downloading ipywidgets-8.1.3-py3-none-any.whl (139 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 139.4/139.4 kB 1.9 MB/s eta 0:00:00 0:00:01
[?25hDownloading jupyterlab_widgets-3.0.11-py3-none-any.whl (214 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 214.4/214.4 kB 4.7 MB/s eta 0:00:00a 0:00:01
[?25hDownloading widgetsnbextension-4.0.11-py3-none-any.whl (2.3 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.3/2.3 MB 13.7 MB/s eta 0:00:00a 0:00:01
[?25hInstalling collected packages: widgetsnbextension, jupyterlab-widgets, ipywidgets
Successfully installed ipywidgets-8.1.3 jupyterlab-widgets-3.0.11 widgetsnbextension-4.0.11

The rest of the tutorials will rely on these libraries and applications, so finish their installation before running the tutorials in this series.

Models for Wallaroo Computer Vision Tutorials

In order for the wallaroo tutorial notebooks to run properly, the videos directory must contain these models in the models directory.

To download the Wallaroo Computer Vision models, use the following link:

https://storage.googleapis.com/wallaroo-public-data/cv-demo-models/cv-retail-models.zip

Unzip the contents into the directory models.

Directory contents

• coco_classes.pickle - contain the 80 COCO classifications used by resnet50 and mobilenet object detectors.
• frcnn-resent.pt - PyTorch resnet50 model
• frcnn-resnet.pt.onnx - PyTorch resnet50 model converted to onnx
• mobilenet.pt - PyTorch mobilenet model
• mobilenet.pt.onnx - PyTorch mobilenet model converted to onnx