Motion and Obstruction Target Tracking

LInk to images: https://imgur.com/a/R8Xt4

Student

• [3563490 UNB] Jason Wuertz
• [3569383 UNB] Jawad Refai

Introduction

Objective

Creating a program that can play and solve the cup game [1]. This will require identifying and tracking a moving object (the target cup) through obstructions (the other cups).

Material and Methods

Data

Our data source will be a live video feed from a webcam or similar. Figure 1 shows an example image that we would expect to process.

Assumptions and Prerequisites

For our project implementation, testing, and demonstrations, we will assume the following about our set-up:

1. Consistent Lighting

   • Brightness
   • Color

2. Constant Objects

   • 3 Identical Cups
   • No other objects expected to enter the scene

3. Minimal Obstruction

   • We expect nothing big to come between the camera and the game set up
   • Handling and Moving of the cups to be done with minimal interference by hands

4. Consistent Camera Distance and Elevation

5. Objects (Cups) Never Picked Up or Rotated

Methodology

We will be using C# to complete this project.

1. We already set up a base project replicating the Python and C++ examples provided with the OpenCV Starter Kit on LEA
2. Using the Xamarin IDE
3. Official tutorials and examples will help us implement what we need step by step such as Shape Detection, etc.

One possible solution we have thought of is as follows:

1. Measure the height of every cup and sort them into an ordered list (if they are the same size then order does not matter).
2. Apply arbitrary labels to the items in the array ( e.g. cups A, B and C)
3. Continue to measure the height of cups, when two cups switch positions in the array then update the labels.
4. When obstruction occurs mark which cup has disappeared.
5. When occlusion is over match the height of the emerging cup to the list of numbers and thus find it's label.

Another solution would instead calculate depth based on the difference between the bottom of the cups as shown here:

Tasks

1. Scene Setup

Assignee - Jason

Resulting setup requires 1 light, 1 green filter, 3 cups, 1 box or similar. Currently we have green sheet taped to the inside of a box in which the cups will be. The green sheet is also used to cover a LED aquarium lamp in order to apply a slight green hue to the scene. This helps in creating a uniform background and increasing the contrast between the red cups and the green background. The camera and light must be perpendicular to the box and on the same vertical axis.

Description

• Physically set up the scene for the cups game.
• We need 3 identical brightly colored cups (Preferably Red)
• Flat environment with a plain high contrast background (Preferably Green)
• Lighting the scene minimizing shadows and glare

Tests

• Using our intended video capture device (Laptop Camera), check the final output when placed in front of the scene.
• All cups must be clearly visible
• There should be enough space between the camera and background for cups to move freely

Risks

• If the scene is not set up correctly, the image produced may be too difficult to analyze.
• Materials of the objects/background could be too reflective
• Unwanted shadows could make shape extraction harder

2. C# Project + OpenCV Setup

Assignee - Jawad

Description

• We need to set up a working C# project and successfully connect it with OpenCV
• At the very least we should have a simple video steam video using the library

Tests

• Replicate the Python/C++ examples provided
• One window with drawn shapes
• One window with camera view and an image overlay

Risks

• The examples provided are written in a different language,
• The C# implementation of OpenCV is written differently and as such translating example code may not be easy

3. HSV Color Space Filtering

Assignee - Jason

Description Post processing task to separate the cups from the background

Tests

• Manual inspection and discussion
• Object detection results

Risks

• Minimum OpenCV HSV Color space prevents optimal filter creation

4. Identify objects in still images

Assignee - Jawad

Descriptions

• Using Shape Detection or similar to detect and label the cups

Tests

• Run the code on an image taken from a video of the cups scene detailed earlier
• We would expect a correct count depending no how many cups are on screen

Risks

• Since the cups are not a simple geometric shape, we may have trouble adapting Shape Detection examples to work optimally.
• Hands coming into frame may cause issues.

5. Identify objects in video feed or video files

Assignee - TBA

Descriptions

• Once we've established object extraction from the previous task, we will need to have it happen constantly and accurately on every frame of a video feed.

Tests

• Run the code using direct video feed from the camera in front of the scene
• Move the cups around to see if the object count remains the same
• Move one or more cups out of the frame to see if the count changes as planned

Risks

• Limited knowledge of the OpenCV library could lead to an imperfect implementation

6. Track objects

Assignee - TBA

Descriptions

• By this point we should have successfully detected objects in the scene
• Now we need to label them and track their movements

Tests

• Display a label on each cup in the scene
• Move the cups around and the label should move with them

Risks

• Inconsistent lighting or other environmental changes could prevent accurate tracking.
• Lack of unique features in the objects could prevent accurate tracking.

7. Track objects with simple obstruction

Assignee - TBA

Descriptions

• The same as task 6, but we should see accurate re-labeling when cups go in and out of view

Tests

• Move the cups in and out of frame
• Move the cups behind each other

Risks

• If simple tracking does not work this will not either.
• Possible inaccuracies could prevent us from using either methods suggested above which will require more brainstorming.

8. Track a specific target throughout a video session

Assignee - TBA

Descriptions

• Now we will attempt to track one cup accurate during an session
• This cup will have the ball under it
• If this is successful, we will have the ball holding cup labeled correctly at the end

Tests

• Assign one cup to be the one with the ball
• Move cups around and in the end compare the labels with actual ball position

Risks

• Cups may be relabeled incorrectly if more than one go out of frame