Scan The Code

Note: Current program is modified from codes made by Raymond

Table of Contents

• What is the task?
• Prerequisites
• How to execute the program
• Explanation of the code
• What still needs to be done

What is the task?

As preparation for the RobotX 2020 Challenge, the task, Scan the Code, is to identify a sequence of colors in real time and successfully record and report the color sequence to the template and project it onto the WAMV console for inspection. We will be completing this task by subscribing to the Neural Net for the coordinates of the light box of the buoy and we will detect the colors using OpenCV, a real-time computer-vision library that is applicable with Python.

Prerequisites/Installation

• Linux operating system with Ubuntu installed or by Dual Booting
• Python 2.7
• OpenCV via command line sudo apt-get install python-opencv

How to execute the program

• After downloading the prerequisites, you should be operating on Ubuntu.
• Download scanthecode.py and the sample images into one folder.
• Open the terminal using Ctrl Alt T
• Change directory into the folder you have created for scanthecode.py

• Run the code in the command line python2 scanthecode.py

• The colors should be displayed in a pop up window

Explanation of the code

import cv2 This import contains opencv libraries which is used to process the colors.

import sys This import statement provides the necessary functions to run Python.

import numpy as np Numpy provides array objects and functions to modify and process these arrays when computing in Python.

import matplotlib.pyplot as plt
import matplotlib.image as mpimg

matplotlib.pyplot and matplotlib.image are the necessary libraries to read and modify the images.

lower_red = np.array([0,50,50])
upper_red = np.array([10,255,255])
lower_blue = np.array([115, 50, 50])
upper_blue = np.array([130,255,255])
lower_yellow = np.array([20,100,100])
upper_yellow = np.array([30,255,255])
lower_green = np.array([90,46,112])
upper_green = np.array([98,250,250])

Using np, arrays are made to construct lower and upper hue/saturation/value threshold for each color.

color_codes = {"red": (0, 0, 255), "blue": (255,0,0), "green": (0, 255, 0), "yellow": (0, 255, 255), "black": (0,0,0)}

The global variable color_codes holds the finalized color to be printed.

def main(argv) function

colors = list(range(1, 6)) This creates an empty array named colors to hold scanned colors.

images = ["i1.png", "i2.png", "i4.png", "i3.png", "i5.png"] This array will hold the name of images. Currently this is hardcoded as placeholders, when a subscriber is established to grab the image Neural Net uses to compute the coordinates of the light box on the buoy.

x1 = [205, 266, 216, 226, 233]
x2 = [236, 284, 233, 244, 256]
y1 = [128, 127, 115, 155, 125]
y2 = [162, 171, 158, 200, 163]

These four variables hold the two x and y variables, each holding the four coordinates of each image. Currently this is hardcoded as placeholders, when a subscriber is established for the Neural Net, the coordinates published by the Neural Net will be placed in.

for i in range(len(images)):
   colors[i] = scan_color(images[i], x1[i], x2[i], y1[i], y2[i])

This for loop runs through all the images using the scan_color function, passing in the image and respective coordinates, and records the color returned into the array colors

#Deletable code
print colors[0]
print colors[1] 
print colors[2]
print colors[3]
print colors[4]

These 5 prints lines prints the BGR values of the color identified for the respective image.

plot(colors) This line displays the color sequence to the console by calling plot function.

def plot(scanned) function

• scanned is an array, (should) consist of name of colors

report = cv2.imread('report.png') Using cv2 imread function for the program to use the report.png template.

    cv2.rectangle(report, (40, 200), (285, 450), color_codes[scanned[0]], thickness = -1)
    cv2.rectangle(report, (355, 200), (610, 450), color_codes[scanned[1]], thickness = -1)
    cv2.rectangle(report, (680, 200), (950, 450), color_codes[scanned[2]], thickness = -1)

Using cv2 rectangle function to fill in the rectangles on to the template (report) with the color that was scanned.

    cv2.putText(report, scanned[0], (120,150), cv2.FONT_HERSHEY_SIMPLEX, 1, (0,0,0), 2);
    cv2.putText(report, scanned[1], (450,150), cv2.FONT_HERSHEY_SIMPLEX, 1, (0,0,0), 2);
    cv2.putText(report, scanned[2], (780,150), cv2.FONT_HERSHEY_SIMPLEX, 1, (0,0,0), 2);

Using cv2 putText function to put the name of the color on to the template (report) in the box above the colors.

    cv2.namedWindow('Scan the Code', cv2.WINDOW_NORMAL)
    cv2.moveWindow('Scan the Code', 200,650)
    cv2.resizeWindow('Scan the Code', 575,350)
    cv2.imshow("Scan the Code", report)
    cv2.waitKey(0)

Using cv2 to create a pop up window to show the report.

def scan_color(image, x1, x2, y1, y2) function

• image is the image frame of the buoy
• x1, x2, y1, y2 are the coordinates for the light box on the buoy; x is width y is height

    img = cv2.imread(image)
    img = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)

Using cv2 imread to reads in the image color values data and cv2 cvtColor to change the color values from BGR to HSV.

color = np.array(img[(y1+y2)/2, (x2+x1)/2]) Color variable will hold the color value at the center of the given coordinates.

    red = 0
    blue = 0
    green = 0
    yellow = 0

Initializes the color indexes variables for the for loop.

    lower_threshold = {"red": lower_red,"blue": lower_blue, "green": lower_green, "yellow": lower_yellow}
    upper_threshold = {"red": upper_red,"blue": upper_blue, "green": upper_green, "yellow": upper_yellow}

Configure threshold for each color into lower and upper thresholds.

    for clr in lower_threshold.keys():
        lower_color = lower_threshold[clr]
        upper_color = upper_threshold[clr]
        for i in range(3):
            if (color[i] >= lower_color[i] and color[i] <= upper_color[i]):
                if(clr == "red"):
                    red += 1
                elif (clr == "blue"):
                    blue += 1
                elif(clr =="green"):
                    green += 1
                elif(clr =="yellow"):
                    yellow += 1

This for loop checks if color is within threshold of the set colors, if it is, it will increment the respective color index.

    if(red == 3):
        return 'red'
    elif(blue == 3):
        return 'blue'
    elif(green == 3):
        return 'green'
    elif(yellow == 3):
        return 'yellow'
    else:
        return 'black'

As the for loop terminates, one of the color indexes will be of equal 3 as it is within the threshold for the respective color, and if the color index equals 3 it will return the respective color detected.

What still needs to be done

• Currently the program is hardcoded and does not take in changeable inputs
• Needs Subscriber to subscribe to the Neural Net to take coordinates of the light box
• Needs Publisher to publish the color sequence
• Needs to be able to detect the beginning and/or end of the color sequence such that it does not know where in the sequence it began reading (ie if the sequence were to be RED BLUE GREEN and the program started at BLUE, etc.) This can be done using a timer or looking at the difference of HSV value of current and previous color detected.