Image Processing Using OpenCV

Introduction

Welcome to our exploration of OpenCV, a powerful tool in the realm of image processing. In today's digital landscape, the manipulation and analysis of images are integral across diverse fields such as healthcare, surveillance, entertainment, and automotive industries. OpenCV, developed by Intel in the late 1990s and maintained by Willow Garage and Itseez, offers a comprehensive library of functions and algorithms for image and video analysis. Throughout this tutorial, we will delve into the practical applications of OpenCV, such as color space conversion, image gradients, and contour detection. With this understanding, we can harness OpenCV’s capabilities to address real-world challenges.

A Brief Background of Image Processing

Image processing, the application of signal processing methods to input images, can enhance an image, extract valuable information, or convert them into a more useful format. This field is divided into two main categories: analog and digital image processing. The primary difference between analog and digital image processing lies in the medium and methods of processing.

Analog image processing is performed on continuous signals using physical manipulations. For example, it involves manipulating physical copies of images, such as prints or photographs. This traditional form of image processing includes techniques like airbrushing, dodging and burning in a darkroom, or even manually altering the image with tools. These methods are often hands-on and require a certain level of artistic skill and understanding of photographic materials and chemicals.

In contrast, digital image processing is performed on discrete pixels using computational algorithms. It relies on digital software to manipulate image data. The process involves three key phases: preprocessing, where images are prepared for analysis(like noise reduction or contract enhancement); augmentation and display, which might include adjusting brightness or applying filters; and information extraction, which involves analyzing the image to identify patterns, edges, or specific features.

The flowchart illustrates a sequence of steps in digital image processing: beginning with the acquisition of an image, it undergoes noise removal to eliminate unwanted variations. The image is then converted to grayscale to simplify processing. Edge detection identifies object boundaries, followed by image segmentation to isolate distinct elements within the image. Alongside, image enhancement improves visual quality, aiding in the segmentation's effectiveness. These stages collectively transform raw image data into a format suitable for analysis and interpretation.

In our tutorial, we will explore three working principles of digital image processing using OpenCV tutorials: Changing colorspaces, Image Gradient, and Contours.

Changing colorspaces allows us to convert images between different color representations, facilitating tasks such as object tracking and image segmentation. Image Gradient techniques like Sobel, Scharr, and Laplacian filters are essential for edge detection, enabling us to identify boundaries and transitions within images. Contour detection involves identifying and analyzing the shapes and boundaries of objects within an image, crucial for shape analysis and object recognition. Together, these principles provide a robust foundation for various advanced image processing applications.

Tutorials

Tutorial 1: Changing color-spaces

Understanding color-space conversion begins with how computers recognize images. Images are identified by their dimensions, which are measured in pixels along the width and height. For example, an image that is 500 pixels wide and 400 pixels tall has a total of 200,000 pixels. Each pixel acts as a discrete point in the image, capable of displaying a particular color or intensity. Common representations for these colors and intensities include grayscale, BGR (Blue, Green, Red), and HSV (Hue, Saturation, Value) color spaces.

• Grayscale: Each pixel's value ranges from 0 to 255, where 0 signifies total blackness and 255 represents full whiteness.
• RGB: Pixels consist of three numbers, each between 0 and 255, indicating the levels of red, green, and blue, respectively.
• HSV: This color space captures 'hue, saturation, and value.' OpenCV scales hue between 0 to 179 and both saturation and value from 0 to 255.

Image processing requires fixed sequences of operations that are performed at each pixel of an image. The image processor performs the first sequence of operations on the image, pixel by pixel. Once this is fully done, it will begin to perform the second operation, and so on. The output value of these operations can be computed at any pixel of the image.

In this tutorial, we will explore image color-space conversion, focusing on transitions between BGR ↔ Gray and BGR ↔ HSV.We will demonstrate how to use essential OpenCV functions like cv.cvtColor() and cv.inRange() for these conversions.

To convert colors, the cv.cvtColor() function in OpenCV is used. The 'flag' parameter determines the type of conversion. For BGR to Gray conversion, the flag is cv.COLOR_BGR2GRAY, and for BGR to HSV, it's cv.COLOR_BGR2HSV.

    import cv2 as cv
    flags = [i for i in dir(cv) if i.startswith('COLOR_')]
    print( flags )

As one of the examples for application, we can try to extract a blue colored object.

import cv2 as cv
import numpy as np
 
cap = cv.VideoCapture(0)
while(1):
 
 # 1. Take each frame
 _, frame = cap.read()

# 2. Convert from BGR to HSV color-space
 hsv = cv.cvtColor(frame, cv.COLOR_BGR2HSV)

# 3. Define range of blue color in HSV
 lower_blue = np.array([110,50,50])
 upper_blue = np.array([130,255,255])

# 4. Threshold the HSV image to get only blue colors 
 mask = cv.inRange(hsv, lower_blue, upper_blue)

# 5. Extract the blue object alone, you can do whatever you want on that image
 res = cv.bitwise_and(frame,frame, mask= mask)
 
 cv.imshow('frame',frame)
 cv.imshow('mask',mask)
 cv.imshow('res',res)
 k = cv.waitKey(5) & 0xFF
 if k == 27:
 break
 
cv.destroyAllWindows()

Below image shows tracking the blue object:

This is a simple and basic method in object tracking using color-space conversion in OpenCV. Likewise, we can apply this method for extracting more than one colored object and use image editing.

Colored object tracking using color-space conversion in OpenCV has several practical applications. For instance, in robotics, it enables robots to recognize and interact with objects of specific colors, such as sorting colored items on a conveyor belt. In sports analytics, it helps track players or balls based on their uniform colors for performance analysis. Additionally, in augmented reality (AR), color tracking facilitates overlaying digital content on real-world objects. This method is also useful in surveillance for detecting and following objects of interest in a scene, such as vehicles or individuals wearing specific colors, enhancing security measures.

Tutorial 2: Image Gradient

In this tutorial, we focus on image gradients, fundamental for edge detection in image processing. Image gradients reveal sharp changes in pixel intensity, marking the contours and boundaries that define objects and shapes. This principle supports edge detection algorithms, essential for various image analysis applications.

We will discuss how to calculate image gradients using OpenCV functions: cv.Sobel(), cv.Scharr(), and cv.Laplacian(). OpenCV provides these three gradient filters—Sobel, Scharr, and Laplacian—each designed for different aspects of edge detection. The Sobel and Scharr filters are optimal for detecting horizontal and vertical edges, while the Laplacian filter is effective for highlighting edges regardless of orientation.

1. Sobel and Scharr Derivatives

The working principle of image gradient is the calculus of the derivatives in an image. The Sobel operator combines Gaussian smoothing with differentiation, making it robust against noise. It is based on the fact that in the edge area, the pixel intensity shows a “jump” or a high variation of intensity. Getting the first derivative of the intensity, we observe that an edge is characterized by a maximum and the second derivative is zero. From this, we deduce that the second derivative can be used to detect edges. Since images are "2D," we need to compute the gradient in both horizontal and vertical dimensions. The Sobel operator achieves this by convolving the image with a pair of convolution kernels, one for each dimension (Gx and Gy). These kernels approximate the derivative of the image intensity in the respective directions. The gradient magnitude can then be computed as:

The Sobel operator is a discrete differentiation operator, computing an approximation of the gradient of the image intensity function. The operator uses two 3x3 convolution kernels, which are essentially discrete differences. These kernels are convolved with the original image to calculate the approximations of the derivatives—one for horizontal changes and one for vertical. The Laplacian filter, on the other hand, calculates the second derivative of the image. This operator is isotropic, meaning it treats all orientations equally, making it effective for detecting edges regardless of their direction. The Laplacian of an image highlights regions of rapid intensity change and is typically applied to an image that has already been smoothed with a Gaussian filter to reduce its sensitivity to noise.

2. Laplacian Derivatives

cv.Laplacian() function calculates the Laplacian of the source image by adding up the second x and y derivatives calculate using Sobel operator:

import numpy as np
import cv2 as cv
from matplotlib import pyplot as plt
 
img = cv.imread('dave.jpg', cv.IMREAD_GRAYSCALE)
assert img is not None, "file could not be read, check with os.path.exists()"
 
laplacian = cv.Laplacian(img,cv.CV_64F)
sobelx = cv.Sobel(img,cv.CV_64F,1,0,ksize=5)
sobely = cv.Sobel(img,cv.CV_64F,0,1,ksize=5)
 
plt.subplot(2,2,1),plt.imshow(img,cmap = 'gray')
plt.title('Original'), plt.xticks([]), plt.yticks([])
plt.subplot(2,2,2),plt.imshow(laplacian,cmap = 'gray')
plt.title('Laplacian'), plt.xticks([]), plt.yticks([])
plt.subplot(2,2,3),plt.imshow(sobelx,cmap = 'gray')
plt.title('Sobel X'), plt.xticks([]), plt.yticks([])
plt.subplot(2,2,4),plt.imshow(sobely,cmap = 'gray')
plt.title('Sobel Y'), plt.xticks([]), plt.yticks([])
 
plt.show()

This image shows that their outputs are different, specifying the direction of derivatives (vertical or horizontal) using the 'yorder' and 'xorder' arguments, respectively. Additionally, the 'ksize' argument lets you define the kernel size. If 'ksize' equals -1, a 3x3 Scharr filter is applied, which typically yields better results than a 3x3 Sobel filter.

Sobel filtering is considered more computationally efficient and easier to implement than Scharr filtering, providing accurate results for general purpose edge detection tasks. Scharr filtering is typically used in cases where high precision edge detection is necessary. High frequency image detailing is common in medical imaging or situations where extreme precision is necessary.

A direct comparison of Sobel and Scharr filtering is listed in the table below.

Tutorial 3: Contours

Contours are curves linking continuous points along edges of regions with the same color or intensity, crucial for shape analysis, object detection, and recognition. In OpenCV, detecting contours is akin to identifying a white object against a black background. The image processes in binary: non-zero pixels represent the object, and zero pixels form the background. For accurate detection, the object must appear white and the background black. Contours feature attributes like area, perimeter, centroid, and bounding box, providing valuable information. Functions such as ‘convexHull()’ use algorithms like Sklansky's to find the convex hull of 2D point sets, suitable for polygons. Our example code focuses on the basic principle of contour detection.

The contour detection process follows these steps: it begins by converting the color image to grayscale, then transforms the grayscale image to binary using a fixed threshold. If the objects in the image are black against a white background, the image inverts so black becomes white and white becomes black, enabling contour detection.

import numpy as np
import cv2 as cv
im = cv.imread('test.jpg')
assert im is not None, "file could not be read, check with os.path.exists()"
imgray = cv.cvtColor(im, cv.COLOR_BGR2GRAY)
ret, thresh = cv.threshold(imgray, 127, 255, 0)
contours, hierarchy = cv.findContours(thresh, cv.RETR_TREE, cv.CHAIN_APPROX_SIMPLE)

In the cv.findContours() function, the parameters are the source image, contour retrieval mode, and contour approximation method, respectively. This function outputs the contours and their hierarchy, retrieving contours from the binary image through the specified algorithm.

The cv.RETR_TREE mode retrieves all of the contours and reconstructs a full hierarchy of nested contours, while cv.CHAIN_APPROX_SIMPLE compresses horizontal, vertical, and diagonal segments, leaving only their end points. This function outputs the contours and their hierarchy, retrieving contours from the binary image through the specified algorithm. The hierarchy information is particularly useful for understanding the relationship between the contours, such as nested objects or holes within objects.

By examining the detected contours, one can extract and analyze features such as area, perimeter, and the bounding box of objects. These features are valuable for tasks like object recognition, where understanding the shape and size of detected objects can lead to more accurate identification. Contour detection is a fundamental technique in computer vision, providing a foundation for more complex image analysis and processing tasks.

Conclusion

Image processing emerges as a powerful tool, allowing us to manipulate and analyze digital images effectively. Throughout this tutorial, we explored essential techniques using OpenCV and python, covering color space conversion, image gradients, and contour detection. Starting with color space conversion, we simplified image analysis, while image gradients informed us about intensity changes, crucial for edge detection and object boundary identification. Contour detection allowed us to segment and isolate important features. Continued application of these techniques will empower users to create engaging visualizations and glean insights, enhancing advancements in artificial intelligence, facial recognition, and complex image processing tasks.

References

1. Nagalakshmi, Dr & Jyothi, Singaraju. (2015). Image Acquisition, Noise removal, Edge Detection Methods in Image Processing Using Matlab for Prawn Species Identification. G Nagalakshmi, S Jyothi.
2. https://docs.opencv.org/4.x/df/d9d/tutorial_py_colorspaces.html
3. https://www.projectpro.io/recipes/what-are-sobel-and-scharr-derivatives-opencv
4. Atul, K. &, & Atul, K. &. (2019, May 24). Scharr operator. TheAILearner. https://theailearner.com/tag/scharr-operator/
5. Rosebrock, A. (2021, April 17). Finding extreme points in contours with opencv. PyImageSearch. https://pyimagesearch.com/2016/04/11/finding-extreme-points-in-contours-with-opencv/
6. automaticaddison, A. (2021, February 28). How to detect and draw contours in images using opencv. Automatic Addison. https://automaticaddison.com/how-to-detect-and-draw-contours-in-images-using-opencv/
7. Simplilearn. (2023, October 11). What is image processing? everything you need to know!. Simplilearn.com. https://www.simplilearn.com/image-processing-article#benefits_of_image_processing
8. "Project Library: What are Sobel and Scharr Derivatives in OpenCV." ProjectPro, 20 Feb. 2023, www.projectpro.io/recipes/what-are-sobel-and-scharr-derivatives-opencv. Accessed 9 June 2024.