This week you are going to implement another reactive robot behaviour on top of what you did last week, which responds to images using the OpenCV library. You will practice how to:

• Convert images into the OpenCV format
• Perform image processing operations on the image
• Command the robot based on your image processing output

Task 1: Setting up your environment

First, you need to setup your environment:

1. Create a package for minitask3
2. Create a file called follower.py and make it executable
3. Download the world file and save it under your worlds folder in your package - create the folder if it does not exist.
4. Download the launch file and save it under your launch folder in your package - create the folder if it does not exist.
5. Don't forget to source your workspace source ~/catkin_ws/devel/setup.bash

Important: Open the launch file. On line 16, change the part that reads $(find xacro)/xacro.py so that it just reads xacro - without .py. This is due to an update to ROS since we created the launch file.

Important2: if you are using your own machine, you will need to make sure numpy and OpenCV are installed before starting this minitask.

sudo apt-get install ros-noetic-image-transport ros-noetic-cv-bridge ros-noetic-vision-opencv python-opencv libopencv-dev ros-noetic-image-proc

Replace python-opencv with python3-opencv if you are using python3.

If everything is set up correctly, the following command should instantiate your robot in the Gazebo world shown below:

roslaunch minitask3 turtlebot3_corridors.launch

Task 2: Exploring image topics in ROS

Your simulated robot has a camera that publishes to image topics. You can run rostopic list to see all the topics currently being published. Which ones do you think relate to the camera?

The topics starting with camera/depth are calibration and depth-sensor data, which we are not interested in as part of this Minitask. The images you can stream from the Waffle's camera are published through the camera/rgb/image_raw. You should subscribe directly to this topic when dealing with color images. Alternatively, you can use the /camera/rgb/image_raw/compressed image if you are short in bandwidth.

The theora topic applies video compression, rather than compressing the images one at a time and is even more effective in terms of compression, allowing considerable network bandwidth savings, at the expense of compression artifacts, potentially increased processor usage, and latency. As a rule of thumb, compressed video streams are useful for visualising your scene, however you shall use the uncompressed images for computer vision if you can.

You can use RViz to see what your robot sees with its camera. Run RViz using roslaunch turtlebot3_gazebo turtlebot3_gazebo_rviz.launch. In the available RViz displays, you need to find the camera and select the correct topic to visualise what the robot sees. If your PC suffers a lot from performance, you may wish to shut down RViz while running your simulation.

Task 3: Capturing images and detecting coloured objects with OpenCV

The aim of this Minitask is to program a beaconing behaviour with the Waffle: You will see some green blocks within the scene, and your aim is to beacon to the green blocks. There is no requirement to find all green blocks in the scene, however your behaviour should get the robot to move toward a single object if there are multiple in sight.

Part 1: Subscribing to image topics

In order to achieve the above goal, in your follower.py, create a simple rospy node to subscribe to image data to be processed by OpenCV. To pass data between the ROS and OpenCV image formats, we will use the cv_bridge package, which can convert between ROS sensor_msgs/Image messages and the objects used by OpenCV. A basic program that instantiates a CvBridge object to convert the incoming sensor_msgs/Image stream to OpenCV messages and displays them on the screen using the OpenCV imshow() function is given below:

#!/usr/bin/env python
import rospy
from sensor_msgs.msg import Image
import cv2, cv_bridge

class Follower:
   def __init__(self):
      self.bridge = cv_bridge.CvBridge()
      cv2.namedWindow("original", 1)
      self.image_sub = rospy.Subscriber('camera/rgb/image_raw', Image, self.image_callback)


   def image_callback(self, msg):
      image = self.bridge.imgmsg_to_cv2(msg,desired_encoding='bgr8')
      (h, w) = image.shape[:2]
      image_resized = cv2.resize(image, (w/4,h/4))
      cv2.imshow("original", image_resized)
      cv2.waitKey(3)

rospy.init_node('follower')
follower = Follower()
rospy.spin()

In order to beacon toward the green objects, you will need to first identify them. Implement colour slicing to segment a coloured object. You will need to find suitable parameters to robustly segment the objects. Probably you would be tempted to look at the combination of red, green, and blue values (e.g. 0 for red and blue, and any value between 0 and 255 for green) to decide if the object is green. Surprisingly, filtering on RGB values tend to provide very poor results for colour slicing. Instead, transforming your color space to HSV [hue (color), saturation (color intensity), and value (brightness)] works wonders! The conversion can be done in a single line with OpenCV:

hsv = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)

Part 2: HSV colour space

You can find out more about HSV (and other colour spaces) by using Resource 1. HSV uses uses the ranges 0 - 180 for hue, 0 - 255 for saturation, and 0 - 255 for value.

The image below shows the OpenCV HSV colour space. You'll find this image useful for ballparking the values of interesting colours later.

You can use an online colour-picker to get approxmiate HSV values, instead of trying to use the image - just be careful, most of them use a different range to OpenCV so you'll need to scale the values to use them.

Part 3: Using the OpenCV bridge

Inside the callback method of your subscriber, include the following code to use the OpenCV bridge:

#lightest = [_,_,_]
#darkest = [_,_,_]

image = self.bridge.imgmsg_to_cv2(msg,desired_encoding='bgr8')
cv2.namedWindow("original", 1)
cv2.imshow("original", image)

#hsv = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)
#lightest = np.array(lightest, dtype = "uint8")
#darkest = np.array(darkest, dtype = "uint8")
#mask = cv2.inRange(hsv, lightest, darkest)
#output_image = cv2.bitwise_and(image, image, mask = mask)
#cv2.imshow("thresholded", output_image)

cv2.waitKey(5000)

You can use Resource 2 to better understand this code. You'll need to understand this code to complete this minitask, and you'll also need to manipulate this code for the coursework.

Uncomment the top two lines. You will need to define a colour range that you're interested in (again, like the LIDAR, looking at one single value won't help you much). We'll be getting your robot to detect the green objects in the world. Use the colour space image to find the values of the lightest shade and the darkest shade of the colour you're interested in.

Hint: to start with a very wide range, then narrow it down.

Then, uncomment the rest of the lines. Run catkin_make, then run your script. An image should show on the screen that contains only objects of that colour. This whole process is called colour slicing.

Given the original scene, your sliced scene should look something like the following:

Hint: this is unlikely to work perfectly (or at all!) the first time. Keep adjusting your colour values until you have just the object showing.

Task 4: Visual beaconing toward green objects

Once the object is sliced, you will implement a proportional controller to beacon towards it. When the object is in sight, your p-control algorithm can be implemented to only correct the heading of the robot. You can find Chapter 12 of Quigley et al.'s Programming Robots with ROS book in Moodle to find some hints on how to implement this minitask.

Please note that your robot should beacon toward a single object if there are two objects in sight.

Hints:

1. You'll need a /cmd_vel publisher
2. Start by dividing the image into two halves
3. How is the image represented?
4. How can you use the image representation to know which half the object is in?
5. You'll need to drive towards the object if it's far away, so you'll need to decide how close to get.

Task 5: Integrating beaconing as a new behaviour

Merge this behaviour with your solution to Minitask 2, so that in case there is no beacon in sight, your robot wanders around, avoiding obstacles.

Task 6: Real robot

Now, we're going to move your code to the robot. There's a few things we'll need to do to make sure your code survives the switch from simulation to the real world :) We will also give you a coloured object to use for this part of the minitask.

1. Follow the instructions on the page Connecting to real robots to bring up your robot and turn on the camera.
2. Run rostopic list - this helps you check the camera is turned on and publishing images, but you'll also notice the image topic is slightly different... So update your code accordingly.
3. Run rqt in a desktop terminal. In the toolbar, click Plugins > Visualization > Image View. Then, there's a drop-down at the top of this window that lets you select your image topic, if you need to further verify what the camera is "seeing".
4. Come get a party cup! This will be the object your robot will look for. You can also use any small objects you have that are of a solid colour (e.g. a notebook).
5. Put your coloured object in front of the camera, and start your script. You will need to adjust the HSV values in your code.

Note: if your object is blue, the robot may not isolate just your object successfully; you can probably see that quite a few things in the lab are interpreted as blue...!

Note: unlike in the simulation, the lab has variable lighting conditions. This may affect how you define the range of HSV values in your code.

Stretch Activity: Depth Images

When using rqt, you may have noticed that you have another image topic available for "depth" data. If you have a look at the Intel RealSense camera on the front of your robot, you'll notice it has several lenses - this allows it to gather data about how far away objects are, which we call "depth" information. The simulated robot also has a depth camera.

Select the depth image topic from the dropdown in rqt. This will show you a greyscale image. What does this greyscale represent? How would you combine this image and the colour sliced image to find out where the object is and how far away it is?

SUBMISSION:

Push your code to gitlab, tagging your version as minitask3.

Resources

Resource 1: HSV

Resource 2: OpenCV bridge information