OpenCV Stereo Vision - cu-ecen-aeld/buildroot-assignments-base GitHub Wiki

Introduction

This wiki page will help you build on from our Stereo VisionPi project. We will go through the steps to interface a Logitech C270 camera to RaspberryPi 3B+ via USB and will focus on easily configuring the camera calibration parameters for the stereo system and image rectification. At last, you'll see how to run the entire project smoothly.

Implementation Steps

Buildroot Setup

  • Research and understand the V4L2 driver for the Camera device.
  • Check if your Camera driver is working fine on Raspbian OS first before moving into Buildroot build system.
  • Once you have the application code running on Raspbian OS, it should be fairly simple to make it work on Buildroot.
  • Use this wiki page to setup OpenCV on your platform. A few additional python packages are needed for this project, which are discussed below in the Additional python packages section.

Configuring camera calibration parameters for the stereo system

The following steps describe the process for camera calibration on the Raspberry Pi, however, they could be performed on you host PC as well.

For camera calibration, we use the calibrate.py script to calculate the camera matrix and the distortion parameters. A few sample images are provided for the checkerboard pattern in the image_set_0 folder. Follow the instruction below to calculate the camera matrix.

git clone https://github.com/cu-ecen-aeld/final-project-KapureCUB/tree/master

cd final-project-KapureCUB/image_set_0

python3 calibrate.py

You can change the images present in this image_set_0 folder to be replace with your own image set and perform the calibration.

For this, change the chessboardSize and frameSize parameters of the calibrate.py script to match to your image set. Refer to the comments in this script for more details.

The script return the camera parameters and distortion parameters in the output.

Pre-requisites for running the script on host PC:

  • Python packages used for calibrate.py
  • Captured chessboard images frames from raspi

Building Application Code

Using the above parameters, a stereo vision depth map could be performed on your Raspberry Pi by following the steps below

  1. Change the cameraMatrix and distortionPara values in accordance to the values you got from the camera calibration in the capture.py script in /root directory of Raspi image

  2. Run the server code by modifying the server ip in the socket_server.py

  • Go to your terminal on your host pc and type ipconfig on windows or ifconfig on linux
  • Check the ipv4 address of you host and update this ip in you socket_server.py script on line 16
  • Run this code on your host

python3 socket_server.py

  1. Run the client application in Raspberry pi
  • In the /root directory on your raspi, change the IP address of the server to bind, serverAddress, to be the same as the host
  • Run the python script

python3 capture.py

  • This will perform the stereo vision calculations on the cameras and send 3 images to the server which are the rectified right, rectified left and depth image

Pre-requisites:

  • Host PC with python 3.10 or above and with python package matplotlib (use pip3 install matplotlib to install)
  • Powerup the RaspberryPi.
  • Connect the HDMI monitor and keyboard via the ports in RaspberryPi for a clear visual experience.
  • Connect 2 Logitech C270 web camera via USB to the RaspberryPi 3B+.
  • Having your specific image running on RPi Buildroot Setup with the ability to capture frames.

Additional python packages

  • We need to include openCV python library for this project

Target packages -> libraries-> Graphics -> opencv3 -> python

  • Following optional packages are added for the future ideas of streaming camera captures via gstreamer

Target packages -> libraries-> Graphics -> opencv3 -> gstreamer-1.x

Target packages -> libraries-> Graphics -> opencv3 -> photo

Target packages -> libraries-> Graphics -> opencv3 -> shape

Target packages -> libraries-> Graphics -> opencv3 -> videoio

Target packages -> libraries-> Graphics -> opencv3 -> video

Target packages -> libraries-> Graphics -> opencv3 -> imgcodecs

Target packages -> libraries-> Graphics -> opencv3 -> stitching

Future Ideas

  • Use gstreamer to stream video
  • Calculate actual distance of object from camera.

References