• a working bitbake environment on the build host
• a MarS Board
• basic knowledge of OpenCV

optional:

• a HDMI monitor for video output
• an USB keyboard and mouse for text input (e.g. waitKey)
• a video file for testing
• a USB UVC (USB Video Class) compatible cam for live video input
• a network connection for easier access to the target (ssh, scp) and web streaming

Make sure that your build/conf/bblayers.conf contains the following layers:

${BSPDIR}/sources/meta-openembedded/meta-networking
${BSPDIR}/sources/meta-openembedded/meta-python

The latest commit contains a recipe image-multimedia-full-with-opencv, bitbake that, install and boot it on the target (MarS Board). Note that other images may not contain the OPenCV libs. But, you can re-use packages from already built images when you work in the same bitbake environment.

Notice that the image is quite large. In the rootfs you find the following new folders and files:

• include files in /usr/include/opencv2
• libraries in /usr/lib/libopencv*

On the target, create two files in a new folder:

hello_opencv.cpp:

#include <opencv2/core/core.hpp>

int main()
{
    cv::Mat image;
    return 0;
}

Makefile:

# the include path /usr/include should already be set

# check /usr/lib/libopencv*.so in case you need more
OPENCV_LIBS = -lopencv_core -lopencv_highgui -lopencv_imgproc -lrt -lpthread -lm -ldl

hello_opencv: hello_opencv.cpp
        g++ -o $@ $< -L/usr/local/lib/  $(OPENCV_LIBS)

Hint: pkg-config --libs opencv lists all libs on which the opencv package depends. Likewise pkg-config --cflags opencv gives you the compiler flags (include path). Using all libs, however, increases the link time so we are using a stripped down version.

Now, build and run hello_opencv by executing make and starting the generated executable.

root@marsboard:~/hello_opencv# make
g++ -o hello_opencv hello_opencv.cpp -L/usr/local/lib/  -lopencv_core -lopencv_highgui -lopencv_imgproc -lrt -lpthread -lm -ldl
root@marsboard:~/hello_opencv# ./hello_opencv

This example does nothing at run-time. It just makes sure that compiling, linking and loading some opencv libraries works fine.

A HDMI monitor needs to be connected to the MarS Board. It should already be connected during boot because the driver will test the i2c connection to the monitor and may fail if you connect the monitor later.

If you don't succeed with a HDMI monitor, you will still find the file image.png on the microSD card.

Add a few lines to hello_opencv.cpp:

int main()
{
  cv::Mat image( cv::Size(320,240), CV_8UC3, cv::Scalar(255,0,0) );
  cv::imwrite( "image.png", image );
  cv::imshow( "image", image );
  cv::waitKey(0);
  return 0;
}

The target image contains, by default, an X server for graphical output and corresponding input (mouse, keyboard). So you have to set the DISPLAY environment variable before execution:

If you have connected a USB keyboard, you may end the demo by typing any key. Otherwise, from the console, you need to enter Ctrl+C to end the demo:

root@marsboard:~/hello_opencv# export DISPLAY=:0
root@marsboard:~/hello_opencv# ./hello_opencv

You should see

• a black "terminal" window with a shell prompt before the demo starts
• a 320x240 pixel blue rectangle in another window titled "image" while the demo is running
• the black "terminal" window again, after the demo has finished

Put a video file on the target, preferably H.264 encoded video in a .mp4 container. For reference, big_buck_bunny_480p_h264.mp4 from https://peach.blender.org/download/ was copied to /home/root. Note that OpenCV relies on libav for video demultiplexing and decoding. Therefore not all combinations of video codecs and container formats may work.

hello_opencv.cpp:

#include <iostream>

#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>

int main()
{
	cv::VideoCapture videoCapture("/home/root/big_buck_bunny_480p_h264.mp4");
	if( !videoCapture.isOpened() )
	{
		std::cerr << "failed to open video capture, exit" << std::endl;
		return -1;
	}
	
	cv::Mat image;
	while(1)
	{
		videoCapture >> image;
		if( image.empty() )
		{
			std::cerr << "failed to get video image, break" << std::endl;
			break;
		}
		cv::imshow( "image", image );
		int c = cv::waitKey(1);
		if( c=='x' )
			break;
	}
	return 0;
}

Build as above and set the DISPLAY variable. The video should play in a window on the HDMI monitor. In order to see some OpenCV functionality, throw in a Canny edge detector filter just before imshow, build and watch again:

    }
    cv::blur( image, image, cv::Size(3,3) );
    cv::Canny( image, image, 50, 3*50, 3 );
    cv::imshow( "image", image );

Note, that this is a pure software (single ARM CortexA9 CPU) processing. You may use top to check the CPU load.

You may change the string argument in the videoCapture c'tor to a web stream. for reference, we use the Durian Blender project.

    cv::VideoCapture videoCapture("http://docs.gstreamer.com/media/sintel_trailer-480p.webm");

Most USB cams are now UVC compatible. The UVC driver should already be included in the kernel. If not, check your kernel configuration on the build host with bitbake -c menuconfig virtual/kernel. The config entry USB_VIDEO_CLASS should be enabled.

Connect the camera to a USB port on MarS Board. On the system console, a message like this should appear:

root@marsboard:~# usb 1-1.2: new high-speed USB device number 3 using ci_hdrc

Check, that the cam is on the USB bus:

root@marsboard:~# lsusb
Bus 001 Device 003: ID 045e:0772 Microsoft Corp. LifeCam Studio
Bus 001 Device 002: ID 1a40:0101 Terminus Technology Inc. 4-Port HUB
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Also check, that a new device /dev/video0 appeared. If not, it might help to load the kernel module manually (modprobe uvcvideo)

Now, change the videoCapture c'tor in hello_opencv.cpp again:

    cv::VideoCapture videoCapture(0);

This time, 0 is a number, not a string. This tells OpenCV to get video input from the default video device which is /dev/video0, the USB cam. Rebuild and execute and watch the live stream.

There are many different types of network cameras and transmission protocols. We focus on the rtsp protocol and use a AXIS M1013 camera for reference.

MarS board and the camera must be connected to the same network and the ports and protocol must be allowed when using a router. The IP address from the camera is required. For reference, the IPv4 address 192.168.2.135 will be used below which you must adapt to your network.

Check the video first from the build host with

• a web browser (simply enter the IPv4 address)
• VLC videolan video player (open network stream, enter the rtsp URI from below)

Change the videoCapture c'tor in hello_opencv.cpp to:

    cv::VideoCapture videoCapture("rtsp://192.168.2.135/axis-media/media.amp");

There are many optional parameters that you can specify like user and password, video codec, video quality, resolution etc.. Consult the camera's manual. There is also an option to use http tunnelling in case the rtsp protocol does not work with your network.