Understanding HolySee - MDHSRobotics/TeamWiki GitHub Wiki
Understanding HolySee
HolySee is Team4141's framework for vision processing. It is designed to run on a separate co-processor, such as the Tegra TX1.
Block Architecture
The framework consists of 3 main components:
-
MSee - This module is the orchestrator of the system. It is implemented in c++. This component is dependent on gstreamer. This component
- manages system configuration
- discovers connected input devices (cameras and lidar)
- discovers Roborio presence and establishes a communication channel with RoboRio via WebSockets
- Used CSee, the shared library of custom gstreamer plug-ins for computer vision and augmented reality
- enables control over which channel is streamed to the operator console
-
CSee - This module is a C++ shared library consisting of custom developed image processing gstreamer plugins for functions such as object identification, object tracking, augmented reality and computer vision. It is dependent on the OpenCV and gstreamer libraries, powerful, high performance c++ libraries what are open source and highly utilized in the industry.
-
ISee - This module provides the UI components on the driver console that enables the operator to select views from the vision system to observe. Due to bandwidth constraints, the system will stream only 1 view at a time.
- ISee enables the operator to select which channel is streamed from the robot to the console
- ISee provides the operator with a video streamer viewer to view the selected feed
Design Goals
- Support multiple video sources
- Enabling video processing of each video source using OpenCV
- Enable the operator to see raw video source
- Enable operator to see processed video
- Enable operator to configure video system
- Enable video system to send messages to RoboRio
- Enable RoboRio to configure and control video system
Streaming Pipeline Architecture
Core to the image processing solution is a high performance streaming pipeline and image processing functions. Gstreamer was selected for the streaming pipeline library because it provided a large collection of ready made high performance components that could be reused. as a set of libraries because they are high performance and have been widely adopted. OpenCV was selected for the image processing because they provided a variety of tools that facilitate image processing development and leverage GPU acceleration. In addition, GRIP, a tool provided by FRC to facilitate image processing filter design and tuning, generates OpenCV code to implement designed filters. In order to manage the bandwidth consumption, only 1 video feed at a time is selected to be streamed to the operator console.
The CV pipeline is a configurable pipeline that enables multiple devices to be connected into the system. Currently cameras and LIDAR devices are supported. Each device can be configured into the pipeline with a few options:
note: highlighed components denotes custom components which are differentiated from gstreamer components.
- Cameras can be configured to have a raw video feed available for the operator to view
- Cameras can be configured to have an augmented reality view
- Cameras can be configured to have a custom appsink which can enable the computer vision system to interact directly with the robot
note: highlighed components denotes custom components which are differentiated from gstreamer components.
- Lidars can be configured to have an augmented reality view
- Lidars can be configured to have a custom appsink which can enable the computer vision system to interact directly with the robot
- Lidars can be configured to save the data to file
In the following example, a lidar and a camera are both connected in the system. Each device is configured with each of the above features enabled.
Establishing connection with robot
DNSSD is used to detect the presence of RoboRio and to announce HolySee as a video system to RoboRio. HolySee initiates a connection to the robot. After a connection is established, the sequence diagram below describes key interactions:
