Introduction

This page should provide an overview of how Autonomy and the ZED camera have been implemented in this project thus far.

Overview

• Autonomy Node
• ZED Tracking
• Aruco

Node and topic breakdown

Packages and nodes

• zed_wrapper: Package that communicates with the ZED camera. (Source)
  • zed_node: Node that publishes images captured by the ZED camera.
• openrobotics_darknet_ros: Package that communicates with the Darknet network. (Source)
  • detector_node: Node that sends images to Darknet, and publishes detections the network makes.
• zed_camera: Package that passes images from the ZED camera to Darknet.
  • zed_img_passer: Node that, when running, listens for images from the ZED camera and passes them to Darknet.
• test_image_generator: Package of test nodes that can log detections or publish a set of .jpg images.
  • image_generator: Node that looks for images in a predetermined folder, and publishes them to Darknet. It also listens for detections from Darknet and prints them to the console/log.
  • image_listener: Node that listens for detections from Darknet and prints them to the console/log.

Topics

• /zed/zed_node/rgb/image_rect_color: RGB color image from the ZED camera.
• /detector_node/images: Channel for sharing images to be processed by Darknet for object detection.
• /detector_node/detections: Channel for sharing object detections (class, position, bounding box) by Darknet.

ZED Tracking

Required Materials:

• ZED Camera
• ZED SDK

Prerequisites

When cloning this repository from source, you must also clone the zed-ros2-wrapper repository. To do so, run these commands:

• cd ROS2/skinny/src
• git clone https://github.com/stereolabs/zed-ros2-wrapper/tree/11ca6484df8aaf2cc6a0ecb524182c0d93c56c42

Connecting to the ZED camera

• TODO discuss starting the ZED camera, allowing it to communicate with YOLO/etc.

YOLO Object Detection Network

Prerequisites

When cloning this repository from source, you must also clone the openrobotics_darknet_ros repository. To do so, run these commands:

• cd ROS2/skinny/src
• git clone https://github.com/ros2/openrobotics_darknet_ros/tree/be4fd04c0ea8fbed80b3549283701e16145422c6

The following files must be in the skinny/ folder:

• coco.names
• yolov3-tiny.weights
• yolov3-tiny.cfg
• detector_node_params.yml

If any of the first three files are not present, you can download them using wget as follows (source). This will download the weights, configuration, and names necessary to evaluate images from the COCO dataset.

wget https://raw.githubusercontent.com/pjreddie/darknet/f86901f6177dfc6116360a13cc06ab680e0c86b0/cfg/yolov3-tiny.cfg
wget https://pjreddie.com/media/files/yolov3-tiny.weights
wget https://raw.githubusercontent.com/pjreddie/darknet/c6afc7ff1499fbbe64069e1843d7929bd7ae2eaa/data/coco.names

If the parameters file is not present, you can create one following the example from here.

Running the Darknet YOLO network

With files present, run the following command (in skinny/, and with the workspace sourced): ros2 run openrobotics_darknet_ros detector_node --ros-args --params-file detector_node_params.yml

The network communicates using the following ROS2 topics:

• /detector_node/images: Images published here are processed by Darknet
• /detector_node/detections: A list of bounding boxes of objects detected by Darknet

Object detection demo

To demo the Darknet network on some premade images, .jpg images can be added to the folder /home/team/Vision/images/. With images in this folder, an image generator node can be run to create images and listen to predictions. The node automatically logs the predictions for each image, as they are processed, to the console.

First, initialize the Darknet YOLO network by starting the detector_node node (see above). Then, launch the image_generator node as follows:

ros2 run test_image_generator image_generator

This will publish .jpg images in /home/team/Vision/images every 3 seconds.

The source directory and publishing interval can be changed by calling the node with ROS2 params as follows:

ros2 run test_image_generator image_generator --ros-args -p im_dir:="/path/to/directory" -p timer_pd:=0.5

• im_dir is the source path of a set of .jpg images (other files can be in there as well, but they will be ignored)
• timer_pd is the time between image publications in seconds (can be a floating point or integer value)

If you have another source of images being published (say, from the ZED camera), a separate node, image_listener, listens for detections from Darknet (without publishing any sets of images). Run the image_listener node as follows:

ros2 run test_image_generator image_listener

Communicating with ZED

If the zed_wrapper and detector_node modules are running, a third node, zed_image_passer, will pass images from the ZED camera to the Darknet detector node. The zed_image_passer node can be run as follows:

ros2 run zed_camera zed_image_passer

• TODO discuss changing FPS

Training Darknet

• How to do this?

//TODO: I (Jett) know nothing about this so I'm trusting you @Calvin @Carson to add info to this page. Structure it however you wish