get started with data streams - acfr/snark GitHub Wiki

Table of Contents datasets playback live georeferencing

This tutorial continues from get started with point clouds. Now we focus on data streams, which is where comma and snark really start to shine.

For this tutorial it is assumed that you have snark installed, and have downloaded the dataset below. It is advisable to first take a look at the get started with point clouds tutorial, but it's not strictly necessary. This tutorial will perform all actions on binary data (it's faster, more efficient), but all the pipelines work in ascii too.

This tutorial covers:

• playback of data
• stream manipulation
• georeferencing point data
• visualisation

The examples run natively on Linux. On Windows, it has largely been tested in cygwin and many things work in the native command prompt too.

Accompanying this tutorial is a single dataset where a sensor vehicle is driven in a circular trajectory, providing localisation (the vehicle trajectory), data from two 2D SICK LiDARS (one horizontal, one vertically oriented) and a Velodyne 3D LiDAR.

The Shrimp perception research ground vehicle has a variety of sensors covering many different modalities, and this was used to gather data at the Australian Centre for Field Robotics' (ACFR) Marulan test facility, in New South Wales, Australia. Note that some image data (Colour GigE, Thermal LWIR, Bumblebee Stereo and Ladybug Panospheric) is also available for this dataset as part of the get started with image data tutorial.

The data are split into separate downloads:

• http://perception.acfr.usyd.edu.au/data/samples/circle/config.tar.gz: 80MB
• http://perception.acfr.usyd.edu.au/data/samples/circle/novatel.bin.gz: 0.5MB
• http://perception.acfr.usyd.edu.au/data/samples/circle/sick-h.bin.gz: 17MB
• http://perception.acfr.usyd.edu.au/data/samples/circle/sick-v.bin.gz: 6MB
• http://perception.acfr.usyd.edu.au/data/samples/circle/velodyne.tar.gz: 938MB

You can comfortably do most of this tutorial with the 'config', 'vertical sick' and the 'novatel' data. If you are connecting via a modem on a rusty phone line, you might get by without the config.

Once downloaded, unzip the data:

 tar -xvf config.tar.gz
 tar -xvf velodyne.tar.gz
 gunzip novatel.bin.gz
 gunzip sick-h.bin.gz
 gunzip sick-v.bin.gz

With the multi-sensor dataset, we have included a config file to describe it. Most of the content describes the imagery which is not part of this tutorial.

Take a look at the config file:

 cat config/shrimp.json

Get a particular entry:

 cat config/shrimp.json | name-value-get sick-v/offset

As an aside, there is nothing special about config files: they are just JSON name-value pairs. If you prefer XML, run:

 cat config/shrimp.json | name-value-convert --to xml > shrimp.xml

Path-value format often is very useful for extracting values with Linux utilities like grep and cut, e.g. try:

 cat shrimp.json | name-value-convert --from json --to path-value | grep sick-v/offset | cut -d= -f2

As we will focus on streams, it makes sense to start by emulating real-time streaming data. We achieve this by using the timestamps in log files to meter out the data at the original rate.

View the whole vehicle trajectory:

 view-points "novatel.bin;binary=t,6d;fields=,x,y,z"

or using the command line input (and using the config file):

 cat novatel.bin | view-points --binary=$(cat config/shrimp.json | name-value-get novatel/binary) --fields=$(cat config/shrimp.json | name-value-get novatel/fields)

...using the config file can be cumbersome to type, but good for scripts.

Don't forget this alternate syntax, "-" refers to stdin as a file:

 cat novatel.bin | view-points "-;binary=t,6d;fields=,x,y,z"

...and as per the previous tutorial, you could even unzip and process in one pipe:

 zcat novatel.bin.gz | view-points "-;binary=t,6d;fields=,x,y,z"

Now let's play the position back in real time:

 cat novatel.bin | csv-play --binary=t,6d | view-points "-;colour=yellow;point-size=5" "novatel.bin;colour=red" --binary=t,6d --fields=,x,y,z

To make it more interesting, let's use the pseudo real-time position to render a CAD model of the vehicle:

 cat novatel.bin | csv-play --binary=t,6d | view-points "-;fields=,x,y,z,roll,pitch,yaw;shape=config/shrimp.obj;flip" "novatel.bin;colour=red" --binary=t,6d --fields=,x,y,z

Note the command line term "flip". In this case the CAD model must be "flipped" because our data is typically in a "north, east, down" frame, but the model is stored with a "z up" frame of reference by default.

Let's georeference some laser data to make a more compelling backdrop.

1. check the laser data format and test it works using head:

 cat config/shrimp.json | name-value-get sick-v
 cat sick-v.bin | csv-from-bin t,3d,uw,ui | head -n3

2. convert to cartesian using points-to-cartesian, and georeference using points-frame:

 cat sick-v.bin | points-to-cartesian --binary=t,3d,uw,ui --fields=,r,b,e | points-frame --discard --binary=t,3d,uw,ui --fields=t,x,y,z --from="0.2082,0.0020,-0.7094,1.5802,-0.0028,-0.0117 + novatel.bin" > sick-v-geo.bin

Note: points-frame is a powerful tool. Above, it transforms from sensor-to-body frame, then the body-to-world frame transform is done using the time-aligned trajectory. This can be used for arbitrary chains of static and dynamic transforms, just keep adding using '+' (e.g. imagine a laser on a robotic arm on a moving platform...)

Now render the scene using the new backdrop, coloured by extents.

1. calculate mean and standard deviation of georeferenced z height:

 cat sick-v-geo.bin | csv-calc mean,stddev --binary=t,3d,uw,ui --fields=,,,z, | csv-from-bin 2d

2. render everything so far (using the line above to help set the colour scale):

 cat novatel.bin | csv-play --binary=t,6d | view-points "-;fields=,x,y,z,roll,pitch,yaw;shape=config/shrimp.obj;flip;binary=t,6d;fields=,x,y,z" "novatel.bin;colour=red;binary=t,6d;fields=,x,y,z" "sick-v-geo.bin;binary=t,3d,uw,ui;fields=,x,y,z;colour=-644:-643"

To render the live laser data at the same time, we'll use FIFO buffers:

 mkdir tmp
 mkfifo tmp/sick-v.fifo
 cat sick-v.bin | csv-play --binary=t,3d,uw,ui | points-to-cartesian --binary=t,3d,uw,ui --fields=,r,b,e | points-frame --discard --binary=t,3d,uw,ui --fields=t,x,y,z --from="0.2082,0.0020,-0.7094,1.5802,-0.0028,-0.0117 + novatel.bin" | csv-paste "-;binary=t,3d,uw,ui" "value=0,0,0;binary=3d" | points-frame --discard --binary=t,3d,uw,ui,3d --fields=t,,,,,,x,y,z --from="0.2082,0.0020,-0.7094,1.5802,-0.0028,-0.0117 + novatel.bin" > tmp/sick-v.fifo &
 cat novatel.bin | csv-play --binary=t,6d | view-points "-;fields=,x,y,z,roll,pitch,yaw;shape=config/shrimp.obj;flip;binary=t,6d;fields=,x,y,z" "novatel.bin;colour=red;binary=t,6d;fields=,x,y,z" "sick-v-geo.bin;binary=t,3d,uw,ui;fields=,x,y,z;colour=-644:-643" "tmp/sick-v.fifo;binary=t,3d,uw,ui,3d;fields=,second,,block,first;shape=line;colour=blue"

The block above looks complicated, so let's break it down:

1. we create a fifo, which is a file buffer and a convenient way to handle multiple streams (as there is only one stdin).
2. we read the raw laser data, play it, convert to cartesian in sensor frame, convert to body then to the world frame, all as before (with the addition of 'play').
3. we append 0,0,0 to the stream and georeference that (which means we are georeferencing the sensor origin) - together, the georeferenced origin and georeferenced end-points describe the laser beams.
4. we output that into the FIFO, all as a background task.
5. we visualise everything as we did before, with the addition of the FIFO, using shape=line.

Note: the same setup works for live streaming live data too, where we typically have network sockets serving data (e.g. using socat/netcat) instead of cat from FIFO.

In the example above we rely on the two separate csv-play apps running in sync, which is ok for a quick visualisation. To synchronise explicitly, csv-play has a multi-i/o functionality where multiple streams can be read in, and metered out in sync. See "csv-play -h" for details. Here's what it looks like for our example:

 rm tmp/*
 mkfifo tmp/novatel.play tmp/sick-v.play tmp/sick-v.out
 csv-play "novatel.bin;tmp/novatel.play;binary=t,6d;fields=t," "sick-v.bin;tmp/sick-v.play;binary=t,3d,uw,ui;fields=t," &
 cat tmp/sick-v.play | points-to-cartesian --binary=t,3d,uw,ui --fields=,r,b,e | points-frame --discard --binary=t,3d,uw,ui --fields=t,x,y,z --from="0.2082,0.0020,-0.7094,1.5802,-0.0028,-0.0117 + tmp/novatel.play" | csv-paste "-;binary=t,3d,uw,ui" "value=0,0,0;binary=3d" | points-frame --discard --binary=t,3d,uw,ui,3d --fields=t,,,,,,x,y,z --from="0.2082,0.0020,-0.7094,1.5802,-0.0028,-0.0117 + tmp/novatel.play" > tmp/sick-v.out &
 view-points "tmp/novatel.play;fields=,x,y,z,roll,pitch,yaw;shape=config/shrimp.obj;flip;binary=t,6d;fields=,x,y,z" "novatel.bin;colour=red;binary=t,6d;fields=,x,y,z" "sick-v-geo.bin;binary=t,3d,uw,ui;fields=,x,y,z;colour=-644:-643" "tmp/sick-v.out;binary=t,3d,uw,ui,3d;fields=,second,,block,first;shape=line;colour=blue"

Note a subtle difference I added to the lines above: points-frame is now using the live tmp/novatel.play instead of the offline novatel.bin, to demonstrate that these pipelines run from live data. The same tool chain that helps us visualise logged data also serves as our real-time GUIs for many different situations!

Let's finish by adding the Velodyne 3D laser via stdin (the initial lines above are repeated):

 rm tmp/*
 mkfifo tmp/novatel.play tmp/sick-v.play tmp/sick-v.out
 csv-play "novatel.bin;tmp/novatel.play;binary=t,6d;fields=t," "sick-v.bin;tmp/sick-v.play;binary=t,3d,uw,ui;fields=t," &
 cat tmp/sick-v.play | points-to-cartesian --binary=t,3d,uw,ui --fields=,r,b,e | points-frame --discard --binary=t,3d,uw,ui --fields=t,x,y,z --from="0.2082,0.0020,-0.7094,1.5802,-0.0028,-0.0117 + novatel.bin" | csv-paste "-;binary=t,3d,uw,ui" "value=0,0,0;binary=3d" | points-frame --discard --binary=t,3d,uw,ui,3d --fields=t,,,,,,x,y,z --from="0.2082,0.0020,-0.7094,1.5802,-0.0028,-0.0117 + novatel.bin" > tmp/sick-v.out &
 cat velodyne/*.bin | velodyne-to-csv -q --db=config/db.xml --fields=t,x,y,z,scan --binary | points-frame --discard --binary=t,3d,ui --fields=t,x,y,z --from="0.0215,0.0026,-0.8822,-0.0014,0.0087,3.1143 + novatel.bin" | view-points "tmp/novatel.play;fields=,x,y,z,roll,pitch,yaw;shape=config/shrimp.obj;flip;binary=t,6d;fields=,x,y,z" "novatel.bin;colour=red;binary=t,6d;fields=,x,y,z" "sick-v-geo.bin;binary=t,3d,uw,ui;fields=,x,y,z;colour=-644:-643" "tmp/sick-v.out;binary=t,3d,uw,ui,3d;fields=,second,,block,first;shape=line;colour=blue" "-;binary=t,3d,ui;fields=,x,y,z,block;colour=grey"