Imaging - acfr/snark GitHub Wiki
- Camera data acquisition and processing often requires simple utilities for basic image operations, viewing, debugging, etc.
- Those operations naturally can be represented as a pipeline of filters, which would require a simple serialization of cv::Mat with the bare minimum: timestamp, image size, and image type embedded in the stream as an image header.
- Often, it is inconvenient to have such in-band data description (e.g. when the image-producing application outputs simply 2d arrays of numbers). Thus, we needed optional separation of data from data description.
- Sometimes, not the full image, but only its fragments are received (e.g. lines from a line camera); in these cases we would need tools to accumulate images.
- camera::pinhole: basic pinhole camera model
- cv_mat::filters: simple cv::Mat filters
- cv_mat::serialization: a simple serialization/deserialization of timestamped cv::Mat
- frequency_domain: ...
- region_properties: ...
- csv-to-svg: take csv points and other primitives, convert to svg (although implemented not in snark/imaging, but in snark/render)
- cv-cat: aquire from camera, transform, filter, and view cv::Mat-style images on the fly
- image-accumulate: accumulate cv::Mat-style images from the streams of numbers
- image-from-csv: take pixels as csv, output cv-cat-style image
- image-to-csv: take cv-cat-style image, output pixels as csv
- image-pinhole: operations on pinhole camera images
- stereo-to-points: use opencv to produce rectified images, disparity images or point clouds
- basler-cat: obtain images a basler gige camera
- gige-cat: obtain images directly from a gigabit ethernet camera
- gobi-cat: obtain images directly from a gobi gige camera
- fire-cat: obtain images directly from a firewire camera
- flycapture-cat: obtain images a flycapture gige camera
- jai-cat: obtain images a jai gige camera
cv-cat takes images on stdin, applys filters, and results to stdout. Of course, this all easily could be done by hand, using opencv in python, but cv-cat in many cases allow to avoid coding altogether.
The serialized image complies with opencv cv::Mat class: [header]pixels.
The full header is: [t][rows][cols][type] in any order; all the fields optional, where t is timestamp as microseconds since epoch (1 January 1970); type is the same as cv::Mat type in opencv.
The missing header fields need to be specified in the command line. The whole header can be missing with the values specified in the command line, allowing to achieve separation between the data and data description.
In addition to the examples here, see the step-by-step tutorial including sample data.
View output from a gige camera:
gige-cat | cv-cat view > /dev/null
Do bayer conversion, resize, and view output from a gige camera:
gige-cat | cv-cat "bayer=1;resize=512,360;view" > /dev/null
Collect a log from gige camera; replay and view:
gige-cat > gige.bin cat gige.bin | csv-play --binary=t,3ui,12000000ub | cv-cat "bayer=1;resize=512,360;view" > /dev/null
Apply your own filter (say, called detect-edges) and view result on the fly:
gige-cat | cv-cat bayer=1 | detect-edges | cv-cat "resize=512,360;view" > /dev/null
View a 2d byte array 200x300 as a grey-scale image (type=ub: a shorthand for opencv 8-bit grey type):
cat array.bin | cv-cat --input="no-header;rows=300;cols=200;type=ub" view > /dev/null
Encode the output from a logged file (gige.bin as above) to a compressed .mp4 video, using a standard linux utility 'avconv'. Convert raw data to colour (bayer), and add a timestamp.
cat gige.bin | cv-cat "bayer=1;timestamp;encode=jpg" --output=no-header | avconv -y -f image2pipe -r 25 -c:v mjpeg -i pipe: -c:v libx264 -threads 0 -b 2000k -r 25 -pre slow -crf 22 gige.mp4
Encode the output from a gige camera as a .mkv video, using a standard linux utility 'avconv' *
gige-cat | cv-cat "encode=ppm" --output=no-header | avconv -y -f image2pipe -vcodec ppm -r 25 -i pipe: -vcodec libx264 -threads 0 -b 2000k -r 25 video.mkv
Encode the output from a logged file (gige.bin as above) a compressed .mp4 video, using a standard linux utility 'avconv'. Convert raw data to colour (bayer), resize, and add a timestamp. *
cat gige.bin | cv-cat "bayer=1;resize=640,480;timestamp;encode=ppm" --output=no-header | avconv -y -f image2pipe -vcodec ppm -r 25 -i pipe: -vcodec libx264 -threads 0 -b 2000k -r 25 -pre slow -crf 22 gige.mp4
todo: more examples
* An issue may be encountered when converting a large number of PPM images to video. The alternative is to try other encoding formats such as jpeg, png, etc.. See command above which encodes frames to jpg then to video.
Turn matrix style data into rectangular cv style images and view them
cat radar.bin | image-accumulate --binary=t,2f,ui,256f --degrees --fields=t,angle,,block,values --size=200,256 --output="rows=200;cols=256;type=3ub;header" --polar --dirty --colourmap=hot --scale=75,110 | cv-cat "view;null"
Combine the examples above to create an .mp4 video from matrix style data. Note: resize image to a multiple of 2, or avconv will complain.
cat radar.bin | image-accumulate --binary=t,2f,ui,256f --degrees --fields=t,angle,,block,values --size=200,256 --output="rows=200;cols=256;type=3ub;header" --polar --dirty --colourmap=hot --scale=75,110 | cv-cat "resize=512,512;timestamp;encode=ppm" --output="no-header" | avconv -y -f image2pipe -vcodec ppm -r 2 -i pipe: -vcodec libx264 -threads 0 -b 2000k -r 2 -pre slow -crf 22 hss-radar-spectrum.mp4
todo: more examples
In addition to the examples here, see the step-by-step tutorial including sample data.
Read stereo images from files or stdin, outputs point cloud or disparity image or rectified image pair to stdout.
input sgbm parameters:
-w [ --window-size ] arg (=5) sgbm SADWindowSize (see OpenCV
documentation)
-m [ --min-disparity ] arg (=0) sgbm minDisparity
-n [ --num-disparity ] arg (=80) sgbm numberOfDisparities
-u [ --uniqueness ] arg (=10) sgbm uniquenessRatio
-s [ --speckle-window-size ] arg (=1000)
sgbm speckleWindowSize
-r [ --speckle-range ] arg (=16) sgbm speckleRange
--disp12-max arg (=1) sgbm disp12MaxDiff
--pre-filter-cap arg (=63) sgbm preFilterCap
output and view point cloud from 2 image files:
stereo-to-points --left left.bmp --right right.bmp --config bumblebee.config --left-path left --right-path right --binary t,3d,3ub,ui \
| view-points --fields t,x,y,z,r,g,b,block --binary t,3d,3ub,ui
output and view disparity from 2 image files:
stereo-to-points --left left.bmp --right right.bmp --config bumblebee.config --left-path left --right-path right \
--disparity | cv-cat --output=no-header encode=ppm | display
output and view rectified image pair from 2 image files:
stereo-to-points --left left.bmp --right right.bmp --config bumblebee.config --left-path left --right-path right \
--output-rectified | cv-cat --output=no-header encode=ppm | display
output rectified image pairs as single png files for further external processing:
cat 20130319T052306.593000.bin | cv-cat "split;bayer=4" | stereo-to-points --config bumblebee.config --left-path left --right-path right --roi 0,1920,0,0,1280,960 --output-rectified | cv-cat "file=png;null"
where bumblebee.config contains:
left=
{
focal-length="1604.556763,1604.556763"
centre="645.448181,469.367188"
image-size="1280,960"
map=/usr/local/etc/shrimp.bumblebee-left.bin
}
right=
{
focal-length="1604.556763,1604.556763"
centre="645.448181,469.367188"
translation="-0.239928,0,0"
image-size="1280,960"
map=/usr/local/etc/shrimp.bumblebee-right.bin
}
assuming a log was created from a Bumblebee camera (e.g. using fire-cat --args > file.bin)
produce and visualise coloured point clouds:
cat file.bin | cv-cat "split;bayer=4" | stereo-to-points --config bumblebee.config --left-path left --right-path right --roi 0,1920,0,0,1280,960 --binary t,3d,3ub,ui --min-disparity=10 | view-points --fields t,x,y,z,r,g,b,block --binary t,3d,3ub,ui
The following also provides the individual (left/right) rectified images separately as png files:
cat 20130319T052306.593000.bin | cv-cat "split;bayer=4;crop-tile=0,0,1,3;undistort=/usr/local/etc/shrimp.bumblebee-right.bin;view;file=right-undistort.png;null" cat 20130319T052306.593000.bin | cv-cat "split;bayer=4;crop-tile=0,2,1,3;undistort=/usr/local/etc/shrimp.bumblebee-left.bin;view;file=left-undistort.png;null"
split the stereo processing pipeline into two parts
Note, this is not so useful as a pipeline, but shows you can output rectified images (e.g. use cv-cat to save as png files to share with someone) and produce point clouds from rectified images (e.g. use a public image dataset that's already rectified)
cat 20130319T052306.593000.bin | cv-cat "split;bayer=4" | stereo-to-points --config=bumblebee.config --left-path=left --right-path=right --roi="0,1920,0,0,1280,960" --output-rectified | stereo-to-points --config=bumblebee.config --left-path=left --right-path=right --roi="0,0,1280,0,1280,960" --input-rectified --binary t,3d,3ub,ui | csv-select --binary=t,3d,3ub,ui --fields=,,,,,,,block --to=0 --sorted > pointcloud.bin
Note the use of roi for the second stereo-to-points cmd with --input-rectified option, to accept as input rectified image pairs from the previous invocation of stereo-to-points with --output-rectified option. The first roi tells stereo-to-points that the left,right image from the bumblebee is at the bottom,top (respectively) of a column of three stacked images as output by the cv-cat split command. The output of stereo-to-points is arranged logically with the left,right images on the left and right of the combined paired image, so the second roi is specified accordingly.
You can use the cam-cat tools (gobi-cat, gige-cat, flycapture-cat, fire-cat and basler-cat) with filters and use them to input OpenCV Mat objects into your C++ programs.
static snark::cv_mat::serialization::options output_options;
static snark::cv_mat::serialization ser(output_options);
cv::namedWindow("image", cv::WINDOW_FREERATIO);
while (std::cin.good() && !std::cin.eof()) {
auto imPair = ser.read(std::cin);
cv::Mat& image = imPair.second;
cv::imshow("image", image);
cv::waitKey(1);
}