• Plan of record
• High Resolution disparity

• Go over documentation

• Patch duplets do best for pose according to
  • A Performance Evaluation of Local Descriptorsby ‘’Krystian Mikolajczyk and Cordelia Schmid’’
    • Patch-Duplets is described in:Patch-Duplets for Object Recognition and Pose Estimationby ‘’Bjorn Johansson and Anders Moe’’
  • Though is to use this idea but record the texture patches with ‘’Scale Invariant Feature Transform with Irregular Orientation Histogram Binning’’, Yan Cui, Nils Hasler, Thorsten Thorm?hlen, Hans-Peter Seidel
  • But done like the Daisy detector
  • Daisy detector?
    • Fast way of accessing circular regions in images

• Active Segmentation with Fixation. Ajay Mishra , Yiannis Aloimonos, Cheong Loong Fah, iccv 2009
  • Code (Malab and windows at least).

• Dense optical flow
  • Maybe with Stereo
• Alternative stereo correspondence algorithm
  • Graph cut on GPU
  • Block matching variants

• Feature based recognition and pose
  • Feature + geometry
• Stereo recognition
• 3d model capture
• Plan:
  • Victor on stereo this week
  • Maria finish up one more calibration test by end of week
  • Maria switch to Stereo on Friday
  • Duplets, Gary continue, work with Suat to test
  • Victor on vacation next week, when back go onto texture based recognition and pose

Minutes

• cv and cxcore coverage are up
  • When to relegate to background
    • cv
      • stereo correspondence
        • Someone dedicated to this for awhile
      • texture operators
        • MSER
    • cxcore — to high coverage
• Texture operator
  • Extremely random trees or
  • Vocabulary trees
  • Naive Bayes
• Disparity in High res stereo
  • Large number of invalid pixels — not finding disparity
  • Kurt suggested dynamic programming for whole image
    • Stereo processing by semi-global matching and mutual information Hirschmuller
• Argus: Argus for Russian work, Itsees for international work.
• Staffing up for CUDA and Benchmark work
  • One month
  • Possible for the high res stereo work to be helped by this
  • Maybe start Maria on this task

Vadim

• This week we continued working on tests, mainly for cxcore and the calibration part in cv, and fixing some found bugs.
  • We’ve got an impressive results on the coverage of cxcore and cv, see the table:
    • http://spreadsheets.google.com/pub?key=tPvCIw1M4fqU9UQCNRTyoLQ&output=html

Here are the details:

1. Testing:
   • Conditional coverage of cvcalibinit.cpp and cvcalibration.cpp has been raised by 8% and 15% respectively Anatoly
   • Implemented tests for cv::projectPoints & cvProjectPoints2, which are important elements of camera calibration algorithm Maria
   • Implemented tests for cv::undistortPoints and cv::initUndistortRectifyMap; raised coverage of cvundistort.cpp to 100/86 Alexey
   • Rewrote unrobust cvSolvePoly (Durand-Kerner algorithm is now used),
   • fixed solvePoly test,
   • extended some other cxcore tests, bringing cxcore coverage to 91/73 Vadim
2. Bug fixing:
   • closed tracs #50, #53, #54, #57, #63
3. A variant of calibration.c sample with the artifical chessboard generator has been added to SVN Anatoly

Victor

• Chessboard detection: a new test for chessboard detection with subpixel accuracy has been implemented.
  • A high-resolution — 4K x 3K — ch image (taken by Radu with canon 40d camera) was run through chessboard detector and then scaled down to 640×480.
  • As a result in a low-resolution image we know corner coordinates with accuracy ~0.15 pixels that we can compare to the results of subpixel corner finder.
• The new test revealed several bugs in find4QuadCornerSubpix that were fixed.
  • The accuracy for both corner detection methods differ from the corresponding accuracy on artificial chessboards.
  • The new function, find4QuadCornerSubpix, shows 45% lower RMS error than findCornerSubpix, 0.43 against 0.78.
    **_High-resolution stereo:_calibrated a stereo pair on chessboard images taken by 2 canon 40d cameras.
  • Calibration shows reasonable results.
  • Getting a good disparity image was a challenge and required parameter tuning (and it still has too many invalid pixels).
  • Running time for stereo correspondence is 88 second per frame.
• Other: the supercomputer group at the local university applies for $100K grant from Russian government on machine learning.
  • They wanted to implement several classifiers and make them high-performance.
  • We agreed that they will submit the code they’ll create to OpenCV (which restricts them not to use any GPL software).
  • As a result, they will work (provided they get the money) on implementing and then optimizing/parallelizing gradient boosting tree classifier and latent svm.

Action Items

Gary

• Copy in coverage
• Run interest point detectors on narrow stereo
• Set up meeting with Suat and Victor to discuss texture plan

Vadim

• To Stefano
  • Dense optical flow
    • Maybe with Stereo
  • Alternative stereo correspondence algorithm
    • Graph cut on GPU
    • Block matching variants

Victor

From last time

Gary

• Get Victor new outlet images
• Multi-scale multi-corner
• Send CMU approach to Victor
• Set up meeting on Texas
  • CC Victor to Dallas
• Edit C++ interface tutorial

Vadim

• SIFT into ROS Maybe send to Gary

Victor

• Write Dallas

James

• Automate rebuild and putting documentation online

Agenda

Consortium Company meeting

Minutes

• Tools to facilitate or eliminate the movement of data back and forth
• Target some high compute apps
• Look at work that has already been done — lots of image processing
  • Some simple noise reduction etc in CUDA SDK
  • MPP ~= IPP but only released in binary
• Make feature list
  • Priority list
    • Have people who have worked on things such as graph cut
• Work on High Res stereo
  • May be able to get a high end stereo rig that is 14 bits and 8M pixel (4Kx2K)
  • Maybe use: ’’Stereo processing by semi-global matching and mutual information_Hirschmuller
• Add texture based recognition And/Or augmented reality
• Others
  • Automatic segmentation
  • Felzenschwalb’s recognition

Action Items

• Link Joe and Texas
  • Wants one
  • Use of NVidia card for Stereo display

Agenda

• GSOC, what do we want
• Consortium progress
• Stefano Fabri
• Next OpenCV Release, March 30th.

Ongoing

• Code coverage
  • Coverage spreadsheet
• Calibration/Stereo
  • Plan of record
• Bugs tracking …

List of wanted additions

• ’’Stereo processing by semi-global matching and mutual information_Hirschmuller
• Segmentation: Might get as intern(?) —
  • Active Segmentation with Fixation. Ajay Mishra , Yiannis Aloimonos, Cheong Loong Fah, iccv 2009
    • Code (Malab and windows at least).
• Texture recognition toolbox
  • Some similarities with MOPED from CMU, but
    • not dependent on SIFT
    • cloud capble
    • accommodate Stereo up front
    • contours
    • flexible objects
  • New texture features:
    • Combine Daisy like idea, or:
      • Scale Invariant Feature Transform with Irregular Orientation Histogram Binning, Yan Cui, Nils Hasler, Thorsten Thorm?hlen, Hans-Peter Seidel
    • With patch duplets described in:Patch-Duplets for Object Recognition and Pose Estimationby ‘’Bjorn Johansson and Anders Moe’’
      • Tested in A Performance Evaluation of Local Descriptorsby ‘’Krystian Mikolajczyk and Cordelia Schmid’’
    • With binarized features as described in_Multiple target localization at over 100 fps BMVC, 2009, S. Taylor and T. Drummond.
      Minutes

• Coverage of
  • cxcore unchanged
  • cv 76/65 → 78/66%
  • cvaux up by 1%
  • calibration coverage 87/80 → 93/83
• Update
  • tests, calibration and stereo correspondence work.
  • Anatoly couple more function tests in calibration pipeline, perspective transformation, compose RT and others
  • Maria started work on stereo correspondence test.
    • Trying to reproduce middlebury framework for stereo correspondence, run all the existing functions (Kolmogorov, BM) through these
• Vadim has read and is ready to implement Hirschmuller’s dense stereo matching
  • parameters pp1 and p2 weren’t listed (will contact the author)
  • compute time and memory: are_O(image size * maximal disparity * 8_)but compute is parallisable
  • Huge images need to be processed by tiles
  • Stereo is probably more pragmatic at HD, not 8mpix due to lens, apperature and just volume of data.
• GSOC
  • Image stitching — put together a streetview trail
  • Denoising, motion stabilization, lighting balance, image enhancement
  • Image collage,
    • segmentation
  • Segmentation in video/video effects
  • Substitutes for SIFT
  • 3D model, silhouetts + strip
• March 30th release
  • Hirschmilleer
  • Documentation
  • Platform testing: Mac 10.6 (highgui uses old carbon. 64 bit doesn’t compile)
    • OpenMP builds
  • Install
  • Update python bindings to new interface
  • Extend python to MLL. Swig bindings are not stable
• Documentation
  • Need regular posting
  • Many functions are not searched for in the C++ documentation
• Make code submission examples
• Sefano Fabri
  • Fast variant of graphcut for stereo
• Consortium, have resumes in

Vadim Report

This week we practically finished working on calibration tests.

• Test coverage progress:
  • cv 76/65 => 78/66
  • calibration 87/80 => 93/83
• See the table here: http://spreadsheets.google.com/pub?key=tPvCIw1M4fqU9UQCNRTyoLQ&output=html

Here are the details:

1. New tests reprojectImageTo3D, composeRT Anatoly
2. New big test of calibrateCamera, which uses artificially generated chessboards, was started Anatoly
3. New big test of stereo correspondence algorithm started that partly reproduces middlebury framework (http://vision.middlebury.edu/stereo/) Maria
4. MSER testing started. mser sample, contributed long ago by Liu Liu, was cleaned up and put as OpenCV sample. Alexey

The work on improving stereo correspondence started.

1. Ported x-sobel pre-filtering and speckle post-filtering from stereolib.c to OpenCV Vadim
2. Studied H. Hirschmuller paper on semi-global stereo matching, readying to implement it Vadim

Bug fixing

1. Fixed 6+ bugs (tickets #53, #69, #72, #87, #88, #89 + a couple of SF bugs)

New Capability:

• Maria modified grubcut demo. Now you can set “likely foreground” and “likely background” pixels by pressing right mouse button with shift/ctrl
  • “surely foreground” and “surely background” are set by pressing left mouse button with shift/ctrl. The functionality is there.
  • While “likely foreground” does affect the result, setting “likely background” usually does not, because the number of pixels added to the background model is too small, comparing to the whole mass of pixels outside the ROI.
  • Perhaps, some weights should be used with these extra pixels, but the current API does not support that yet.

Action Items

Gary

• Edit C++ interface tutorial

Vadim

Victor

From last time

Gary

• (./) Copy in coverage
• (./) Run interest point detectors on narrow stereo
• :( Set up meeting with Suat and Victor to discuss texture plan
• :( Edit C++ interface tutorial

Vadim

• To Stefano
  • Dense optical flow
    • Maybe with Stereo
  • Alternative stereo correspondence algorithm
    • Graph cut on GPU
    • Block matching variants

Victor

Agenda

• Stereo analysis between stereolib in ROS and OpenCV’s stereo algorithms

Minutes

Vadim

My findings are the following:

• differences between OpenCV and stereolib are mainly due to the parameter interpretation:
• OpenCV uses CvStereoBM::textureThreshold as is.
  • stereolib transforms it to state→textureThreshold * state→SADWindowSize * state→SADWindowSize * state→preFilterCap / 100;
    OpenCV uses CvStereoBM::uniquenessRatio (denoted uthresh0 below) as a ratio in percentage: *uthresh0/100
    • stereolib basically uses a half of that – in the beginning of the function it transforms the parameter as:
      • uthresh = uthresh0 * 0×8000 / 100; ~ uthresh * (1 << 15) / 100
        • and then it uses SSE instruction equivalent to
        • uthresh >> 16, which is approximately equal to uthresh0/200
• So if you call OpenCV function with the scaled textureThreshold and uniquenessThreshold, then the results become very similar visually.
  • Numerically they are a bit different, because of different implementation of subpixel interpolation, OpenCV uses script algorithm, while stereolib uses some approximation.
  • Still, on Tsukuba only in 12% of pixels the difference exceeds 8 (~half-a-pixel, since the disparity is scaled by 16) and all those pixels are at the object boundaries, where the algorithm is not accurate anyway.

Regarding the processing speed.

• When OpenCV is built without OpenMP support, stereolib is ~15-20% faster (with speckle filtering turned off, since it’s the same code in both cases). I think, stereolib is faster because it does not handle the image border and because it uses faster subpixel interpolation.
• When OpenCV is built with OpenMP support, it is ~40-50% faster than stereolib (with speckle filtering turned off).
• To enable SSE2 (and OpenMP), you can run CMake with flags USE_SSE2=ON and ENABLE_OPENMP=ON.

Action Items

• Vadim to move on to implementing ‘’Stereo processing by semi-global matching and mutual information Hirschmuller’’

Agenda

• Outlets
• Features
• Stefano convergence to a plan
• GSOC
  • Put another way, what would you be willing to mentor? Name your topic.
• Switching OpenCV license from BSD to Apache (see also http://en.wikipedia.org/wiki/Apache_License). The reason is protection from lawsuits:
  • If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed.
    • Also known to those “skilled in the art” as “the nuclear option” — you sue someone over something in OpenCV, you lose your license to OpenCV.
    • Issue on how to convert?
• Also, there is a copyright issue with externally donated code — we don’t necessarily own the copyright, just the license to use. This would involve setting up an OpenCV foundation and having anyone who contributes give their copyright to the foundation.
  • The reason for this is a bit murky — by giving it under BSD they are allowing its use according to license, but they still own the actual writing.
  • I don’t know what it means in practice.

Ongoing

• Code coverage
  • Coverage spreadsheet
• Calibration/Stereo
  • Plan of record
    • {i} When do we call this ’’"closed"’’?
• Bugs tracking …

List of wanted additions

• /!\ ’’Stereo processing by semi-global matching and mutual information_Hirschmuller
• Segmentation: Might get as intern(?) —
  • Active Segmentation with Fixation. Ajay Mishra , Yiannis Aloimonos, Cheong Loong Fah, iccv 2009
    • Code (Malab and windows at least).
• /!\ Texture recognition toolbox
  • Some similarities with MOPED from CMU, but
    • not dependent on SIFT
    • cloud capble
    • accommodate Stereo up front
    • contours
    • flexible objects
  • /!\ New texture features:
    • Combine Daisy like idea, or:
      • Scale Invariant Feature Transform with Irregular Orientation Histogram Binning, Yan Cui, Nils Hasler, Thorsten Thorm?hlen, Hans-Peter Seidel
    • With patch duplets described in:Patch-Duplets for Object Recognition and Pose Estimationby ‘’Bjorn Johansson and Anders Moe’’
      • Tested in A Performance Evaluation of Local Descriptorsby ‘’Krystian Mikolajczyk and Cordelia Schmid’’
    • With binarized features as described in_Multiple target localization at over 100 fps BMVC, 2009, S. Taylor and T. Drummond.
      Minutes

Outlets

• Latest version didn’t build. But today it does compile
  • Patrick should contact Victor when the node is ready and Victor will make sure the detector works
  • Collect new bag files to be sure
• Outlets will be white 2×1
  • The outlet detection is 180 degree rotation invariant, or else it is a bug
  • Robot should be able to rotate the plug — Victor thinks there should be distinguishable chessboard on both sides.
• When we tested, as measured in the camera frame, X,Y < 1mm error with random short lived blibs to 2mm lighting dependent
  • Error to ~1mm if projected to the wall plane,
  • Z accurate to +/-5mm in the wall plane
  • This suggests that you do purely relative ray fly in using readings in the image plane (chessboard location vs hole location)
    • Detect contact when near the real Z

• Features
  • Dataset generator
    • Works by finding camera extrinsics (PnP) to the points, build a box relative to that
    • Look for readme file
• Licensing
  • Apache # 0 is good for the anti-lawsuit option
    • but it not compatible to GPL 1.0 and # 0 (probably), so we might be stuck
    • We probably will not change
  • Copyrights to code
    • When submitting code, turn over the copyright to OpenCV
    • But the text is also covered by BSD
    • Datafiles can be an exeption
• Stefano
  • Image stitching
  • Variation of SIFT
    • Scale Invariant Feature Transform with Irregular Orientation Histogram Binning
      • Yan Cui, Nils Hasler, Thorsten Thorm ?hlen, Hans-Peter Seidel
      • ICIAR 2009
• Want a test framework for measuring feature point performance
  • Michali Cordelia Schmid affine invariant descriptors
    • Apply affine transform (know ground truth)
  • Lowe 3D→2D
    • Can do with Gazebo
  • Want to build this toolbox
• GSOC
  • Need mentors
    • Recognition toolbox
• Code coverage went up 1% for calibration … mostly due to the fact that stereo correpsondence tests wasn’t finished yet
  • Find homography should have tests (C++ function calls the C) … all the geometric stuff
• Documentation auto-generate and put up
  • Add more on C++

Priorities going forward

1. Outlets for the next 2 weeks
2. Textured object recognition
3. Stereo is high priority
   • Someone allocated to stereo
     • Hirschmuller (Vadim thinks this week for draft version)
       • Multi-window version improvement … correlation based
     • Hierarchical method
     • Graphcuts
4. Documentation
   • Higher priority than tests now (as per user group poll)
5. Texas help
   • Color blobs need to be more stable.
6. 3D tools — easy to compute PnP, rotate, back project, render into a scene
   • We have most of these tools, but they are not in a coherent framework
7. Testing
   • Stereo correspondence
   • All new code goes in with tests
   • Incremental improvements

Vadim

1. Regression test on MSER has been added Alexey
2. Finished big camera calibration test (monocular case for now) that uses artificially generated chessboards for random camera intrinsic parameters Anatoly. Here are our findings:
   • camera matrix is estimated reasonably well (the relative error is within ~1-3%) especially when the aspect ratio is fixed and/or the principal point is explicitly centered.
   • k1, p1 and p2 distortion coefficients are also estimated rather precisely.
   • k2 and k3 estimates are actually terrible. Sometimes they are 2x, 3x … 10x from the actual values, even if the chessboard finder is not invoked but the projected corners are passed directly to the calibration procedure (with very very little noise added). Possible reason is that when most of the board views are somewhere in the image center rather than near the border, those k2 and k3 are multiplied by very small numbers (r^4 and r^6, respectively, where 0<=r< ~0.5-1), so it’s difficult to estimate them precisely.
   • translation vectors are estimated pretty accurately.
   • rotation vectors are estimated pretty well too when the board is sufficiently close to the camera.
3. We can conclude that in order to estimate distortion parameters precisely we probably need to revise the calibration methodology. The calibration pattern should better cover almost the whole frame and there should be many corners in each view, especially near the image border, where the relative weight of k2 and k3 is larger. May be we should try to detect board edges rather than corners and try to find such distortion coefficients so that the lines become straight.
4. Reproduced middlebury framework, various quality metrics. Started StereoBM integration and parameter tuning Maria
5. Fixed automated building of ROS packages dependent on OpenCV in our buildbot (our scripts broke at some point after the directory structure in ROS was modified) Anatoly.
6. OpenCV’s StereoBM implementation was compared in-depth with the latest version in ROS. The found differences have been mostly eliminated by adjusting CvStereoBMState parameters. Vadim
7. Some draft implementation of H. Hirschmuller algorithm is ready, started debugging and testing Vadim

Code coverage. calibration dynamic (aka conditions) coverage was improved by 1% this week (from 83 to 84), otherwise, there are no other changes:
http://spreadsheets.google.com/pub?key=tPvCIw1M4fqU9UQCNRTyoLQ&output=html

Misc: Alexey Latyshev is on vacation for the whole this week (Feb 1-7)

Action Items

Gary

• Edit C++ interface tutorial
• (./) Tell Patrick to tell Victor when he’s done modifying the outlet package
• Generate lots of new bag files in different conditions
  • Oriented in each direction
  • Think of what you want to mentor for GSOC
• (./) Have Suat send code to Victor
• Get auto-generated docs put up to wiki every night

Vadim

• Find out what Stefano means in terms of mentoring
  • Probably work in terms of image stitching
  • Maybe student can build ground truth test set on natural scenes
• Think of what you want to mentor for GSOC

Victor

• Think of what you want to mentor for GSOC

From last time

Gary

• :( Edit C++ interface tutorial

Vadim

• (./) To Stefano

Victor

Agenda

• Hirschmuller/stereo
• Outlets
  • Fast grid detector
• Recognition toolbox
  • Need a new dataset for tracking
• GSOC
  • Meet with Jean-Yves about stitching
• Kurt(?)

Ongoing

• Code coverage
  • Coverage spreadsheet
• Calibration/Stereo
  • Plan of record
• Bugs tracking …

List of wanted additions

• Stereo processing by semi-global matching and mutual information_Hirschmuller
• Segmentation: Might get as intern(?) —
  • Active Segmentation with Fixation. Ajay Mishra , Yiannis Aloimonos, Cheong Loong Fah, iccv 2009
    • Code (Malab and windows at least).
• Texture recognition toolbox
  • Some similarities with MOPED from CMU, but
    • not dependent on SIFT
    • cloud capble
    • accommodate Stereo up front
    • contours
    • flexible objects
  • New texture features:
    • Combine Daisy like idea, or:
      • Scale Invariant Feature Transform with Irregular Orientation Histogram Binning, Yan Cui, Nils Hasler, Thorsten Thorm?hlen, Hans-Peter Seidel
    • With patch duplets described in:Patch-Duplets for Object Recognition and Pose Estimationby ‘’Bjorn Johansson and Anders Moe’’
      • Tested in A Performance Evaluation of Local Descriptorsby ‘’Krystian Mikolajczyk and Cordelia Schmid’’
    • With binarized features as described in_Multiple target localization at over 100 fps BMVC, 2009, S. Taylor and T. Drummond.

Minutes

• Hirschmuller doesn’t work as well in low texture
  • Consumes a lot of memory, questionable even on HD to 100’s of megabytes
  • Need to process tiles but then loose the semi-global property and so loose context in low texture scenes
  • Use Kurt’s differences between box, will speed up algorithm
    • Will work better in low texture and large resolution images
  • Want to try with Robot’s textured stereo at 640×480
    *_Outlets_
  • New bag files are about 100% detection
    • Companding (bit compression) works better
  • Pose priors
    • Unstable: angular degrees of freedom hard to estimate except roll (rotation around Z)
    • Want to use hints to restrict the angular pitch and yaw in the pose estimation
      • Represented by rotation vector … not roll pitch raw
        • Angle axis harder to restrict
      • Find outlet, run FindExtrinsicParams, Rays will go through outlet holes. Can use ray piercing
      • If you know the outlet size, then piercing can give you depth by scaling
    • Latest code that works with new bags is checked in
  • Parameters
• Texture recognition toolbox
  • Document, so far internal https://docs.google.com/a/willowgarage.com/Doc?docid=0AW1WrefIjUo-ZGRzeHZxZmRfNWd2a3A3cWZr&hl=en&invite=CPLmi4cF.
  • Scalable
    • Extremely random trees
    • Vocab tress
    • Fast NN
  • Want to do
    • Datasets — real ground truth tracking
      • Rendered objects like Lowe
      • Large data set of labeled object
    • Test engine with all current and growing number of features
    • Texture object challenge, announce at OpenCV
    • Challenges can be challenging … with resources
      • Get an intern to do full time during the challenge
• GSOC
  • Will talk to J-Y about stitching
• Tweet more

Victor

• Outlet detection: analyzed about 5 bag files created by plugin_sprint team. Outlet holes are saturated for the earlier bag files (2010-02-02), causing noisy detection.
  • Also gamma parameter had to be changed for these holes to make one way descriptor work correctly.
  • Latter bag files (2010-02-08) have much nicer views of an outlet, all holes are very high contrast and can be easily detected (with gamma changed pack and increased scale).
  • All 6 datasets have been labeled for easier retrieving of detection statistics (Alexey)
• Started object recognition toolbox document to put together all the parts.
• Conducted experiments with Radu’s stereo pair using block matching and graph cuts, Vadim has taken this over as I got dragged back to outlet detection.
• Found a problem with loading outlet.launch due to an exception in reading outlet template. This is reproduced on exactly one machine, still investigating the root cause.
• Graphics project status: pending contract approval by nVidia
• Benchmark project status: negotiating SOW, tentative start date March 1st.

Vadim

1. Hirschmuller algorithm (SGM – semi-global [stereo] matching) has been finished and tested Vadim
   • For now the mutual information cost function is not implemented, instead, the simpler Birchfield & Tomasi cost function is used.
   • On scenes with a rich texture it works very well, and gives noticeably cleaner and sharper disparity maps, comparing to the block matching algorithm. The current speed is on Tsukuba is 350msec on my # 4GHz Core2Duo laptop (single-thread C code). I believe, with SSE the running time can be reduced downto 80-100msec for single-threaded version on the same hardware – the core loop operates on 16-bit integers; besides there are many min/max operations, for which there are specialized batch SSE instructions. Block matching algorithm runs in ~40msec on the same machine (single-threaded SSE2 code).
     • On areas with no texture the algorithm can fail. Moreover, since the algorithm takes a lot of memory ( O(*)), sufficiently large images (like 1024×768 or higher) can only be processed by tiles (i.e. the “semi-global” algorithm does only use the information inside the tile), which should overlap for more robust behaviour. The running time increases because of overlapped tiles, and yet the wrong depth maps can still come out. Using color information helps, but not completely solves the problem.
     • Attached are results of the 2 algorithms on Tsukuba (stereo BM was executed with 7×7 and 17×17 block sizes) and the stereo pair that Radu provided. The original 10mp image was too much for both of the algorithms: StereoBM had too many options to choose from (since the blocks are matched independently), and the tiny tiles in SGM just did not include enough context for robust estimation. After reducing the image by factor of 4 on each dimension (=> 16x smaller area) the algorithms started to produce reasonable output (with maximum disparity == 128).
       • Currently I’m trying to create some intermediate 1-pass algorithm (SGM is 2-pass algorithm) that will use the same cost function as in Kurt’s algorithm (per-block absolute difference between clipped x-sobel derivatives, with optional color support), but will also use context information from the other blocks on the same line and from the line above (i.e. the first pass from Hirschmuller algorithm). This hybrid algorithm consumes very little memory (slightly more than BM algorithm) and at the same time should not be much worse than the original algorithm. I tried to disable the second pass in the original Hirschmuller algorithm, and on Tsukuba it still produces quite decent results.
2. Finished stereo test. All the test data from Middlebury framework is added, StereoBM algorithm has been added into the test. Maria
3. Fixed several build problems and other bugs: tickets #74, #91, #101, #10# Anatoly
   • Found some more bugs, filed 2 new tickets: #113, #11#
4. Code coverage was improved a bit, the new results are available here: http://spreadsheets.google.com/pub?key=tPvCIw1M4fqU9UQCNRTyoLQ&output=html
5. Stereo matching results:
6. Our raw stereo, 17×17
7. Raw color
8. Tsukuba 17×17
9. Tsukuba 7×7
10. Tsukuba color

Action Items

Gary

• Edit C++ interface tutorial
• Get textured light images to Vadim

Vadim

• Get Hirschmuller point clouds to Radu
• Run on textured light

Victor

From last time

Gary

Vadim

• (./) To Stefano
  • Dense optical flow

Victor

Agenda

• GSOC
  • Mentors
    • Background subtraction, feature based tracking
      

      
      
      Nicolas Saunier, Ph.D.
      
      Professeur Adjoint / Assistant Professor
      
      Departement des genies civil, geologique et des mines (CGM)
      
      Ecole Polytechnique de Montreal
      
      http://nicolas.saunier.confins.net
      
      

    • Implement some well known CV algorithms.
      

       Mark Asbach
      
      Institut fur Nachrichtentechnik
      
      RWTH Aachen
      
      D-52056 Aachen
      
      

    • Image stitching and/or image collage *
      

       Gary Bradski
      
      Senior Scientist, Willow Garage
      
      Consulting Prof. Stanford U.
      
      OpenCV Founder, Technical Content Owner
      
      

    • ?
      

      
      
      Vadim Pisarevsky
      
      OpenCV founding team/Czar
      
      

    • ?
      

      
      
      Victor Eruhimov
      
      OpenCV founding team/Senior Researcher
      
      Argus/Itseez founder
      
      

• Stereo algorithm progress
• Bugs progress … when to call a code freeze to do bug fix and documentation spruce up?
• Plug in reports
• Texture based object recognition
• BG segmentation
  • Have myCreateFGDStatModel(), myUpdateBGStatModel(), and myRleaseBGStatModel() and one data structure MyBGStatModel
    • Notably, we are missing myBGStatSegment which just does foreground segmentation, not learning and segmentation.

Ongoing

• Code coverage
  • Coverage spreadsheet
• Calibration/Stereo
  • Plan of record
• Bugs tracking …

List of wanted additions

CODE FREEZE: March 9th

• Get in GSOC application.
• (./) ’’Stereo processing by semi-global matching and mutual information_Hirschmuller
• Segmentation: Might get as intern(?) —
  • Active Segmentation with Fixation. Ajay Mishra , Yiannis Aloimonos, Cheong Loong Fah, iccv 2009
    • Code (Malab and windows at least).
• Texture recognition toolbox
  • Some similarities with MOPED from CMU, but
    • not dependent on SIFT
    • cloud capble
    • accommodate Stereo up front
    • contours
    • flexible objects
  • New texture features:
    • Combine Daisy like idea, or:
      • Scale Invariant Feature Transform with Irregular Orientation Histogram Binning, Yan Cui, Nils Hasler, Thorsten Thorm?hlen, Hans-Peter Seidel
    • With patch duplets described in:Patch-Duplets for Object Recognition and Pose Estimationby ‘’Bjorn Johansson and Anders Moe’’
      • Tested in A Performance Evaluation of Local Descriptorsby ‘’Krystian Mikolajczyk and Cordelia Schmid’’
    • With binarized features as described in_Multiple target localization at over 100 fps BMVC, 2009, S. Taylor and T. Drummond.

Minutes

Stereo

• Hirschmuller results when mixed with block matching
  • Much less memory use — can handle 10Mpix images no problem
  • Works better on lower texture
  • On Tsukuba, results are good except for boundaries, haven’t tested exact results yet
    • Will get exact comparisons when its integrated into the test framework
  • Faster than original algorithm, much faster than graph cuts, but 10x slower than block matching
    • Can probably speed up by factor of 4-6x
• Need to create a test set using Willow’s stereo
  • Need to finally get beyond the very limited Middlebury stereo
  • Maybe use laser pointer to align 3D scanned object with stereo imagery
  • Possibly use the “Borg Scanner” at Stanford

Bugs

• When to call a code freeze?
  • March 31st release
    • So March 9th will start release process
      • Bug fix
      • More test code
      • Documentation advance

GSoC

• Will have to get clear mentor statements
• Gary will work on application, putting up wiki
• SOW should be clear
• Track by email, twitter

Plugging In

• See Victor’s report below
• Plugging in worked well
• Need to generalize the outlet detection
• Have to add in my gradient grid features
  • Has to be stable to scale change
    • Geo matching is invariant to scaling
  • Has to be invariant to affine transform
• Make learning the outlet quick and general
• Far outlet detection
• Door handle

Textured object recognition

• I’m working on new features for that
• Suat
  • Look at his code
• Need
  • Recognition pipeline
    • Sparse
    • Grid
• Start with data
  • Specific tasks:
    • Sparse textured recognitino
    • Transparent
    • Textureless
    • 3D textured data
• Dataset generator
• Test set generator

Background

• Put it into a 3D capture framework with addition of grabcut
• Generalizing the calibration object
• Security

Vadim

1. Some fusion of Hirschmuller and Kurt algorithms (called SGBM – semi-global block matching) has been created Vadim
   • On the one hand, SGBM is similar to the Kurt’s algorithm:
     • the square blocks centered at each pixel are matched, rather than individual pixels. the sum are computed using sliding windows, so the overall complexity is O(W*H*D), where W is the image width, H – height, D – maximum disparity
     • x-sobel is used to emphasize the texture.
     • the whole processing is done with a single pass through the image, so the amount of consumed memory is O(W*D), rather than O(W*H*D) in the original Hirschmuller algorithm. Therefore, no tiling mechanism is needed, the whole image is processed within the same loop.
     • the best matches are verified using the uniqueness threshold, i.e., they must be significantly better than the second-best matches.
     • after the disparity is computed, the speckles are filtered off using the same procedure as in StereoBM.
   • On the other hand, SGBM is similar to Hirschmuller’s algorithm:
     • dynamic programming is used to find the best disparity for each pixel. Unlike the original algorithm, SGBM is single-pass algorithm, thus it only does the real dynamic programming optimization in each row, and it also tries to minimize the difference between the subsequent rows,
       • i.e. it is greedy algorithm in the vertical and diagonal directions.
     • the individual pixels are matched using subpixel-accurate Birchfield metrics, rather than just simple absolute difference.
     • the reverse image2→image1 disparity map is computed at the same time as the normal one, and then used to find occlusions.
   • I also added support for color images, that improved disparity accuracy on Tsukuba and Radu stereo pairs.
   • The algorithm runs about the order of magnitude slower than StereoBM but much faster than the original algorithm. Also, a significant speedup is possible, the algorithm is very SIMD-friendly.
2. Some initial experiments with Intel TBB have been done, StereoBM has been rewritten using TBB Anatoly
   • These are running times on different stereo pairs:

• As you can see, except for the pair #5 TBB is on-par or even a bit faster than OpenMP.

Victor

• Outlet detection:
  • tested on several bags, plugin_sprint needed detection for frontal views and it worked fine after an increase in scale search region,
  • 100% accuracy, 0 false alarm rate.
  • A method for tracking parameters (for speeding up scale search by narrowing the search region after a successful detection) based on Kalman filter has been implemented (Alexey).
  • Several experiments on a possibility of eliminating one way descriptor (that is sensitive to camera parameters and [somewhat] to lighting conditions) have been conducted.
  • Matching unclassified keypoints causes several false positives that can be eliminated by additional information (edges, regions).
    • Right now a geometric matching method that detects a set of keypoints together with a set of uniform regions have been implemented and is being tested.
      *_Plug detection:_
  • tested on multiple bag files collected during plugin_sprint,
  • fixed a bug and implemented an improvement for low resolution chessboard detector.
  • The latest version works for all bags but two where some of the black squares are not detected due to low lighting.
    • This can be fixed but I suggest to change the pattern from chessboard to a set of circles for more robust detection.

Action Items

Gary

• GSOC application
• Stereo textured images to Vadim
• Find out about making ground truth set. Maybe talk Morgan at Stanford to get his scanner for making ground truth stereo
• Get Suat’s code to Victor
• Write recognition framework
• (./) Get texas access for Argus
  • Scott doesn’t want to invest or maintain the old system while working on the new, so he’d rather they not be used much at Willow until the new ones are created.

Vadim

• See if we can get the improved Background subtraction code into opencv
• Try Hirschmuller on textured light stereo images (try pure Hirschmuller and the modified variant).
  • Return point clouds for Rudu to look at

Victor

• Look over Suat’s code

From last time

Gary

• :( Edit C++ interface tutorial
• :( Get textured light images to Vadim

Vadim

• :( Get Hirschmuller point clouds to Radu
• :( Run on textured light

Victor

Agenda

• Stereo issues with Kurt
  • Calibration GUI tool
• Sparse Pose and Sparse Bundle Adjustment (SPA & SBA) — Kurts algorithm in
• Object Recognition Stack
  • Textured objecs
  • Scalable recognition
  • Geometric confirm.
• Bugs progress … when to call a code freeze to do bug fix and documentation spruce up?
• Plug in, might need minor adjustments, but seem OK for now
• BG segmentation
  • Have myCreateFGDStatModel(), myUpdateBGStatModel(), and myRleaseBGStatModel() and one data structure MyBGStatModel
    • Notably, we are missing myBGStatSegment which just does foreground segmentation, not learning and segmentation.

Resource Allocation

1. ObjectRecognitionStack I’ve created a wiki page: http://pr.willowgarage.com/wiki/ObjectRecognitionStack, This probably heavily involves Victor on the texture recognition and pose toolbox and will pretty much start now. Victor, Me, Suat (starting in April)
   • I’m hoping/assuming that the outdetector stuff is table, or is rebuilt as a subset of the texture recognition and pose toolbox.
2. Kurt wants someone to interact (aka deliver) ports of his sparse bundle adjustment and sparse pose adjustment algorithms. Yes we have some sort of bundle adjustment — Kurt’s stuff probably runs >10x faster. This is the core of visual odemetry. Who?
3. Stereo — running the textured light 640×480 on pure Hirschmuller, want to see point clouds. Running BM, BM+Hirschmuller and Hirschmuller on the 1Mpixel images from Kurt. Return point cloud. After this, I suppose test code for Hirschmuller and integrate. Vadim
   • Probably we want a full calibration GUI that goes on up through stereo and rectification.
4. Texture grid — solving problems like doorhandles, far outlets etc. Gary
5. Code freeze March 9, OpenCV v# 5 March 31st. [Will need bug clean, more doc, more test,All, but who is main?]

Ongoing

• Code coverage
  • Coverage spreadsheet
• Calibration/Stereo
  • Plan of record
• Bugs tracking …

List of wanted additions

CODE FREEZE: March 9th

• Get in GSOC application, post it http://opencv.willowgarage.com/wiki/GSOC_OpenCV2010.
• (./) ’’Stereo processing by semi-global matching and mutual information_Hirschmuller
• Segmentation: Might get as intern(?) —
  • Active Segmentation with Fixation. Ajay Mishra , Yiannis Aloimonos, Cheong Loong Fah, iccv 2009
    • Code (Malab and windows at least).
• Texture recognition toolbox
  • Some similarities with MOPED from CMU, but
    • not dependent on SIFT
    • cloud capble
    • accommodate Stereo up front
    • contours
    • flexible objects
  • New texture features:
    • Combine Daisy like idea, or:
      • Scale Invariant Feature Transform with Irregular Orientation Histogram Binning, Yan Cui, Nils Hasler, Thorsten Thorm?hlen, Hans-Peter Seidel
    • With patch duplets described in:Patch-Duplets for Object Recognition and Pose Estimationby ‘’Bjorn Johansson and Anders Moe’’
      • Tested in A Performance Evaluation of Local Descriptorsby ‘’Krystian Mikolajczyk and Cordelia Schmid’’
    • With binarized features as described in_Multiple target localization at over 100 fps BMVC, 2009, S. Taylor and T. Drummond.

Minutes

Stereo

• Hirschmuller is faster
  • Seems better on lower texture, you get Real Hirschmuller > modified Hirchmuller > BM
• Need to create the inscribed rectangle where there are only valid disparities between the left and right camera. The statitics for this rectangle can be kept throughout the process.
  • Top you find the lowest non-valid pixel and that bounds the top.
  • In Kurt’s code, these are dtop, dbottom,
  • Can we put the left-right check from Hirschmuller to BM … but hurts efficiency.
  • Kurt will get a bag … want to study L-R algorithm on it

SBA

• SBA is the back end bundle adjuster
• Need a front end match “proposer”.
  • Patrick has Nistor trees
• Need sub-class detector/descriptor
• Even more speedup with more careful dataflow analysis. Semi-Hirschmuller

Detector/Descriptor

• Need to get a design draft of how to put in our detectors/descriptors
  • Good sub-matcher
  • Then can put into nodelettes style as per PCL (Point Cloud Library).

Outlet

• Plane Optimizer
  • Victor made this, but can’t get pose from TF due to timestamp (?) problem
    • Frame should be published in the bag file, but Victor isn’t seeing it
• Now we’re going to plug in sideways to shorten the outlet cord

Resources

• Maria returns March 2nd, spend on release
• Kurt’s task, fusing with descriptors is within Victor’s
• Optimization person is in play (Alexi S.) … Can Willow get one more
• Sparse Bundle Adjustment delay integration post release?
• Graphics — approved contract today. If so, start March 1st. People are in place to switch over so this is OK
• CPU — SOW is approved. Tentative date is March 1st. Probably delayed to later in March.
  • CPU legal put in boilerplate “closed” license. Needs to be changed.

Background Subtraction

• Vadim look over to see if it’s easy to put in foreground only code (without causing the algorihtm to continue learning) into the Rainer BG framework.

Vadim

1. semi-Hischmuller algorithm has been optimized by more than the order of magnitude (!) Computing correspondence for the stereo pair from Radu took # 5s in the previous version; now it takes ~350msec (640×480) on the same laptop. Some further speed improvement is certainly possible, because the algorithm does not do very much extra work comparing to the block matching algorithm. I expect that the single-threaded version can probably process the Radu stereo pair in ~100ms or even faster. BM algorithm process the stereo pair in ~40ms. Vadim
2. Performance of the OpenCV implementation of the BM algorithm from stereolib was improved by ~20% and now it is as fast as the original sterelib code (in a single core; on dual- or quad-core machines it’s significantly faster) Vadim
3. Some first experiments with the new images from Radu & Kurt have been done. The results are surprisingly bad :(, both with BM and semi-Hirschmuller algorithm. Another surprise is that on the pictures without textured light the results look better! The results are attached. The possible reason is too severe rectification applied that distorts the proportions. The green rectangle is the area where disparity should be valid (by the request from Kurt). Vadim
4. Integration of TBB continues. Several improvements in TBB-ed StereoBM have been done, so that no repeated memory allocations are invoked during the processing. The gap between OpenMP and TBB is now much smaller, although the OpenMP version still adapts better to the number of CPUs Anatoly
5. 3 machine learning-related trac tickets (#56(SVM), #64(SVM), #129(Boost)) have been closed Maria

Example of new stereo results on Mpix camera (no projected texture):

• Old Block Matching Method very fast, fraction of a second
  
• New, semi-Hirschmuller results.
  • Note that it is better but probably ~0.5 sec
  • We should probably think of publishing this — it’s a tradeoff between extremely fast and memory efficient BM and very slow and huge memory hog Hirschmuller (GB per pair?). Semi-Hirschmuller is memory efficient, pretty fast and certainly better than BM in results.
    
  • Misc: Maria is on vacation from last Tuesday till March 2nd.
  • Plans: finish TBB integration, try real Hirschmuller on Radu data, start preparations to the code freeze

Victor

Outlet detection:

• Added a region-based filtering to geomertical model of an outlet. A set of points is sampled randomly from the largest uniform region in the training image and then mapped into the test image using the hypothesized affine transform. Deviation of intensities on this point set is used as a filtering criterion. This still does not allow to get rid of one way descriptor, but allows higher misclassification rate of keypoints with small overhead.
• Pose estimation: A function for estimating a pose of a planar object with a given object normal is implemented. This problem is solved in closed form by projecting rays into target plane and finding the distance to the plane by minimizing RMS error. The function still has to be tested on real bag files, but I still haven’t succeed in getting outlet priors working.
• Object recognition: Started to prototype a generic interface for descriptors. The nearest goal is to make several descriptors implemented in OpenCV interchangeable on outlet detection problem.

Action Items

Gary

• GSOC application
• C++ tutorial
• C++ docs still don’t read functions
• Get new plug bag files to Victor

Vadim

• Test L-R constraints on Kurt’s stereo images
• Re-run Hirschmuller, semi-Hirschmuller and BM on textured 640×480 stereo images that have been properly rectified.
• Look over Rainer’s Background Subtraction code to see if it’s easy to allow foreground segmentation only separate from learning. If so, put it in, if not, we’ll delay this functionality.

Victor

• Look over Suat’s code
• Work with Patrick/Radu to architect interest point detector/descriptors

Radu

• Suat’s code to Victor and Patrick.
• Work with Patrick/Victor for deciding how interest point detectors/descriptors should be sub-classed (especially to work easily with PCL).

Patrick

• Work with Victor/Radu to architect interest point detector/descriptors

Kurt

• Send Vadim bag/images to test L-R constraints

James

• Can we get the C++ functions to be searchable in the Wiki … otherwise we’re going to have to do some Python code spelunking ourselves.

From last time

Gary

• :\ GSOC application
• Stereo textured images to Vadim
• :\ Find out about making ground truth set. Maybe talk Morgan at Stanford to get his scanner for making ground truth stereo
• Get Suat’s code to Victor
• Write recognition framework
• (./) Get texas access for Argus
  • {X} Scott doesn’t want to invest or maintain the old system while working on the new, so he’d rather they not be used much at Willow until the new ones are created.

Vadim

• See if we can get the improved Background subtraction code into opencv
• Try Hirschmuller on textured light stereo images (try pure Hirschmuller and the modified variant).
  • Return point clouds for Rudu to look at

Victor

• Look over Suat’s code

Agenda

• GSOC application due by the 8th …, need to fill in more exactly what people want to mentor.
  • OpenCV GSOC 2010
• March 31st release
  • What do we want in/out?
  • Report on the stupid C++ wiki lower case function name search bug
• We had a visit with Ajay Mishra, ‘’Active Segmentation with Fixation’’. We (Willow) might try doing a joint project together to be defined, but perhaps pulling items out of clutter.
• Plug in reports
  • What goes into OpenCV from this?
• Texture recognition toolbox
• SOW work
  • GPU
  • CPU

Ongoing

• Code coverage
  • Coverage spreadsheet
• Calibration/Stereo
  • Plan of record
• Bugs tracking …

List of wanted additions

CODE FREEZE: March 9th

• Get in GSOC application.
• (./) ’’Stereo processing by semi-global matching and mutual information_Hirschmuller
• Segmentation: Might get as intern(?) —
  • Active Segmentation with Fixation. Ajay Mishra , Yiannis Aloimonos, Cheong Loong Fah, iccv 2009
    • Code (Malab and windows at least).
• Texture recognition toolbox
  • Some similarities with MOPED from CMU, but
    • not dependent on SIFT
    • cloud capble
    • accommodate Stereo up front
    • contours
    • flexible objects
  • New texture features:
    • Combine Daisy like idea, or:
      • Scale Invariant Feature Transform with Irregular Orientation Histogram Binning, Yan Cui, Nils Hasler, Thorsten Thorm?hlen, Hans-Peter Seidel
    • With patch duplets described in:Patch-Duplets for Object Recognition and Pose Estimationby ‘’Bjorn Johansson and Anders Moe’’
      • Tested in A Performance Evaluation of Local Descriptorsby ‘’Krystian Mikolajczyk and Cordelia Schmid’’
    • With binarized features as described in_Multiple target localization at over 100 fps BMVC, 2009, S. Taylor and T. Drummond.

Minutes

• (James)Python bindings
  • There are too many python bindings (6?) … need to unify. Start over?
  • Swig covers more but isn’t as stable
  • Current bindings are good, but don’t support C++
    • machine learning is not supported
    • HOG, Calandar not covered
  • Could make a C layer to wrap C++ (which sometimes wraps the C) to make Python easier
  • Sample code with current wrappers requires explicit allocation
    • It would be nice to get rid of this
  • James will look today at the C++ interface
  • Numpy array interface looks like its working fine
  • Can use numpy arrays as if they were cvmat
  • cvmat can be missaligned
  • Would be good to have numpy as src and dst
• Eigen
  • Support can be added
  • 2 weeks eigen
  • 2 weeks numpy
• Victor strongly for release
• Vadim right after release, can’t make large changes
• Kurt Region of interest now fixed for stereo Vadim
  • Need to call this explicitly
  • Kurt thinks this should be the default
  • Will need to change way stereo is called
  • Hirschmuller looks good
    • What happens in the L-R check. Doesn’t impose 2x cost
    • Not commited yet
    • Send message Vadim
      *_Plug in_
  • New normal bias in estimation in
  • Noise is 2x lower ~0.6mm
  • Still want to put it in, see if the bias per socket still needed
  • Closed form solution
    • Minimize re-porjection in object
    • Kurt but do you know the covairance in object plane .
      • Might be a matter of just rescaling in Z
      • Covariance in image plane needs to be reporjected into socket plane
      • Why not use standard numeric optimization
      • Worry about the covariance
      • Coordinate with Wim and Patrick
      • Need to work with the trunk version of vision_opencv
        • Might need to branch
          *_Features_
  • Design of API
  • Need to converge design
  • Haven’t seen Suat’s code
  • Need consistent API
  • MLL as example, tried to unify but failed
    • handle missing inputs on decision trees
  • Scalable
    • Approx NN
    • extremely random trees
    • naive bayes
    • vocabulary tree [Patrick did already]
      *_GSOC_
  • Vadim
    *
• porting OpenCV algorithms from OpenMP to TBB, adding threading to some new algorithms, such as planar object trackers (Calonder, One-way).
• extending HighGUI: more advanced image visualization (pixel value, zoom, loupe, remembering window positions and sizes, “save as” …), separate control panel with buttons, opengl support.
• camera calibration GUI and more advanced calibration algorithms (with more accurate distortion parameter estimation)
• building 3D shape out of silhouettes
  *
  • Victor A project around object recognition
  • Gary: Image stitching and/or collage
    *_Release_
  • Gary talk to Steve
  • Vadim estimate timing
• Hirschmuller
  • Gary run test code
  • Put description in the docs Vadim
  • Paper for ECCV?
• ECCV
  • Normal priors closed form solution
  • Stereo
    *_SOW_
  • Sanjay — seems clear
    • All the code belongs to the corporation
    • But not a standard
    • NNU applied for scalable machine learning grant (Latent SVM will get in)
    • VP to sign
    • Host sandybridge at Willow
  • Andy — new SOW
  • GPU … can we make a demo? BM stereo. Then
  • Projector, get regular one. Gary ask Kurt about opencv team a projector.

Vadim

1. extra functions cv::getValidDisparityROI and cv::validateDisparity (together with the C counterparts, cvGetValidDisparityROI and cvValidateDisparity) have been added, by the request from Kurt. The first function returns the ROI in the computed disparity map that has been computed using only valid pixels of the rectified images. the example pictures have been sent on Saturday. The second function performs left-right consistency check and can be used to reject some incorrectly computed disparities. The algorithm is taken from the Hisrchmuller paper and it requires the core stereo algorithm to store the pixel/block matching costs Vadim
2. The original Hirschmuller algorithm (running on blocks, with 1×1 blocks being a partial case) has been implemented and tried on the “a chair at WG office” and Tsukuba stereo pair. The point clouds for the first pair have been sent. The computed disparity looks very-very close to the approximate Hirschmuller, and yet the performance is 2x lower at least. Since it’s easy to switch between the two variants, by setting StereoSGBM::fullDP to true/false, both of the variants are available, but it looks like there is no reason not to prefer the original algorithm, at least on our typical data Vadim
3. Gaussian mixture-based background/foreground algorithm in cvaux, based on the paper by KaewTraKulPong P and Bowden R, “An Improved Adaptive Background Mixture Model for Real-time Tracking with Shadow Detection”, has been rewritten:
   - C++ interface has been provided in addition to the C interface
   - the optional learningRate parameter has been added that controls the model update speed. When it is 0, the model is not updated at all; when it is negative, default update speed is used, otherwise the specified value is used.
   - performance has been improved a lot.
   Vadim
4. The status of TBB integration: OpenCV build system has been extended to detect TBB on various platforms, but that appears to be non-trivial. TBB source packages do not include an installation script or pkg-config script. But we found that Ubuntu 10.4 ships with pre-packaged TBB # 2 with the pkg-config script. These packages installed on Ubuntu 9.10 and work flawlessly. Ubuntu 9.10 comes with TBB # 1 that has some bugs critical for us. So we added TBB detection script to OpenCV that uses environment variables on Windows and Mac, and pkg-config on Linux. In addition, we check that TBB version is >= # 2 Anatoly
5. Two contributed patches have been applied: >2gb memory support in highgui from our long-term user Shiqi Yu, and fullscreen highgui windows capability from Yannick Verdie.
6. Misc: Maria is back from vacation and will continue testing of the stereo algorithms, including the new Hischmuller.

Victor

• Outlet detection:
  • tested on yet another plugin pose (from aside). The detection worked fine after adjusting the scale search space and gamma parameter.
  • In future I plan to make a search over different gamma parameters and then reduce the search space for both scale and gamma over time with a kalman filter.
• A new version of planar pose estimation with known plane normal has been implemented. The new algorithm solves the two-dimensional optimization problem (finding the distance to the object plane and rotation angle inside the plane) in closed-form, minimizing the reprojection error in the object plane (not the object plane — a closed-form solution would not be applicable there).
  • The standard deviation on bag files with non-moving forearm camera facing an outlet is two times smaller than that of the current method.
• The work on outlet detection test is in progress Alexey.

Action Items

Gary

• Get GSOC application in
• Get Suat’s code to Victor
• (./) Talk to Steve about release extension
  • ‘’Stick to the March 31 deadline, call in release # # Then do # # If the incremental difference is not great, then the user adaptation problems won’t be as large either.’’
• See if we can get a textured light projector to the OpenCV team
• Wiki doc but on C++ lower case functions.

Vadim

• Send Kurt message when stereo changes are checked in
• Stereo example code to Gary. We want to run: BM, Hirschmuller and “BMH”: Block matching Hirschmuller.
  • It should report run time and design some metrics for quality.
  • Get data for Middlebury has ground truth,
  • Willow stereo can at least have number of disparities found for quality. Don’t know what else other than … use of point clouds for recognition?

Victor

• Write up outlet pose for ECCV

James

• Look over how to make numpy, PIL and cvmat all usable in the python framework.

From last time

Gary

• :( GSOC application
• Stereo textured images to Vadim
• Find out about making ground truth set. Maybe talk Morgan at Stanford to get his scanner for making ground truth stereo
• Get Suat’s code to Victor
• Write recognition framework
• (./) Get texas access for Argus
  • Scott doesn’t want to invest or maintain the old system while working on the new, so he’d rather they not be used much at Willow until the new ones are created.

Vadim

• (./) See if we can get the improved Background subtraction code into opencv
• (./) Try Hirschmuller on textured light stereo images (try pure Hirschmuller and the modified variant).
  • Return point clouds for Rudu to look at

Victor

• Look over Suat’s code

Agenda

• GSOC application
  • OpenCV GSOC 2010
• March 31st release
  • CODE FREEZE
  • C++ wiki lower case function name search bug still there
  • Bug close out
  • Test function focus
  • Documentation priorities
  • Background subtraction
    • C++ Examples
    • New stereo
    • Python
    • MLL usage
  • Samples
    • Data collection utilities
      • Use BG+segmentation
      • Boxes relative to a chessboard
• Texture recognition toolbox
• SOW work
  • GPU
  • CPU

Ongoing

• Code coverage
  • Coverage spreadsheet
• Calibration/Stereo
  • Plan of record
• Bugs tracking …

List of wanted additions

CODE FREEZE: March 9th

• (./) Get in GSOC application.
  • (./) ’’Stereo processing by semi-global matching and mutual information_Hirschmuller
• Segmentation: Might get as intern(?) —
  • Active Segmentation with Fixation. Ajay Mishra , Yiannis Aloimonos, Cheong Loong Fah, iccv 2009
    • Code (Malab and windows at least).
• Texture recognition toolbox
  • Some similarities with MOPED from CMU, but
    • not dependent on SIFT
    • cloud capble
    • accommodate Stereo up front
    • contours
    • flexible objects
  • New texture features:
    • Combine Daisy like idea, or:
      • Scale Invariant Feature Transform with Irregular Orientation Histogram Binning, Yan Cui, Nils Hasler, Thorsten Thorm?hlen, Hans-Peter Seidel
    • With patch duplets described in:Patch-Duplets for Object Recognition and Pose Estimationby ‘’Bjorn Johansson and Anders Moe’’
      • Tested in A Performance Evaluation of Local Descriptorsby ‘’Krystian Mikolajczyk and Cordelia Schmid’’
    • With binarized features as described in_Multiple target localization at over 100 fps BMVC, 2009, S. Taylor and T. Drummond.
      Minutes

• GSOC 2010
  • Victor

Assessment of OpenCV object recognition toolbox on several different problems:

a) PASCAL 2009
b) Caltech 101
c) The ETH Zurich shape database

During the first phase, sample algorithms will be created to assess OpenCV capabilities in these recognition problems. Input/output engines will be implemented if necessary, one or several recognition algorithms will be implemented and compared to the state of the art results. The convenience of the API will be also assessed. During the second phase, the feedback will be used to improve object recognition toolbox both in terms of algorithms and API. The project will show what someone can do with a recognition problem having just OpenCV, provide useful feedback for the object recognition toolbox and produce a set of recognition samples.

• Vadim
  • HighGUI
• Gary
  • Image stitching
  • Gary will finish the application on Wednesday, April 10
• Stereo
  • Boundaries for speckle detector — ‘’probably this is now fixed_and same as in Kurt’s original code
    • These were put in so that processing doesn’t exceed the image boundaries
  • Can set ROI image1 and ROI image2 to get maximal image in stereoBM states. If not set, the function will compute all pixels except
    • Vadim will change this to be the default
  • Now have sample code so that you can do Hirschmuller, BM, semi-Hirschmuller
    • This is all running on Middlebury Maria
  • Now take in account processing effects on disparity window, so there will be no border effects even without the speckle filter
• Features
  • Calandor, 1 way and Ferns. Calander dense (PCA version) is a regular vector
  • Include the descriptor with matcher class
  • IPL has to
  • How does Gary’d “patchlettes” fit in??
• Plugs and Outlets
  • Seems to be working right now on outlet wall and in simulation
  • Offset calibrates robot calibration — think this is robot arm dependent
  • ECCV
    • Middleburry
    • Data
    • Hirschmuller used xsobel
• Sparse bundle adjustment and Sparse pose adjustment — discuss getting this in post release
• Versioning
  • Sent James and Brian messages about branch and fixes
  • CC me
• Code freeze
  • C++ wiki search bug on all lower case function
  • Work down bug lists
  • Add tests for new functions
  • Documentation
    • Background subtraction
  • Data collection untilities
    • Available as a sample code
• New branches
  • cvpatented branch
  • user branch
    • subdivide these into small modules that can build separately Motion, 3D, …
    • Look at Willow’s build bot to kick back “doesn’t build email”
    • clean out of unsupported and not building packages
      • move to graveyard

Vadim

1. OpenCV can now fully supports 64-bit MacOSX. Carbon UI has been replaced with Cocoa (thanks to Andre Cohen_) and Quicktime backend for Video I/O has been replaced with QTKit (thanks to Nicholas Butko_). The code is in SVN [Vadim
2. The long-standing problem with old and new CPU support has been mostly solved (the remaining pieces of code are been adjusted now).
   • In OpenCV # 0 SSE2, SSE3 etc. support could only be enabled or disabled at compile stage, so the code was either slow (without SSE instructions) or crashed on some old hardware.
   • Now we call CPUID in the beginning and determine which instructions CPU supports. Then, the optimal code branch is selected at runtime. Thanks to this feature, the runtime dispatching, we can safely use new_Nehalem instructions_and even_AVX_and ’’do not need to build a special version_for the new CPUs.
   • The code will be committed after a bit more testing Vadim
3. 64-bit Windows support is being improved.
   • Our buildbot now includes Win64 build.
   • Visual Studio builds the code successfully, except for ffmpeg interface module (opencv-ffmpeg), because of missing ffmpeg binaries.
   • Mingw builds core libraries well (after a few fixes), but highgui currently fails to build. The problem is being investigated. Alexander
4. Stereo SGBM has been integrated into our test framework Maria
5. stereo_match.cpp, [introduced by_*Victor*_
   ], has been extended to run stereo correspondence on already rectified images, to run the modified SGBM or full Hirschmuller SGBM, in addition to block matching. The sample is in SVN Vadim

• Misc: Starting from the prev. week, Alexander Shishkov, Nizhniy Novgorod State Univ. postgraduate student has joined the team instead of Anatoly Baksheev, who will work on improving OpenCV stereo under NVidia GPU acceleration umbrella project :) Welcome, Alexander and thank you very much for your work, Anatoly!

Victor

*_Outlet detection:_

• no reports on outlet misdetections. One report on chessboard misdetection for gazebo (simulated) data, the detector was filtering out a chessboard with narrow white border. The problem was fixed by increasing the white border.
• Outlet detection code is being refactored. The first pass has been completed, the new wrapper interface is used for one way descriptor, a lot of code was pruned out.
  • The current version has indicated an instability in the geometric matching engine that is being fixed. The second pass on the interface will be done afterwards.
• A regression ros test for outlet detection has been implemented. It runs the outlet node on several bag files, stores the poses and compares them with older results (the number of poses and the contents are compared). Alexey

Action Items

Gary

• Run my patchlettes detector on the far outlet data
• Define user contrib processes
• Define how to create cvpatentend directory
• Help fill out C++ documentation

Vadim

• (./) Set the stereo defaults to return only valid disparity processing areas
• Write up a description of semi-Hirschmuller
• Help fill out C++ documentation

Maria

• Get in Middlebury data

Victor

• Set up GSOC 2010 mailing list
• (./) Send James and Brian, CC’Gary a ping on the patch version of OpenCV
• Send email about dataset labeler … Alexy check if it still work

From last time

Gary

• (./) Get GSOC application in
• (./) Get Suat’s code to Victor
• (./) Talk to Steve about release extension
  • (./) ‘’Stick to the March 31 deadline, call in release # # Then do # # If the incremental difference is not great, then the user adaptation problems won’t be as large either.’’
• :( See if we can get a textured light projector to the OpenCV team
• :( Wiki doc but on C++ lower case functions.

Vadim

• (./) Send Kurt message when stereo changes are checked in
• (./) Stereo example code to Gary. We want to run: BM, Hirschmuller and “BMH”: Block matching Hirschmuller.
  • It should report run time and design some metrics for quality.
  • Get data for Middlebury has ground truth,
  • Willow stereo can at least have number of disparities found for quality. Don’t know what else other than … use of point clouds for recognition?

Victor

• Write up outlet pose for ECCV

James

• Look over how to make numpy, PIL and cvmat all usable in the python framework.

Agenda

• -GSOC application-_’’Done__
  • OpenCV GSOC 2010
  • Fill out your project pages.
• Python
  • The Python bindings are improving rapidly, so would it be possible to include a Windows OpenCv # something Beta/Experimental release as part of the online download packages? I expect that many of us work in both Linux and Windows, but lack readily available build capability in Windows, and therefore can’t otherwise test OpenCv python programs on Windows using the improvements from SVN trunk. Furthermore, as the Python bindings had some limitations in the major # 0 release (no samples, etc), to encourage Python use I suggest a formal release of # something and its more usable Python bindings in the not too distant future.
• March 31st release
  • Patented code directory with SIFT.
  • User contrib
    • Need to define process
    • need to add directory.
  • CODE FREEZE
  • C++ wiki lower case function name search bug still there
  • Bug close out
  • Test function updates
  • Documentation progress
  • Samples
    • Data collection utilities
      • Use BG+segmentation
      • Boxes relative to a chessboard
• Texture recognition toolbox
• SOW work
  • GPU
  • CPU

Ongoing

• Code coverage
  • Coverage spreadsheet
• Calibration/Stereo
  • Plan of record
• Bugs tracking …

List of wanted additions

CODE FREEZE: March 9th

• (./) Get in GSOC application.
  • (./) ’’Stereo processing by semi-global matching and mutual information_Hirschmuller
• Segmentation: Might get as intern(?) —
  • Active Segmentation with Fixation. Ajay Mishra , Yiannis Aloimonos, Cheong Loong Fah, iccv 2009
    • Code (Malab and windows at least).
• Texture recognition toolbox
  • Some similarities with MOPED from CMU, but
    • not dependent on SIFT
    • cloud capble
    • accommodate Stereo up front
    • contours
    • flexible objects
  • New texture features:
    • Combine Daisy like idea, or:
      • Scale Invariant Feature Transform with Irregular Orientation Histogram Binning, Yan Cui, Nils Hasler, Thorsten Thorm?hlen, Hans-Peter Seidel
    • With patch duplets described in:Patch-Duplets for Object Recognition and Pose Estimationby ‘’Bjorn Johansson and Anders Moe’’
      • Tested in A Performance Evaluation of Local Descriptorsby ‘’Krystian Mikolajczyk and Cordelia Schmid’’
    • With binarized features as described in_Multiple target localization at over 100 fps BMVC, 2009, S. Taylor and T. Drummond.
      Minutes

• GSOC:
  • People have pretty much updated their individual projects. We just wait for word from Google.
• Python bindings for windows will be built and released with # # To do this with SVN, we’ll figure out a way to put in pre-built bindings.
• Release
  • Lots of tickest being fixed
  • Bugs and docs
  • # 1 will go out, when people find problems, # 1.1 …
• Code contributions
  • Doxygen (?)
    • OpenCV development mailing list someone is trying to do this
• Might have to update our test system to work easily with cpp.
  • Or use ctest or other
  • No global places to put, but it all goes to the same binary.
    • So if one test fails, the whole thing crashes
• GPU-CPU
  • Need to organize memory acces.

Vadim

Last week we started preparations to OpenCV # 1 release. Mostly we spent time cleaning bug tracker and fixing various compile problems.

1. ticket #155 closed (bug in Haartraining application, reported by R. Lienhart student Christian Ries) Maria
2. closed tickets ## 49, 66, 76, 77, 115, 140, 163, 164, 165, 170, 171, 174, 176 Vadim
3. stereo test has been fixed to reflect the latest changes in StereoBM (by default, the border is not processed). SGBM (2 variants) and BM have been tested on the standard middlebury images. As the result, we now get 68, 69 and 77th places, respectively. Too low for the paper yet Maria
4. quality of SGBM has been slightly improved by taking into account pixel brightness component in addition to X-Sobel. The images have been sent in a separate mail. Vadim
5. OpenCV now builds successfully on Win64 (except for the optional opencv-ffmpeg component).
   • Looks like OpenCV # 1 will be the first OpenCV version that is tested and pre-packaged for 64-bit Windows [Alexander]
6. Samples for fern-based object detector and Calonder descriptor added to samples/c Alexey

Plans for the next week: close more tickets, fix and extend documentation, test installation and source packages packages on various platforms.

Victor

• Outlet detection:
  • Implemented a geometric model that combines points, edges and regions matching. The motivation for this model was to lower the requirements for (or, ultimately, get rid of) keypoints classifier. We already had bits and pieces of this model and the job was to put them all together.
    • A hypothesis on affine transformation is used to project template keypoints, edges and regions to a test image. Edges are matched using simple chamfer matching distance. The template image is segmented into several regions with low variation of intensity, and several points are sampled randomly within the largest region.
    • These points are projected into the test image and standard deviation of intensities in these points is compared with the corresponding standard deviation in the template image.
    • First experiments show that the model is capable of finding outlets in challenging datasets where the current method (based on one way descriptor) had problems (for instance, 2010-02-02-14-35-01.bag).
    • The experiments were done without any keypoint classifier, the outlet is found using geometry only. Because the hypotheses are produced by geometric hashing algorithm, the processing time is 20ms per frame (15 for keypoint detection, 5 for matching) which is a huge difference with the current method that does search over scale and needs 1s per frame for robust detection. More experiments are needed to analyze the method robustness on different datasets.
  • The one way descriptor method has been improved by implementing a search over several values of gamma parameter. This makes the algorithm work robustly on datasets with different cameras, camera settings and lighting conditions.
  • The outlet detection code has been refactored, lots of unused code has been removed. Another pass of refactoring is needed for outlet_model.cpp — implementation of hole extraction.
• Plug detection:
  • another problem with checkerboard reported by Patrick has been fixed. Features detected on the border of the checkerboard caused filtering of the detection. I have implemented a workaround, but in the longer term the low-res checkerboard detection has to be rewritten with a better geometric model (such as geometric hashing).
• Generic interface for descriptors has been prototyped. Outlet detection is already using the new wrapper interface for one way descriptor. Implementation of a wrapper for SURF is on the way.

• GPU stereo BM implementation studied in detail and several initial optimization attempts were done. Most of them are parameters tuning for a specific card (GT220). Also we experimented with the registers count used by nvcc compiler. The tuned version is about 25% faster on the largest resolution (from 1.91sec to 1.38sec on # 7Mp stereo pair, see attached image).
  • Current streaming multiprocessors occupancy is 1.0, global memory reading is coalesced (as in the initial implementation). Profiler shows # 19 GB/s total memory throughput, which is several times lower than theoretical bandwidth limit (10.7 GB/s). Currently we believe that the limiting factor is global memory latency (images fetching, SSD and disparity arrays accessing). Now we need to redesign the algorithm to gain further improvement.
  • OpenCV stereo tests adopted for GPU. Now we can measure accuracy of stereo correspondence for data with given ground truth.
  • NVidia NEXUS installed, studied and tested on several sample projects and GPU stereo BM implementation. But this tool is mostly debug-oriented, so it is not important for us at the current stage of the project.
    

Action Items

Gary

• Define code contrib. process
  • Pros and cons of user documentation

Vadim

• Define code test system for user contrib
• Define “other” directories for patented and contrib code

Victor

James

From last time

Gary

• Run my patchlettes detector on the far outlet data
• (./) Update my GSOC project page.
• :( Define user contrib processes
• (./) Get in GSOC application
• Define how to create cvpatentend directory
• Help fill out C++ documentation
• Define user contrib. process

Vadim

• (./) Set the stereo defaults to return only valid disparity processing areas
• (./) Write up a description of semi-Hirschmuller
• Help fill out C++ documentation
• (./) Update GSOC project page

Maria

• (./) Get in Middlebury data

Victor

• (./) Set up GSOC 2010 mailing list
• (./) Send James and Brian, CC’Gary a ping on the patch version of OpenCV
• Send email about dataset labeler … Alexy check if it still work
• (./) Update GSOC project page

Agenda

• Large format stereo
• Release update
  • User contribution
    • New directory
  • CV Patented/cv_nonfree
    • Directory that user can download if wanted. Specifically: perhaps just binaries that can be used to test against SIFT?
  • Test system
  • Code coverage
    • Coverage spreadsheet
  • Calibration/Stereo
    • Plan of record
  • Bugs tracking …
• GSOC intern updates — go through the list

Next

• Release: March 31st
• Segmentation: Might get as intern(?) —
  • Active Segmentation with Fixation. Ajay Mishra , Yiannis Aloimonos, Cheong Loong Fah, iccv 2009
    • Code (Malab and windows at least).
• Texture recognition toolbox
  • Some similarities with MOPED from CMU, but
    • not dependent on SIFT
    • cloud capble
    • accommodate Stereo up front
    • contours
    • flexible objects
  • New texture features:
    • Combine Daisy like idea, or:
      • Scale Invariant Feature Transform with Irregular Orientation Histogram Binning, Yan Cui, Nils Hasler, Thorsten Thorm?hlen, Hans-Peter Seidel
    • With patch duplets described in:Patch-Duplets for Object Recognition and Pose Estimationby ‘’Bjorn Johansson and Anders Moe’’
      • Tested in A Performance Evaluation of Local Descriptorsby ‘’Krystian Mikolajczyk and Cordelia Schmid’’
    • With binarized features as described in_Multiple target localization at over 100 fps BMVC, 2009, S. Taylor and T. Drummond.* Stereo
  • A good overall reference to stereo algorithms is:
    • http://www.vision.deis.unibo.it/smatt/seminar_stereo_vision.html
  • Bilateral filter followed by a local smoothing method:
    • http://www.vision.deis.unibo.it/smatt/lc_stereo.htm

Minutes

• Large format stereo
  • Large format stereo has problems scaling (local minimum in matches)
  • NASA and U. of Delaware have experience, should talk to them
  • Calibration can be a problem, window size has to flex
  • Higher resolution also tends to mean smaller pixels and at that lower scale, you can get less contrast per pixel
  • Stereo doesn’t tend to scale.
    • What works at 320×240 can break down at 640×480 and up. Probably because:
      • Calibration can break down
      • Contrast between pixels get less
  • NASA
    • Really nice cameras
    • They have texture (planet surfaces)
    • Their shots are pretty much frontal-parallel
  • It might be interesting to look at other algorithms that are more local. A good overall reference to stereo algorithms is:
    • http://www.vision.deis.unibo.it/smatt/seminar_stereo_vision.html
  • I’ve been impressed by the bilateral filter followed by a local smoothing method:
    • http://www.vision.deis.unibo.it/smatt/lc_stereo.htm
  • Plan
    • Finish up plans with Hirschmuller
    • Careful calibration tests.
      • Check if calibration issue — Just take center 640×480. Get the sub-window well calibrated. Will tell us if the calibration is the issue.
        • Closed aperture vs open
      • Use chessboards with more corners Vadim
      • Will have to change distortion model Kurt
        • Local variation in the lenses might have to be accounted for
        • This would involve local look-up functions
    • Get texture into the scene
    • ~HD resolution is higher priority Gary
    • _Gather data:
      • Calibration pattern, laser line scan, actual object with and without textured light (can release this data)_Gary
    • Color can help alignment, can we test on color Vadim
  • Kurt stereo comments from email

Here my assessment of some problems
and the way we should go:

- I don’t expect to be able to extract stereo from scenes to the same
degree that we can with human eyes, without adding texture. Partly
this is hardware (sensitivity, S/N), and partly software (not using
all available cues such as edges, shadows, highlights, global
constraints). The recent tests with the 35mm cameras seem to confirm
this. Using the SI cameras won’t change this. We’ll still need to
project texture.

- We don’t need a lot more pixels on target. HD size (1920×1080) with
a wide-angle lens seems like it would be fine for getting good dense
depth data.

- Global shutter would be much better than rolling shutter, especially
if we’re projecting texture. That’s why I’m not happy with the SI
(Altasens) or Sony sensors.

So how should we proceed? Here are my suggestions.

1. Get a good megapixel stereo setup with texture projection running.
   I have such a setup in my office, global shutter with 1.3MP
   high-sensitivity sensors, ethernet interface. I’ve asked Blaise to
   hook up the drivers in Linux and a projector board to trigger them,
   and he’s going to work on it this week. I already got some images to
   Vadim and he processed them, and they look pretty good. This
   prototype will help move us forward on algorithms and evaluation of
   whether we can achieve the 3D results we need.
2. Move to IR projection and a 3-head design. Two imagers just look
   for texture in the IR band with filters, one is color for registered
   visual texture. The current imager I’m targeting is the CMOSIS
   CM2000, which has pretty good S/N in HD resolution, and is global
   shutter CMOS.
   I have two groups looking at making this device, although it’s not
   certain either will move forward. Need to press on this.
3. Algorithms. If we’re using texture projection, SGBM or something
   similar but simpler may work very well, although we still need to work
   on getting edges sharper, which is the primary limitation of
   texture-projected stereo.

[older email]

• If we could coordinate on this project so we agree on what we’re
  trying to produce, that would be great.
• Large-scale stereo is very difficult, requiring
  advances in a number of fronts, including calibration. It’s not just
  a question of computational power.
  • We should talk with the NASA/Ames
    folks, who have some experience with this sort of thing in doing their
    DEM work. I did briefly when I was there; unfortunately their
    situation is much better than ours, in terms of having great texture,
    and being almost front-planar and within a restricted disparity range.
• There are other groups doing large-format stereo, I think one at the
  U of Delaware.

Release:

• Tickets
  • All of the ’’patch_tickets have been closed along with the out of date tickets
  • OpenCV forum, there are bug reports.
• Coverage — no new tests
• C++ docs
  • Found in PDF, so problem is not intrinsic
• User contributed code
  • Doxygen docs
    • Another contributor is working on doxygen comments for OpenCV in a separate branch
  • Each contribution
    • Top level directory called cv_contrib
    • Subdirectory for each contributin
    • include and src
    • CMakeList.txt
    • include code will have root CMakeList.txt
    • black list
    • separate svn for cv_contrib
• Carbon bindings for HighGui in Mac need to be put back (build problems)
• Other than that, the release is in pretty good shape.

Descriptors:

• Patrick’s post, works for descriptors that are vectors, but not to multi-vector
  • Also non-standard metric matchers (intersection)
• Victor’s approach does not give an explicit representation of a descriptor
  • More generic, just
    • Compute descriptors
    • Classifify points
    • Has subset of vector descriptors
• Add SIFT as User contrib Alex Boverin
• New proposed interface is: http://www.ros.org/doc/api/outlet_detection/html/classcv_1_1DescriptorMatchGeneric.html

Vadim

The work on the release continues.

1. Several tickets have been closed (## 92, 121, 130, 131, 136, 147, 160, 162, 167, 190, 196). Work on a few more tickets is in progress [Vadim, Maria]
2. Draft version of installation package for Windows has been built and sent to James. Besides a few minor glitches it seems to install to work fine Vadim
3. Calonder descriptor code has been cleaned up, the sample find_obj_calonder.cpp has been added Alexey
4. Because of quite a few problems with building and using OpenCV relate to OpenMP (in particular, at least open 4 trac tickets relate to OpenMP), it’s been decided to drop OpenMP and replace with TBB (StereoBM has been already ported to TBB sometimes ago by Anatoly).
   • Last week Haar/LBP object detector, HOG detector and Distance transform have been converted to TBB.
   • In CMake scripts TBB detection part has been improved as well.
   • It’s planned to convert the remaining OpenMP code in vision algorithms (SURF, sparse optical flow) to TBB before the release Vadim

Plans for the this week:

• close some more tickets,
• fix and extend documentation,
• test installation and source packages packages on various platforms,
• finish preparations to the release.

Victor

• Descriptor interface:
  • Implemented support for SURF in the new descriptor interface.
    • Now switching from one way to SURF for outlet detection is a matter of changing 3 lines of code.
  • The interface has been restructured and documented, doxygen-generated doc is here
    • http://www.ros.org/doc/api/outlet_detection/html/classcv_1_1DescriptorMatchGeneric.html
• One way:
  • the ROS version has been synchronized with opencv version (latter was outdated, former contains several bug fixes as well as support for kd-tree search).
• Other:
  • NNSU university has started working on the implementation of Gradient Boosting Trees (GBT) and Felzenschwalb’s latent svm.
  • They will be designing both within opencv framework and the intention is to submit the code to opencv once it is in good shape (at least the single-threaded reference implementation).
  • They already have a stripped implementation of GBT that now uses opencv decision trees (while making the move to opencv data structures they found a bug fixed by Maria).

GPU

• Several attempts to optimize Joe’s stereo implementation with CUDA were made. We measured that texture fetching (left and right image) is the most time consuming part of the algorithm (about 50% drop in running time if we comment fetching out) and reading min SSD from global memory is the second part (30% drop in running time after commenting out). Fetching uses texture cache, so we have no success with optimization here. We tried to use shared memory for caching min SSD, but performance also degraded. The reason was that each block was consuming much more shared memory and as a result occupancy decreased.
• We have found source codes of two other stereo GPU implementations. One of them is quite interesting (http://www.cs.unc.edu/~gallup/stereo-demo) and we studied it in details (and adopted for work with larger resolutions). Implementation minimizes global memory accesses by using shared memory for storing image’s tiles. But 16kb of shared memory limiting parallel execution of blocks and this implementation is useful only for medium resolution images (up to 900×750).
• We developed GPU implementation based on calculation of integral sums (squared differences actually). Although this implementation will consume a larger amount of memory (about 1GB), it is supposed to load the memory bus more effectively (and as a result scale better on Quadro cards that have 10x memory throughput compared to the cards we use now). We have the first version working and are optimizing it now.

Action Items

Gary

• Define code contrib. process
• Post to user group about where to post bugs
• Send Vadim example user contrib

Vadim

• Create cv_contrib with example make files etc

Victor

• See about encouraging a user contrib of sift

James

• C++ functions in wiki doc

From last time

Gary

• Define code contrib. process
  • Pros and cons of user documentation
• Ask James about converter

Vadim

• Define code test system for user contrib
• Define “other” directories for patented and contrib code

Victor

• Ask James about status for release

James

Agenda

Release

• User contrib oi vey
• Can release and patch
• Can go to google news and set a filter for ‘’OpenCV bug’’

GSOC

• go through list… make list, rank list

Going forward

• Big effort on texture based recognition.

Next

• Release: March 31st
• Segmentation: Might get as intern(?) —
  • Active Segmentation with Fixation. Ajay Mishra , Yiannis Aloimonos, Cheong Loong Fah, iccv 2009
    • Code (Malab and windows at least).
• Texture recognition toolbox
  • Some similarities with MOPED from CMU, but
    • not dependent on SIFT
    • cloud capble
    • accommodate Stereo up front
    • contours
    • flexible objects
  • New texture features:
    • Combine Daisy like idea, or:
      • Scale Invariant Feature Transform with Irregular Orientation Histogram Binning, Yan Cui, Nils Hasler, Thorsten Thorm?hlen, Hans-Peter Seidel
    • With patch duplets described in:Patch-Duplets for Object Recognition and Pose Estimationby ‘’Bjorn Johansson and Anders Moe’’
      • Tested in A Performance Evaluation of Local Descriptorsby ‘’Krystian Mikolajczyk and Cordelia Schmid’’
    • With binarized features as described in_Multiple target localization at over 100 fps BMVC, 2009, S. Taylor and T. Drummond.* Stereo
  • A good overall reference to stereo algorithms is:
    • http://www.vision.deis.unibo.it/smatt/seminar_stereo_vision.html
  • Bilateral filter followed by a local smoothing method:
    • http://www.vision.deis.unibo.it/smatt/lc_stereo.htm

Minutes

• Bad problem with C++ documentation. It is a real functionality barrier not to be able to find all functions
  • Names have changed cvSmooth gaussianblur etc
• twhurl, splitweet
• Probably should update ros cv_bridge to use cv::Mat instead
  • http://www.ros.org/wiki/cv_bridge/Tutorials/UsingCvBridgeToConvertBetweenROSImagesAndOpenCVImages
  • IplImage* imgMsgToCv(sensor_msgs::Image::ConstPtr image_message, string cv_encoding=“passthrough”)
    **
• Test system
  • We have our own
  • Switch to using Google Test? http://code.google.com/p/googletest/
• Future
  • Texture based feature
  • Work on dataset so we can create the “solved problems in vision” challenge.
  • Anybody

Victor

• Outlet detection:
  • tested the new geometrical model on all available datasets with 2×1 white outlets. The results show that we can detect outlets without one way descriptor, by matching edges (chamfer distance) and regions together with keypoints.
  • Average detection time is 30ms.
  • The new method is better than one way on low quality images (low contrast, low resolution), and worse on high quality images.
  • Both methods have very low false alarm rate (less than 1%).
  • The new method can be generalized for other textureless planar objects such as european outlets.
• Descriptor interface:
  • Implemented support for kd-tree nearest neighbor search for all descriptors that can be represented by a vector.
  • Interface has been improved.
  • A sample showing visual odometry task (matching between two sets of keypoints) has been implemented.
• Other:
  • planning for GSoC project.

CUDA

• Accomplishments:
• Integral Images.
  We have implemented several versions of block-matching based on integral images (see draft foils). Performance of the current version is 1.2sec per # 7MP frame (1800×1500) on NVidia GT220 (versus 1.38sec by our tuned implementation from OpenVIDIA). We have more ideas on how to optimize this implementation — have started to work on them.
• Other activities:
  • GPU memory access experiments (texture fetching, global memory access for different data types).
  • Reading Hirschmueller GPU paper (Ines Ernst and Heiko Hirschmuller (2008), Mutual Information based Semi-Global Stereo Matching on the GPU, in Proceedings of the International Symposium on Visual Computing (ISVC08), 1-3 December 2008, Las Vegas, Nevada, USA).
  • Joe’s suggestion for parallel execution with several disparities discussed. Some preparations for implementation were made.

• Plans:

1. Optimize Integral Images implementation.
2. Start to implement Joe’s suggestion to process several disparities simultaneously (for better texture cache utilization).

Vadim

The work on the release almost finished (although we would not mind spending some more time on documentation and bug reports).

1. Several tickets have been closed (## 86, 93, 122, 123, 128, 182, 194, 201, 211, 214, 218, 219, 221, 222, 226). Work on a few more tickets is in progress [Vadim, Maria, Alexey]
2. All of the OpenCV threaded code, except for the old-style haartraining application and part of the new traincascade application, have been converted from OpenMP to TBB, in particular, sparse optical flow, SURF Vadim and decision tree engine in MLL Maria. The new TBB code demonstrates pretty good threading efficiency.
3. Written documentation on several cxcore classes Vadim, on the cascade detector Maria and on the Calonder descriptor Alexey
4. Scanned the mailing list at Yahoogroups for the bug reports since 2010 January, one severe bug in FindHomography has been found and fixed [Alexey, Vadim]
5. The OpenCV build and test framework (buildbot-based) has been improved (added some hooks for Python tests on Windows) and optimized to spend 1 hour less on the complete build & test procedure. Several new test failures have been reproduced and put to the tracker (tickets #29 and 113) [Alexander]

Plans for the nearest 2 days: close some more tickets, finish most important parts of the documentation, test the binary and source packages on various platforms, finally, do the release.

Action Items

Gary

• C++ documentation
• Get Vadim a contrib function asap with mocked out test code
• Send Victor email to see him in beginning of May in US
• Send Vadim email about CVPR

Vadim

• Set up user contrib
• Renew visa to cover CVPR

Victor

• Renew visa to cover CVPR

James

• C++ documentation

From last time

Gary

• Define code contrib. process
• (./) Post to user group about where to post bugs
• Send Vadim example user contrib

Vadim

• Create cv_contrib with example make files etc

Victor

• See about encouraging a user contrib of sift

James

• (./) C++ functions in wiki doc

Agenda

Release

• Release report
• User contrib oi vey oi vey oi vey

cv_patented

• SURF
  • http://www.faqs.org/patents/app/20090238460

TBB

• GPL with runtime exception

GSOC

• OpenCV on Android/meeting with streetview team
• Refine projects
• Go through lists

Going forward

• Big effort on texture and feature based recognition.
  • We need to unify the interface between what’s used in ROS and in OpenCV
• Other:
  • Face detection and face pose for Texai
  • HD stereo.
  • Image stabilization
  • What needs doing in VO

Next

• Release: March 31st
• Segmentation: Might get as intern(?) —
  • Active Segmentation with Fixation. Ajay Mishra , Yiannis Aloimonos, Cheong Loong Fah, iccv 2009
    • Code (Malab and windows at least).
• Texture recognition toolbox
  • Some similarities with MOPED from CMU, but
    • not dependent on SIFT
    • cloud capble
    • accommodate Stereo up front
    • contours
    • flexible objects
  • New texture features:
    • Combine Daisy like idea, or:
      • Scale Invariant Feature Transform with Irregular Orientation Histogram Binning, Yan Cui, Nils Hasler, Thorsten Thorm?hlen, Hans-Peter Seidel
    • With patch duplets described in:Patch-Duplets for Object Recognition and Pose Estimationby ‘’Bjorn Johansson and Anders Moe’’
      • Tested in A Performance Evaluation of Local Descriptorsby ‘’Krystian Mikolajczyk and Cordelia Schmid’’
    • With binarized features as described in_Multiple target localization at over 100 fps BMVC, 2009, S. Taylor and T. Drummond.* Stereo
  • A good overall reference to stereo algorithms is:
    • http://www.vision.deis.unibo.it/smatt/seminar_stereo_vision.html
  • Bilateral filter followed by a local smoothing method:
    • http://www.vision.deis.unibo.it/smatt/lc_stereo.htm

Minutes

• Release # 1 is out
  • Need to update http://opencv.willowgarage.com/wiki/OpenCV%20Monthly
  • Not fully announced yet
    • http://opencv.willowgarage.com/wiki/Welcome/Introduction
    • http://opencv.willowgarage.com/wiki/OpenCV%20Change%20Logs
• Create contributed directory

Detector Descriptor Interface

• [Vic] Likes the approach of explicit representation of data
  • See doxygen here http://www.ros.org/doc/api/outlet_detection/html/classcv_1_1DescriptorMatchVector.html
• How would Nistor trees fit into this?
  • At each level of the tree, you need to compute distances
  • Is there a notion of distance or score in Victor’s scheme
    • Thinking of confidence function, ratio of first to second as in Lowe’s use
    • Wasn’t thinking of distances since don’t appl
    • Can introduce notion of distance to any function — a function that returns a vector of distances, not just class decisions (existing integer vector implementation in vic’s code)
    • Separate object for each node would make Nistor tree cumbersome
• Vocab tree by Patrick is templated
  • Nistor stay with explicit representation
• Changing matchers
  • Classify vectors function can be used to choose brute force or index approach
  • Function to retrieve a matrix of descriptors.
• What about query descriptors? Trained descriptors are stored, how to get run time match descriptors?
• 2 interfaces? features_2d
  • Single object that computes features with explicit representation
• If a feature maps into a fixed dimensional space
  • Use normal metric
    • Then features_2d
  • But if it’s something like array of decision trees
    • Then go with generic interface
• 2 interfaces descriptor computation + matching glued together
  • vs. Descriptor computation and generic matcher separate
  • To compare everything, use wrapper for feature_2d to combine desc. compute and match so you can use general one for both
  • In descriptor

Vadim

1. OpenCV # 1 has finally been released!
   Here is the announcement in the development mailing list: http://bit.ly/afgckK
   I decided not to rush with a broad announcement, to let the most active people test the package first. It now seems to was a good decision to postpone the release by a few days, as several bugs have been fixed and the bug tracker has been cleaned a bit more.
2. We closed 24 trac tickets (## 32, 63, 82, 107, 117, 127, 138, 153, 161, 168, 195, 198, 207, 208, 231, 233, 236, 237, 240, 241, 244, 245, 246, 252) and more than 12 tickets at SF bug tracker [Vadim, Alexander, Maria].

Victor

Object recognition:

• refactoring of geometric matching code.
  • Generation of hypotheses has been separated from matching so that different transformation classes (affine, perspective, etc) can be used together with different hypotheses generation algorithms (exhaustive search, hashing, RANSAC etc) for matching different features (points, edges, regions).
  • Outlet detection is using the new interface, object detection sample is on the way.
• Implemented support of the new keypoint detection interface (from features_2d) for MSER and Harris corner detector. The code will be committed to OpenCV after the release Maria

GPU

Accomplishments:

• NVidia GPU stereo implementation has been optimized to 0.79sec/# 7Mp (original runtime was 1.9sec). Disparity loop unrolled (for better texture cache utilization) and type conversion eliminated. Alternative Integral Images GPU stereo implementation has been optimized to 1.08sec/# 7Mp by loops unrolling and avoiding bank conflicts. Both implementations uses plain block-matching, so later we may incorporate pyramids and some runtime-saving tricks (ie disparity step = 2).
• We started new stereo implementation based on Joe’s suggestion to keep image tiles in shared memory (SMEM). But we keep SSD columns in SMEM for different disparities (it takes SSDWindowSize * MaxDisparity * sizeof(int) of SMEM). The initial version is quite memory and time consuming (about 6sec/# 7Mp), and we need to work on it. Probably it is better to avoid column SSD computation and rather use SMEM for image tiles storing.
• Remap and undistort for a stereo pair have been implemented. Results: 60fps/2Mp (1920×1080) for both left and right images on GT220.

All the experiments performed on NVidia GT220 with MaxDisparity = 224 and SSDWindowSize = 19 on 1800×1500 (# 7Mp) image.

Plans:

• Experiment with SMEM approach.
• Probably start to experiment with a pyramidal approach and some other tricks.
• Create dataset with actual camera resolution.
• Start to work on Hirschmueller SGM algorithm.

Action Items

Gary

• Ask Kurt what the needs are in VO.

Vadim

Victor

• Iterate on interface with Patrick

Patrick

• See Victor above

James

From last time

Gary

• :( C++ documentation
• X-( Get Vadim a contrib function asap with mocked out test code
• (./) Send Victor email to see him in beginning of May in US
• (./) Send Vadim email about CVPR

Vadim

• :( Set up user contrib
• /!\ Renew visa to cover CVPR

Victor

• /!\ Renew visa to cover CVPR

James

• :D C++ documentation

Agenda

• GSOC
  • Today: slot allocation
  • April 18st is next deadline — students selected
  • See to do list
  • See the timeline
• CVPR
• Solved problems in vision
• Grouply site offer

Minutes

• Transalate bgfg_segm.cpp into C++
• Create cpp directory inside of samples
• Vadim created connected_componenets.cpp and morphology# cpp

Next

• Python release
• Code contribution process into its own directory … high priority
• CVPR
• VO (Victor)
• Detector descriptor pipeline
• Improve camera calibration … yet more for more stable distortion parameters
  • Use edges and other geometric figures
  • It already allows for arbitrary 2D and 3D patterns, needs tools for finding different patterns
  • Allow “infinite” grid (chessboard that goes off camera)
  • Improving distortion
    • Show “infinite” board
    • Detect all fiducial points/corners (could fit curve) Try to optimize all the points
      • Most information is at the edge
      • Most of the improvement will be in finding better, subpixel edges
      • Can use results from LePetit — very fast PnP solves ¹
• Barcodes
  • Product bar code reading
  • 2D bar codes (call existing library)
    • Mail: Feb 25 libdmtx — put this into cv_contrib
  • An AR tookkit, similar to libdmtx
• Approximate NN
  • k-means in FLANN, is opencv’s k-means faster — question for Marius
  • Same kd tree

¹ F.Moreno-Noguer, V.Lepetit and P.Fua , Accurate Non-Iterative O(n) Solution to the PnP Problem, IEEE International Conference on Computer Vision, Rio de Janeiro, Brazil, October 200# (with code in Matab and C++)

Grouply

• Problems:
  • They decide to start charging us later
  • They go out of business

Solved Problems in Vision

• Will launch (probably) at ECCV workshop

Vadim

1. Some minor maintainance work after # 1.0 release has been done. 19 tickets have been closed (## 52, 108, 135, 189, 209, 224, 239, 251, 258, 261, 269, 270, 272, 273, 274, 276, 277, 278, 280).
   • While the number may seem huge, given that # 1 has been just released, in fact most of the bugs have been either already fixed in # 1.0, or are duplicate bug reports, or are quite minor problems. The only two important bugs:
2. cvHaarDetectObjects crashed when no face grouping is made (parameter minNeighbors=0) and # OpenCV build failed on MacOSX 10.6 when using command line makefiles (but Xcode built it fine). Vadim
3. Created opencv/samples/cpp directory where we put samples using the new C++ interface. There are 2 samples for now: morphology# cpp (rewritten morphplogy.c sample) and connected_components.cpp. Vadim

Victor

• Object recognition:
  • Experiments with descriptor interface are in progress. The two interfaces (explicit representation of descriptors and a more generic API) can be merged together.
  • The main issue now is inconvenience of using one way descriptor for VO — it lacks flexibility in adding/removing descriptors.
  • A bug in one way descriptor has been corrected. [Victor]
• Keypoints:
  • Keypoint detection has been ported from features_2d to OpenCV and enhanced with MSER and Harris corner detector.
  • A sample of using keypoint detection and descriptor matching has been implemented.
  • A testbench for comparing the performance of keypoint detectors and descriptors is in progress. Maria
• Image stabilization
  • Several methods for image stabilization have been investigated. The paper ¹ is the candidate for implementation. Alexey

¹ Ken-Yi Lee et al, Video Stabilization using Robust Feature Trajectories, ICCV’09.

GPU

1. Both versions of GPU stereo implementations (NVidia and Integral Images) have been tested on Quadro FX 5800. NVidia’s version demonstrates # 0fps, Integral Images version – # 7fps. This numbers are for the highest resolution of Silicon Imaging camera (2048×1152), MaxDisparity=256 and SSDWindowSize=19. Speedup ratio on Quadro is about 5x, proportional to the increase of the number of multiprocessors (30 on Quadro vs 6 on GT220). We have made some new minor performance optimizations, but they gave us only 5% of improvement in runtime.
2. Pyramidal version of NVidia GPU stereo block matching have been implemented. It uses only one additional pyramid level (1/2 in width and height) because we try to avoid errors on coarse levels. First version demonstrates 8fps (on Quadro) without any quality degradation. If we use smaller SSD window (11 instead of 19) during second pass (disparity refinement) we can achieve 20fps without visible disparity map degradation. But we plan to find ways to avoid errors caused by the pyramidal approach.
3. Also we tried to implement Joe’s suggestion to keep image tiles in SMEM. But this approach for our MaxDisparity and SSDWindowSize values requres too much SMEM and unsuccessful therefore. MaxDisparity x SSDWindowSize = 256 × 19 = 4864 bytes per block only for the left image tile and single pixel per block. Also procedure for choosing the minimum disparity between different threads takes to much time (about 0.9sec on GT220 even after avoiding bank conflicts). So, this approach looks unperspective for us.

Plans:

1. Experiments with pyramidal approach.
2. Experiments on Fermi card (if we get it this or next week).

Action Items

Gary

• Write Vincent about epnp license
• Write sample cv_contrib

Vadim

• Set up cv_contrib

Victor

Patrick

James

From last time

Gary

• (./) Ask Kurt what the needs are in VO.

Vadim

Victor

• Iterate on interface with Patrick

Patrick

• See Victor above

James

Agenda

• GSOC — 8 slots
• Texture rec toolbox
• VO
• Reorg of directory structure
  • Adding user contrib there

Going forward

• CVPR Demo
• Solved problems in vision
• Improved camera calibration
  • Better descriptors
  • Implement ¹
  • 2D bar codes
  • 1D bar codes
  • Infinite grid

¹ F.Moreno-Noguer, V.Lepetit and P.Fua , Accurate Non-Iterative O(n) Solution to the PnP Problem, IEEE International Conference on Computer Vision, Rio de Janeiro, Brazil, October 200# (with code in Matab and C++)

Minutes

• GSOC
  • Looks good for 8 slot, but final decisions on April 26th
  • Had 50 applications, figure out how to nicely tell the others they weren’t selected. Many are very good, maybe as interns next year etc.
  • Pays to look over the 50 slots
• Reorg of OpenCV directory structure
  • Can be done in 2 weeks
  • We’re waiting for this to move descriptors in ROS to OpenCV
    • Other problems might happen due to the change
    • Maria working with descriptors inside ROS
  • Build bot
  • Can we go in steps?
    • Move several folders to different locations
    • Vadim prefers to do it all in one step to get user contrib started
    • Step 1: Enable the module structure (fix folder structure)
    • Step 2: Divide up cvaux while Maria works on moving over the toolbox
  • Vadim wants to do in branch first, then in trunk
  • Discuss this offline, since we want to finish descriptors are high priority but don’t want to create havok with change.
• VO
  • 2 images and point cloud from 2 frames
  • Matches 2 frames using STAR and Calandor descriptor (ferns) using X,Y,Z
  • Then does bundle adjustment over all the frames
  • Output is reconstruction of camera trajectory and location of 3D points
    • Walls found well
    • Victor is trying to enable the same thing for monocular cameras
      • Estimate translation of camera for point to point (SFM)
      • Assume some points lie in plane.
        • Easier Faugera paper, 4 pt method estimates Essential matrix and R,T.
        • Closed form to 1 of 8 possible configurations, use visibility constraints to eliminate back of camera solutions
        • Then use epipolar projection error on other points to resolve final view
      • Ransac
        • 4 random points
        • Compute R,T with 4pt algorithm
        • Compute epipolar reprojection error on all points
        • Chose best
      • This initializes monocular correspondence between 2 frames
        • Then use sparse bundle adjustment to improve 3D point location and R,T
        • Compute 3D point cloud error using extrinsics and intrinsics
        • cvTriangulatePoints — This doesn’t show up in the docs
        • Created ticket for this #302 Gary
  • Somewhat related to Gary’s stitching project in GSOC
  • Use all this to initialize a “STF pipeline” or “STF Stack” in OpenCV
• Texured object toolbox
  • [Alexander C.] has a dataset from Romain T. at Willow with different objects on a turntable
  • Use this to do a demo at ICRA and/or at CVPR.
    • Cropped images from the bag files
    • Match test image to one of the objects using bag of features
    • Then move to object pose
    • Use stereo data for 3D using SolvePnP()
    • We’ve suffered from too little texture here [Anatoly/Suat]
  • 3D capture is different, here we’re going for pose
    • Recognize proposed object list
    • Given the point cloud in a reference image
    • Reproject points to get confirm or reject
  • Test bench for detectors is already in place
    • Tests stability of the detectors
      • test set with known homographies
      • see how well the detectors match between planes
      • distort images and see how well the points stay in place
      • all opencv detectors work with this test bench
• CVPR Demo
  • Visa problems for CVPR? Vadim hasn’t applied yet … time is getting short.
  • Do you have to go to US Embassy or not — Vadim not expired, so maybe OK
  • Run ICRA demo
  • Stephan doesn’t want to show his work until ECCV
  • Try to show our version of “MOPED” there to get recognition and pose in clutter.
• Solved problems in vision challenge
  • Might use our version of MOPED
• Improved Camera Calibration
  • Markers in OpenCV that are very robust to detect
  • Vijay was using the chessboard in the robot arm. Gets failures
    • We’re collecting the failures which tend to occur in
      • low lighting
      • bright backgrounds
    • We need to correct this but:
  • We want a blob based marker that can be detected robustly anywhere
  • Quality of sub-pixel refinement … chessboards aren’t great for this … up to 0.5 pixel error
  • Better sub-pixel makes a big difference
  • Design for few false alarms and then for grid. Geometric matching then is robust.
• Barcodes
  • 2D and 1D barcode reading …
  • Post CVPR
• Problem with eigen

Vector4d a;
Vector3d b = a.start(3);
Mat c;
eigen2cv(b, c);

you will get garbage in c. Vadim has reproduced this and is looking into it now.

• Fixed now:
  • these are updated convertors (also put to SVN). It now should work fine. The problem was that by default Eigen uses column-major data layout, while I expected it to be row-major.

New proposed directory structure

opencv/

. 3rdparty/      # various 3rd-party libs, like libjpeg, clapack etc

. data/            # some pre-trained classifiers

doc/             # tex-based documentation for the key modules; will be retained

include/opencv       # header files for backward compatibility

samples/{c,cpp,python}     # samples in different languages

modules/              # source code for all the modules

. core/              # former cxcore, cut-down [quite] a bit – drawing and some other functionality will probably be moved other modules

. include/opencv2/core/.h # header files without cx or any other prefix

src/                            # sources and internal headers

test/                           # core tests

vision/  or imgproc/        # former cv or a part of it

. include/opencv2/vision/.h # header files without cv or any other prefix src/ test/

ml/

. include/opencv2/ml/.h src/ test/

highgui/

. include/opencv2/highgui/.h src/ test/

features2d/   # 2d keypoint detectors and descriptors from cv & cvaux

. include/opencv2/features2d/, src/, test/

background_segm/ # background segmentation from cvaux

. .. # also include/opencv2/background_segm/, src/, test/

vidsurv/ or blob_tracking/ # vs part of cvaux

. ..

visual_odometry/ or 3d_reconstruct/  # another part of cvaux

. ..

python/  # former interfaces/python

. ..

ffmpeg/ # former interfaces/ffmpeg

. ..

traincascade/ or traincascade_app/ or app_traincascade # former apps/traincascade

. ..

old/ or obsolete/ or depreciated/ # some obsolete functionality

. ..

utils/           # some helper utilities

• libcxcore.so* → libopencv_core.so*
• cv220.dll → opencv_vision_220.dll
• libcvaux.dyld* → libopencv_features2d.dyld*
  • libopencv_background_segm.dyld*
  • …

then, in order to add a new module, you will need to put the module directory to opencv/modules and add the module name to some global list in opencv/modules/CMakeLists.txt. That’s it.

Vadim

1. The minor maintainance work after # 1.0 release continues. 6 tickets have been closed (## 271, 286, 287, 289, 291, 292) + other two bugs, reported by mail – bug in DFT, bug in drawContours, bug, have been fixed Vadim
2. added 2 new C++ samples – contours# cpp (reimplementation of contours.c) and the incomplete version of object segmentation sample Vadim
3. our OpenCV buildbot-based build system was fixed to build 64-bit OpenCV with mingw-w64 and successfully run Python tests on 64-bit Windows [Alexander
4. started investigation of the googletest and doxygen for the sake of better OpenCV and especially cvaux modularity, to allow users to contribute new code that would have been automatically documented and more easily tested.

Maria Dimashova

Texture descriptor test bench

• I implemented the test on detectors repeatability. Some information about this test is in
  • opencv_extra/testdata/cv/detectors/detector_repeatability_test.txt.
• You may see the results of the test (i.e. the location and region repeatability of detectors) in
  • opencv_extra/testdata/cv/detectors/algorithms/[detectorName]_res.xml files.
• The location repeatability is small in some cases, may be because of inadequate detector parameters fitting. As regards the region repeatability, I implemented it for case of scale invariant detectors only (the region is a circle). But there is a region repeatability criterion for an affine invariant detectors (the region is ellipse) in Mikolajczuk papers too. It seems that OpenCV detectors are scale invariant only. Affine normalization based on the second moment matrices is described in Scale & Affine Invariant Interest Point Detectors, Mikolajczuk. This procedure transforms the circular region of key point into the elliptic region. Affine normalized detectors have higher region repeatability in case of significant images transformations, because the scale change is, in general, different in each direction.
• I think it would be useful to implement this affine normalization, not only for the detectors repeatability test. We can get more stable descriptors for such affine normalized detectors. Am I right? What do you think about this?

Action Items

Gary

• “Finish” off gradient techniques
• Collect clutter data for our version of MOPED

Vadim

• (./) Check the Eigen-cv::Mat image hang problem

Victor

Patrick

James

From last time

Gary

• :-( Write Vincent about epnp license
• :-( Write sample cv_contrib

Vadim

• ~ Set up cv_contrib

Victor

Patrick

James

Agenda

• GSoC: students set. Any issues?
• OpenCV directory structure re-org
• 2D feature set
  • Test system set up with data
  • Unclear what Suat is doing. Compatible?
• VO
• CVPR
• GPU
• 3D box capture latest opencv_extra, go to opencv_extra/3d/tracker3D

Going forward

• CVPR Demo
• Solved problems in vision
• Improved camera calibration
  • Better descriptors
  • Implement ¹
  • 2D bar codes
  • 1D bar codes
  • Infinite grid

¹ F.Moreno-Noguer, V.Lepetit and P.Fua , Accurate Non-Iterative O(n) Solution to the PnP Problem, IEEE International Conference on Computer Vision, Rio de Janeiro, Brazil, October 200# (with code in Matab and C++)

Minutes

• GSOC
  • Features with Nicholas, Victor sync
    *_2D Features_
  • Maria finishing up test bench for the 2D
    • Then on to affine adjustment of features
    • Features stuff might be a good CVPR demo
    • Detectors are in OpenCV, all of them
    • Can use test bench
    • Descriptors are now in cvaux … don’t know if these are checked in yet
  • Allow comparision against SIFT … how to link to it?
    • Allow configuration to include “external” code
  • Alexander working with mono matching. Set of descriptors matched together
    • Move on to case where have 3D view to mono recognition and pose
      *_Tracking a 3D box relative to a chessboard_
  • … https://code.ros.org/svn/opencv/trunk/ – contains both opencv and opencv_extra

• Re-org directory structure
  • Modules build
  • Before check in, test and sample build too
  • Finish this week
    • Then check that the ROS packages still build

• VO
  • Status not changed.
    • Planar SFM
    • in poseest package
    • wrapped pose estimator 2d package as a plug in. Should work then in VSLAM package
      • If so, can reconstruct point clouds etc
  • Hope to finish by next week
  • Suat doing this with stereo pose

Victor

Visual odometry:
- Implemented structure from motion algorithm that consists of RANSAC on top of planar structure from motion (faugeras & lustman), the best configuration is selected by the minimum average epipolar projection error. The algorithm works on scenes that have significant planar parts. A test on simulated data showed that the algorithm is capable of finding a correct solution with 20% points lying in the same plane, with 1 pixel noise in image projections and 5% wrong point matches. Subsequently sparse bundle adjustment is applied to refine both point cloud and extrinsic parameters. The code is in posest package, I am in the process of wrapping it into pe::PoseEstimator2d object that will be integrated into vslam [victor]

Object recognition toolbox:
- a testbench for detectors stability has been implemented according to the approach of Mikolajczyk et al “A comparison of affine region detectors” IJCV’06 paper. The testbench evaluates location and region repeatability for all available detectors (fast, gftt, harris, mser, star, surf). The testbench works with features_2d::FeatureDetector interface so any detector implementing this interface can be added for testing. Descriptors interfaces (DescriptorExtractor, DescriptorMatcher, DescriptorMatchGeneric and their descendants) have been ported to OpenCV cvaux. They will be moved to a special package after Vadim finishes with OpenCV structure reorg. Maria
- An experiment with recognizing textured objects from turntable dataset by matching descriptors between training and test images has been performed. A training dataset consists of several cropped images of each object. For each descriptor in a test image we find the closest descriptors in the training dataset and assign the test image a label of an training object that has the most number of matches. Matches with confidence (defined as [distance to the second closest neighbor]/[distance to the closest neighbor] – 1) lower than a fixed threshold are filtered out. This method gives 100% accuracy on all textured objects (descriptors from background are filtered out by confidence) but the choice of confidence threshold might depend on data and experiment on a larger dataset is needed. [Alexander]

• GPU
  • Frame work to increase form 11 to 14fps on Fermi on 2048×1152 disparity 256, 19 window size
  • Seems this will be good enough for CVPR demo
  • 2 Fermi cards might get this to frame rate

Vadim

The big task being performed last week and continued this week is OpenCV directory structure reorganization. The goal is to make OpenCV more modular and more friendly for the future extensions and users contributions. The current state (not committed to SVN yet) is that the following modules build flawlessly:

core (former cxcore),
imgproc (most of the former cv functionality),
highgui,
ml,
legacy (most the former cvaux functionality),
as well as the new specialized modules:
tracking (optical flow, motion templates, video surveillance part of cvaux),
features2d (the new framework that is being implemented + SURF + StarDetector + FAST + MSER),
objdetect (haar + hog + ferns),
background_segm (3 algorithms from cvaux)
calib3d (epipolar geometry + camera calibration + pose estimation)
flann (bindings for FLANN)

Plans:

1. finish the reorganization, test everything, submit to SVN
2. add Doxygen support to OpenCV build system (automatic documentation generation from OpenCV headers)
3. add Googletest (or CxxTest or …?) support, write 1-2 sample tests.

GPU/Kirill Kornyakov

• Both versions of the stereo BM algorithm implementations were optimized for Fermi.
  • NVidia’s implementation shows 10.8fps and Integral Images implementation shows 11.8fps.
    • This numbers are for our Fermi card and the same 2048×1152 image, MaxDisparity = 256, SSDWindowSize = 19.
• We still maintain two implementations because they are based on different ideas, but they influent each other and this competiton moves us toward.
  During this work we’ve tested several optimization approaches. Among them is SMEM approach, which also was not succesful (we discussed it earlier). And right now we have no significant ideas for further optimization and plan to switch to the algorithmic improvements.

Plans:

• Algorithmic improvements of the stereo implementation.

Maria Dimashova

• I implemented the test on detectors repeatability. Some information about this test is in opencv_extra/testdata/cv/detectors/detector_repeatability_test.txt.
  • You may see the results of the test (i.e. the location and region repeatability of detectors) in
    • opencv_extra/testdata/cv/detectors/algorithms/[detectorName]_res.xml files.
  • The location repeatability is small in some cases, may be because of inadequate detector parameters fitting.
• As regards the region repeatability, I implemented it for case of scale invariant detectors only (the region is a circle). But there is a region repeatability criterion for an affine invariant detectors (the region is ellipse) in Mikolajczuk papers too.
• It seems that OpenCV detectors are scale invariant only. Affine normalization based on the second moment matrices is described in “Scale & Affine Invariant Interest Point Detectors”, Mikolajczuk.
  • This procedure transforms the circular region of key point into the elliptic region. Affine normalized detectors have higher region repeatability in case of significant images transformations, because the scale change is, in general, different in each direction.
• I think it would be useful to implement this affine normalization, not only for the detectors repeatability test. We can get more stable descriptors for such affine normalized detectors. Am I right? What do you think about this? …[Gary things “yes”]

Action Items

Gary

• Talk with Kurt about integrating Suat’s work
• Get object images
• Get more data

Vadim

Victor

Patrick

James

From last time

Gary

• (./) “Finish” off gradient techniques
• :( Collect clutter data for our version of MOPED

Vadim

• (./) Check the Eigen-cv::Mat image hang problem
• ’’in progress_Re-org directory structure

Victor

Patrick

James

Agenda

• Gary is at ICRA 2010 in Achorage. Hope to still hold meeting
• OpenCV and image meta-data. Some discussion with Sarnoff (soon to be absorbed into “SRI”)
  • Maybe we can get them to open/contribute some of their video stitching
• Object Recognition
  • Progress update on features_2d and PnP matching. Stereo to stereo, stereo to mono
  • Sidd wants to contribute:
    • MOPED: A Scalable and Low Latency Object Recognition and Pose Estimation System ‘’Martinez, Manuel, Collet, Alvaro, Srinivasa, Siddhartha. ICRA 2010.’’
    • Basically, we could replace their dependence on SIFT with simple use of any detector-descriptor
• OpenCV re-org into to functionality stacks.
  • Pose estimation should just work across functions (chessboard needs to also put out R,T matrix and quaternion representation. Work more easily with TF from ROS.
  • 3D box relative to calibration pattern — better integrated/higher priority in object recognition stack.
• OpenCV Cheatsheet. C++ and Python. Could be very useful.

Minutes

• Test code
  • Cxx test engine
  • Google unit test
• Possibility of Open source parts of videobrush from Sarnoff?
• Object recognition
  • 100% filter out bad matches with low distance between best and 2nd best match [Alexander]
    • May not generalize. Need more feature
    • Add pose estimation to validate result
    • Close to MOPED, we may get MOPED contributed (talking with Sidd at ICRA)
  • Sync up with Suat
  • See about getting enough feature points on the object for ransac to work
• Stereo
  • Camera info from ROS
    • Hard to convert to OpenCV intrinsic params
    • Need translation 3D point to pixel
• Suat
  • Suat has a wiki page on his work product:
    • http://pr.willowgarage.com/wiki/Stereo_Object_Recognition
  • Currently he’s getting stereo-to-stereo keypoint object recognition (MOPED) working, hopefully first results by Friday.
    • Next is to integrate vocab trees from Patrick for fast pre-filtering over large object databases, and monocular-to-stereo keypoint recognition using Victor’s 2D/3D matching in package posest.
  • Using features_2d,
    • frame_common, posest, sba, and vslam_system packages seems to be paying off in allowing us to collaborate efforts.
• Camera info between ROS and OpenCV
  • Add flag
• With pose make it easy to use any type to any other type
  • Rodrigues, R,T and to quaternion

Vadim

The re-org of directory structure now looks like:

opencv/
include/opencv/               # headers for backward compatibility
modules/

# decomposed cxcore, cv, highgui, ml, cvaux:

core/                 # core functionality
imgproc/              # image processing part of cv
highgui/              # GUI
ml/                   # machine learning routines
video/                # optical flow, motion templates, Kalman filter,
blob tracking, background/foreground segmentation
calib3d/              # camera calibration, epipolar geometry, stereo correspondence
features2d/           # 2d feature detectors & descriptors
objdetect/            # Haar/LBP & HOG object detectors,
fern-based planar object detector
legacy/               # obsolete functionality
user_contrib/         # user contrib

# some other stuff

python                # new-style Python interface
ffmpeg                # ffmpeg bindings
traincascade          # Haar/LBP training application

samples
doc
...

In addition, we should create “OpenCV Central” like Matlab Central.

• cxcore.h, cv.h and other old-style headers include the new style headers, so, in theory, no changes at compile stage are required from user side. However, because of renamed modules, the appropriate changes need to be made at linking stage.
  • In Linux this is handled by the new opencv.pc.
• The whole OpenCV is now builds fine. The new directory structure will be committed to SVN within a couple of days.

Plans:

1. Commit changes to SVN
2. Adjust makefiles for OpenCV-based modules in ROS
3. Add support for Doxygen & googletest.

Victor

Object recognition toolbox:

• Experiments on recognition of textured objects from dataset showed 100% accuracy after filtering of ambiguous correspondences (by comparing the minimum distance to the second closest distance).
  • A PnP pose estimation (from training stereo data and test image data) is in progress. [Alexander]
• DescriptorMatchGeneric wrappers of calonder and fern descriptors have been implemented Maria
• SIFT (imlementation of Andrea Vedaldi) has been added to OpenCV. A DescriptorExtractor and DescriptorMatch wrappers have been implemented. Maria

Visual odometry:

• fixed several bugs during integration of PoseEstimator2d into vslam. The integration is still in progress. [Victor]

Other:

• May 3rd is a holiday in Russia.
• Victor was traveling to Moscow Apr 30 for US visa renewal.
• Alexey is leaving Itseez in May. We thank him for the work he did and wish him a successful career! We are interviewing several candidates now.

Action Items

Gary

• Talk to Scott about a “OpenCV Central” web site
• Contribute focus detector
• Substantial data set for textured objects and pose
• (./) Find out what Suat is doing, send coorinating email
• sync Victor with Sidd about MOPED

Vadim

• Write test system example to user_contrib
• Start OpenCV cheatsheet

Victor

• sync with Suat

Agenda

• Multiple calibration models for calibration and stereo rectification
• Resize RGB vs YUV … where the time is spent
• YUV<—>RGB conversion back and forth test
• Support for projected fringe pattern to get a depth pattern
• New calibration patterns
• CVPR

Depth from Phase or Gray Pattern

• The big idea is to use a projector to throw bands of light onto the scene.
  • Some techniques do a relatively simple gray-code on binary projector images:
    • http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.21.5393&rep=rep1&type=pdf
  • Fancier ones use a sine-wave and phase-shift it each frame:
    • http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.12# 6857&rep=rep1&type=pdf
  • These guys talk more about calibration:
    • http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.11# 5314&rep=rep1&type=pdf
  • Structured light from phase patterns Structured_light_phase_2006_Zhang_High-resolution 3D Zhang Siggraph 2006

Minutes

• Calibration already allows multiple models
• Resize image
  • YUV can be in many formats Y full, U&V half resolution in horizontal or vertical or both
  • Convert→RGB→Resize→YUV
    • Unless each plane is separate, then you can call resize on each plane
• Projected fringe pattern code
  • Post CVPR
• New calibration patterns
  • Infinite chessboard
  • Blob chessboard
  • 3D offset box
• Descriptor toolbox
  • Romain dataset, 7 objects
  • Using ROS, calc stereo for training image, calc point clould, subtract table and background
  • Estimate finish in a day or two
  • Obj rec demo
    • Given a training set with pan tilt, recognize test image and find pose
    • Matching descriptors in test image with training set descriptors
    • Finding images where there are best matches
    • Solve PnP using stereo using Ransac
    • Test data for cluttered scene
      • Need to find in clutter
      • Moped clusters descriptors, localized in space
      • for 7 objects, works at 640×480. Want to scale to 100 objects
    • Set of existing objects is at http://vault.willowgarage.com/wgdata1/vol1/IROS/bags/textured/
      • The package for pan_tilt table is dp_ptu47_pan_tilt_stage
• CVPR
  • Victor: Visa 3rd or 4th week of May. Can be in SiValley June 6th
  • Vadim: About the same schedule ~June 6th
• NVidia
  • Stereo Block Match
  • 30fps with 2 cards
  • Middle next week, will try to implement Hirschmuller … would be nice to get for CVPR
    • Demo: 30fps high res stereo with Hirschmuller
      • Original algorithm is not easy to parallelize
• Stereo
  • Hirschmuller 10x slower than BM on CPU
    • Needs lots of texture since not all context used
      *

Vadim

• Still testing directory structure overhaul.

Victor

Mono visual odometry: implemented a plugin of PoseEstimator2D to vslam framework. First pair of frames is initialized with SFM and every next frame uses 3D point clouds from previous frames to solve PnP problem. Spent the most of the time solving small problems. I am still debugging the code now. [Victor]

Object recognition toolbox: implemented automatic segmentation of object mask from background and pan-tilt table. Implemented a RANSAC version of solvePnP, it is in debugging stage. [Alexander]

Detectors/descriptors testbench: spent some time looking at Mikolajczuk’s MATLAB code detectors/descriptors quality estimation and making OpenCV testbench closer to it. Implemented a new area-based metric for estimating detectors invariance. Implemented a function for estimating descriptors matching quality. Maria

Other: four new candidates interviewed for a replacement of Alexey. We are going to make an offer to one of them this week.

Action Items

Gary

• (./) Send Dallas resize explantion
• Close out YUV→RGB conversion bug
• Ask about hotel accomodatation
• Get dataset to Victor for object
  • Individual objects
  • In clutter
  • In refrigerator
• Talk to Scott about a “OpenCV Central” web site
• Contribute focus detector
• Substantial data set for textured objects and pose

Vadim

• Write test system example to user_contrib
• Start OpenCV cheatsheet

Victor

• Send Gary existnig textured data link

From Before

Gary

• :( Talk to Scott about a “OpenCV Central” web site
• :( Contribute focus detector
• :( Substantial data set for textured objects and pose
• (./) Find out what Suat is doing, send coorinating email
• :( sync Victor with Sidd about MOPED

Vadim

• Write test system example to user_contrib
• Start OpenCV cheatsheet

Victor

• (./) sync with Suat

Agenda

• Textured object recognition
• Test system
• Cheat sheet
• Android OpenCV (from GSoC)
  • Stitching interface
• New calibration patterns
• Future
  • Support for fringe patterns
• CVPR
  • NVidia
  • Robots
• C-turtle
• Stereo

Depth from Phase or Gray Pattern

• The big idea is to use a projector to throw bands of light onto the scene.
  • Some techniques do a relatively simple gray-code on binary projector images:
    • http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.21.5393&rep=rep1&type=pdf
  • Fancier ones use a sine-wave and phase-shift it each frame:
    • http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.12# 6857&rep=rep1&type=pdf
  • These guys talk more about calibration:
    • http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.11# 5314&rep=rep1&type=pdf
  • Structured light from phase patterns, See May 18 2010: ‘’Zhang Siggraph 2006’’

Minutes

• Need to get team members able to commit to ROS on Willow’s site
• Current algorithm works for all 5 objects, except 2 of them have non-textured sides
• Need data now
• Textured object recognition
  • Some detectors are in features_2d in outlet detection
  • module called features_2d that implements everything. It’s in trunk
    • has detectors, descriptors and training net
    • Moped approach is being implemented as a ROS package
  • Model needs stereo keypoints in 3D, keypoint match, spatially cluster keypoints (one object), ransac and geomeric check
    • Keypoints are found
    • clustered spatially (per object)
    • Ransac to find pose (PnP)
    • Geometric check
  • Follows Moped pretty much, use different kind of clustering, use arbitrary detectors/descriptors. Don’t match point clouds, we use PnP
  • Questions
    • Does it work with other detectors and descriptors
      • STAR and SURF are used right now
      • Plan to try with SIFT also
      • Need to try Calander
      • Multi-scale haris
      • Multi-scale FAST
  • Suat
  • Need to get data now — this is the bottleneck right now
• SURFALS Surface patches in mesh
• Stereo
  • Wavy pattern in stereo … half pixel issues.
    • Rectify 2 images half-pixel offset. Take the best match but never at a half pixel off, at most a quarter pixel off
      • Hirschmiller, Vadim used the Birchfield matcher — computes the minimal difference within half-pixel of each location (interpolation)
        • Link is Stereo by Birchfield and Tomasi
    • Use large and small windows at the same time.
• Eigen interface
  • Header files.
  • Alignment issues
  • Now have conversion wrappers so can mix and match Eigen
    • will put out doxygen comments
• C-turtle, 23rd of May.
  • Using trunk of OpenCV with ROS
  • ROS trunk should try to track OpenCV Trunk
    • Trunk (most recent) Is # 1+, but not very latests
    • Released version (latest, a little behind) —> C-turtle
    • Box turtle (old stable)
• VO
  • Bugs in monocular VO
  • Prints point clouds to consol
    • SFM spoiled to rviz
    • Some transform SBA (a lot of programs use the transform from world to camera, but we use the coords of camera in the world)
  • Run stereo vs mono
  • (Full vslam now works with loop closure) Bundle adj of all the points, subsecond times
• Android
  • Give Ethan OpenCV SVN access to debug the code
  • Then maybe a month or so later we can merge to trunk

Victor

Visual odometry:

• debugging the code.
  • Fixed several bugs, including a major bug in computing inliers.
  • Right now the point cloud gets corrupted somewhere in between initial structure from motion estimation (R and T look quite sensible too) and rviz where the point cloud does not make any sense. [Victor]

Object recognition:

• refactoring of ransac version of PnP.
  • Experiments on a larger test dataset show stable recognition of Green tea, Pattern classification and tuna salad, and unstable recognition on Blue people and Campbell soup__when__they are viewed from untextured sides.
• Implemented hierarchical clustering and included it into the pipeline for recognizing several objects in the same scene
  • MOPED uses mean shift segmentation that we don’t have and it will take some time to implement it,
    • we have k-means but it wouldn’t work on this type of data — a single cluster with scattered outliers, hierarchical clustering was easy to implement and it is a good proxy for mean shift on this type of data.
    • The attachments contain the results of clustering of keypoints matched to a single object and the inliers corresponding to the best solution of ransaced PnP among all clusters.
  • We urgently need data with several textured objects in the same scene to test on. [Alexander]
    Detectors/descriptors testbench:
• implemented a helper for computing descriptors that can choose keypoints either from applying a detector or by projection from another image with a given homography.
• There is an option of filtering keypoints based on confidence.
• Tests for surf and sift descriptors are implemented. A problem with sift crashing has been resolved. All changes are merged into opencv trunk.

Other: Ilya Lysenkov will start working for Willow project instead of Alexey from May 24th.

Vadim

1. Finished reorganization of OpenCV directory structure, fixed some critical problems caused by the reorganization.
2. Studied Doxygen, integrating it to OpenCV build system and added some experimental documentary comments to OpenCV headers. As a result, now Doxygen detection and execution is now integrated to OpenCV CMake build system. HTML docs (currently covering most of cxcore) are automatically generated.

Here is how cv::Mat documentation looks like (to get the proper look, save all the 3 files into the same directory).

Plans:

1. finish with Doxygen.
2. add Googletest + some experimental tests
3. make a short document on how to prepare user contributions.

Action Items

Gary

• Get other team members on ros commits
• Get data to Victor

Vadim

Victor

• Send James message about OpenCV revision to go into C-Turtle ROS

From Before

Gary

• (./) Send Dallas resize explantion
• Close out YUV→RGB conversion bug
• (./)Ask about hotel accomodatation
• Get dataset to Victor for object
  • Individual objects
  • In clutter
  • In refrigerator
• Talk to Scott about a “OpenCV Central” web site
• Contribute focus detector
• Substantial data set for textured objects and pose

Vadim

• Write test system example to user_contrib
• Start OpenCV cheatsheet

Victor

• Send Gary existnig textured data link

Agenda

• features2d
  • Test program
  • Example of how to add
• OpenCV and ROS … good to go?
• User contrib
• CVPR
  • 6/15/2010 17:20—21:00 Robot Vision and Manipulation on Willow Garage’s PR2 (Tuesday)
  • 6/16/2010 17:20—21:00 OpenCV on NVidia Steriods (Wednesday)
• Calibration pattern generalization
• 1 and 2d bar codes
  • Pipeline for getting homography from marked points
• GSOC results
  • Android port … trouble with cmake
  • Stitching

Depth from Phase or Gray Pattern

• The big idea is to use a projector to throw bands of light onto the scene.
  • Some techniques do a relatively simple gray-code on binary projector images:
    • http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.21.5393&rep=rep1&type=pdf
  • Fancier ones use a sine-wave and phase-shift it each frame:
    • http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.12# 6857&rep=rep1&type=pdf
  • These guys talk more about calibration:
    • http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.11# 5314&rep=rep1&type=pdf
  • Structured light from phase patterns, See May 18 2010: ‘’Zhang Siggraph 2006’’

Minutes

• Features2D
  • Algorithmic test
  • Victor to send test instructions
  • Can we get homography to represent a change in view point of “X” degrees — this is good for testing purposes
    • R,t surface normal and distance to surface. And returns the homography
      • Plane and rotation and translation
      • perspective warp
  • Textured object detection
  • Looks like enough resolution on the new stuff, but didn’t get bags, got images, so no 3D
  • Suat workking on — combine things into a single point cloud/object
    • Want to have him working on creating 3D models
  • Documentation finished tomorrow
  • 2 different things:
    • Images in — visualize matches
    • What Kurt wants
      • Stereo images with camera parameters
        • Have test set and training set
        • Train on training set, makes viewpoint based model
        • Run on test set and see how you do
• R and T in different form
  • polymorphic
  • seq, cvmat
  • In c api, tried to make functions accept different forms of matrix
• Python
  • numpy arrays standard
  • Functions are polymorphic, so could work with
  • Tickets to make C and C++ more polymorphic
• Doxygen
  • Simplify documentation of new functionality
  • Old functionality would be shallow
  • 1 or 2 documentation systems?
    • Contributed functionality →Doxygen
    • New functionality →Doxygen
  • Only for user contrib
  • Stick with Latex→Sphinx
• OpenCV and ROS
  • Headers in ROS packages
    • cv.hpp cvaux.hpp caused problems
    • Victor added proxy headers so that packages will compile
      • Still one package: Outlet pose
      • Need to change outlet pose, not enough to just put in proxy headers
• Work on CVPR tutorial when Vadim gets in
  • Cheatsheet
• Vadim June 4th
• Victor …

Maria on testing 2D features

• I implemented the test on detectors repeatability.
  • Some information about this test is in opencv_extra/testdata/cv/detectors/detector_repeatability_test.txt.
  • You may see the results of the test (i.e. the location and region repeatability of detectors) in
    • opencv_extra/testdata/cv/detectors/algorithms/[detectorName]_res.xml files.
    • The location repeatability is small in some cases, may be because of inadequate detector parameters fitting.
  • As regards the region repeatability, I implemented it for case of scale invariant detectors only (the region is a circle).
    • But there is a region repeatability criterion for an affine invariant detectors (the region is ellipse) in Mikolajczuk papers too.
      • It seems that OpenCV detectors are scale invariant only. Affine normalization based on the second moment matrices is described in
        • “Scale & Affine Invariant Interest Point Detectors”, Mikolajczuk.
      • This procedure transforms the circular region of key point into the elliptic region.
      • Affine normalized detectors have higher region repeatability in case of significant images transformations, because the scale change is, in general, different in each direction.
• I think it would be useful to implement this affine normalization, not only for the detectors repeatability test. We can get more stable descriptors for such affine normalized detectors.

Vadim

1. “Doxygenated” the following OpenCV modules (C++ part only): core (finished), imgproc, calib3d, video, features2d, objdetect, highgui.
2. Started integration of googletest into OpenCV project tree.

Plans:

1. finish with Doxygen, improve the appearance of the docs, experiment with the search
2. add some experimental googletest-based tests
3. make a short document on how to prepare user contributions.

Victor

Object recognition: bag files captured by Radu show low texture/resolution, there are not enough correct matches.

The dependence of correct matches on resolution has been investigated: images of “Pattern Classification” book and green tea box ware taken by a consumer IP camera from different distances and inliers were found by ransac+homography calculation. The results are presented below. 1 meter is a critical distance after which homography calculation is unstable for the book that is textured. Tea box is detected from 0.5m only for 640×480.

Results:
Inliers count
distance   640x480                1280x720
about 1m     25                       44
about 0.7m   68                       86
about 0.5m   109                     220

Similar experiment with Greenfield tea.
640x480                1280x720
2                          6
0                          7
8                          15

An algorithm for automatically extracting training images from a bag file is implemented. It incrementally adds new frames that have low inlier count with existing training set. The documentation is in progress. The new package name is textured_object_detection.

Detectors/descriptors: implemented support for angle recalculation in sift wrapper (similar to surf). As a result sift descriptor shows better matching that surf on test data. Fixed problems with setting keypoint scale KeyPoint::size for several descriptors (sift, gftt, mser).

OpenCV/ROS: fixed compilation problems for several ROS packages induced by a new opencv structure. Added a patch for opencv that fixes the most of compiling problems for tags/latest revision of packages.

Kirill/GPU

Accomplishments:

• Current code ported to the NVidia Driver API and debugged. Multi-GPU version have been created, it shows 19.5 fps on Fermi+Quadro configuration, where Fermi does 75% of work and Quadro 25%. We measure work time of only GPU matching, other stuff like color conversions and visualization is out of consideration.
• Some activities with the DemoCVPR application:
  - OpenCV stereo BM and SGBM (Hirschmueller) added. Switching in realtime between algorithms implemented.
  - Information window created (performance, stereo parameters, resolution).
  - Refactoring in order to get more flexibility, first of all for integration with camera code.
• Started to think about porting Hirschmueller and Daisy (http://cvlab.epfl.ch/~tola/daisy.html) algorithms to the GPU. Here we need to estimate time needed for implementation.

Plans:

• Tune the multi-GPU version.
• May be integrate with Joe’s code, that receive frames from cameras and do deBayer transformation.
• Design GPU versions of the Hirschmueller and Daisy, probably start this work.

Action Items

Gary

• Test out user contrib process
  • Contribute focus detector
  • Contribute anysize pyramid/laplacian code
• Close out YUV→RGB conversion bug
• Hotel accommodations for Vadim

Kurt

• Send HD stereo with calibration camera parameters

James

• Eitan to turn calibration R&T translation difficulties into tickets

Vadim

Victor

• Send Gary a instructions on how to use the features2d test program

From Before

Gary

• ? Get other team members on ros commits
• (./) Get data to Victor

Vadim

Victor

• (./) Send James message about OpenCV revision to go into C-Turtle ROS

Agenda

• Matching
• VO
• cvpr
• svn

Minutes

Matching

• At high resolution, FAST+SURF is working fairly well for doing pnp ransac verification
  • Alexander and Maria on this
• Data collection
  • Taking Tea box through all rotations in video to try to get model
  • Dataset at http://itseez.com/data/canon-base-1.0.tar.gz

VO

• SFM generates decent pose estimation
• But full pipeline is producing bad results in rviz
• Wim wants VO for Texai which only has monocular
  • Have wheel odometry to help

Documentation

• Thinking of possibility of writing directly in Sphinx
• Victor still no visa
• Vadim on the 4th

Victor

Object detection:

• created a dataset with a stereo pair of Canon 40D cameras. Each image has been scaled down to 1280×853, a training set of 3 objects has been created. Initial experiments show promising results: there is a lot of keypoints and matching works well enough for ransac+pnp to find a sufficient number of inliers. Some examples of inliers and correspondences are attached. [Alex]

Detectors/descriptors:

• a new sample keypoints_matching.cpp has been implemented. For a single input image it generates its random perspective transformation and matches descriptors from original and transformed images. It can optionally calculate inliers with ransac+findHomography. clear() method has been implemented for !GenericDescriptorMatch interface. Maria

OpenCV:

• there has been a number of issues with building ros packages against the latest opencv, including deprecated headers and several changes around flann namespace. All of that has been fixed and the new opencv has been released together with vision_opencv 1.# # [Victor]

SVN

• Eliahoo can’t
  • Create account
  • request an account
  • can’t check in
  • Nathnan not too responsive about this
  • Brian asked

Vadim

Here is the progress:

1. A short break was made with Doxygen expansion after some problems with it have been discovered. In particular, Doxygen does not distinguish between these 2 functions:
   

   
   //! computes convex hull for a set of 2D points.
   CV_EXPORTS void convexHull( const Mat& points, vector<Point>& hull, bool clockwise=false );
   //! computes convex hull for a set of 2D points.
   CV_EXPORTS void convexHull( const Mat& points, vector<Point2f>& hull, bool clockwise=false );
   

   • because Point and Point2f are different specializations of the same template class Point_<>. It looks like similar problems have been reported a couple of years ago in Doxygen bug tracker.
   • Now considering RST (Sphinx) as a possible alternative.
2. Some OpenCV .tex files have been fixed to produce cleaner HTML OpenCV docs without garbage unprocessed “:math:`…`” strings. Also it’s now possible to build PDFs out of the generated rst files, so in theory the rst can be the original format.
   • That is, instead of (latex→pdf & latex→rst→html) we can produce both online and offline documentation from rst (via rst→html & rst→latex→pdf).
3. The ancient OpenCV mechanism for reading and writing “registered” data types, such as CvMat, CvHistogram, CvSeq etc. has been updated to support the new C++ structures more easily using cv::RTTIImpl<> class. cv::HOGDescriptor has been used to test this. This functionality is going to be used for the currently implemented descriptor toolbox.
4. 5 tickets (various patches from users) have been closed (## 330, 355, 356, 359, 360).

Action Items

Gary

• Get Vadim a place to stay
• (./) Complain about Ilya Lysenkov not being able to get commit to SVN

Agenda

• CVPR
• Features 2D (Introduce at CVPR Tutorial
  • Documentaton

Minutes

• CVPR
  • 10 to 15 foils for Features 2D
    • Have people do sample code, add to it
• Demo at CVPR
  • Finds pose of object in clutter
  • Feature 2D
    • New dataset http://itseez.com/data/canon-base-1.1.tar.gz
  • Joe will run NVidia demo
    • Demo runs on one Fermi card (90ms/frame) correlation + post processing on CPU
    • Fermi and Quadro (slower card)
    • Expect 20fps ultimately
    • Can get working in parallel — process on GPU, post process on CPU in tick tock fashion, probably 10ms per card speedup
    • Integrate with real camera
• Object Recognition
  • Scaling
  • Clutter
  • Interface for window based search (Patrick did)
    • Called Features2D grid Adapter class
  • Sparse descriptors in dense framework

• GSOC, got progress
  • Victor:
    • Whole pipeline running for Pascal dataset using bag of words
  • Gary:
    • Port of OpenCV to Android with build system nearly done using Android make system
    • Shperical Image stitching and blending version 1 pretty much done
  • Vadim:
    • Implemented GUI part of HighGUI in QT

• In features_2d, we have the grid_adapter.h class
  

  
  #ifndef FEATURES_2D_GRID_ADAPTER_H
  #define FEATURES_2D_GRID_ADAPTER_H

#include <features_2d/detector.h>

namespace features_2d {

/**

• \brief Adapts a detector to partition the source image into a grid and detect
• points in each cell.
  */
  class GridAdapter : public FeatureDetector
  {
  public:
  GridAdapter(const cv::Ptr& detector, int max_total_keypoints,
  int rows = 4, int cols = 4)
  : detector(detector),
  max_keypoints(max_total_keypoints),
  rows(rows), cols(cols)
  {
  }

cv::Ptr detector;
int max_keypoints;
int rows;
int cols;

protected:
virtual void detectImpl(const cv::Mat& image, const cv::Mat& mask,
std::vector& keypoints) const;
};

} //namespace features_2d

#endif
grid_adapter.h (END)

Vadim

Here is the progress:

1. Several bugs (findHomography(), cvLoadImage() …) have been fixed (trac tickets: ##304, 338, 369, 370)
2. OpenCV cheatsheet is in progress. It’s being done in Sphinx. The current very initial version is attached.

Victor

• Object detection:
  • created a new dataset http://itseez.com/data/canon-base-1.1.tar.gz, consisting of 10 textured objects. The dataset is captured with Canon 40D stereo pair and all images are resized to 1280×85# [Alexander]
  • Experiments on high resolution images show good recognition rate for all objects except for ‘stapler’ (that is textureless) and ‘diploma’ (is being investigated now).
  • An example of detection is attached (test image and rviz point cloud, each color corresponds to an object: roze – 3M disks box, yellow — fallout cd, white — pcmcia adapter box). Experiments on low resolution (640×480) show unstable matching and recognition, close to what we saw on bag files captured by Radu. [Maria, Alexander]
    *_Visual odometry:_
  • Implemented a test system for the whole mono pipeline. The test system feeds sba::voSt object with keypoints obtained by projecting a simulated point cloud to simulated camera positions. Several bugs have been corrected and the pipeline works correctly on simulated data now. A test on new_college shows reasonable camera trajectory (straight line diverging to the right) and point cloud (visible ground plane, although noisier compared to stereo) on the first several seconds and then both become obviously corrupt. [Victor]
• OpenCV: added one way descriptor into descriptor testbench. Modified one way descriptor interface for a more convenient parameters input/output (single file instead of several). [Ilya]

Kirill/GPU

Accomplishments:

• DemoCVPR performance measuring and tuning. Whole processing pipeline initially run with 8.5-10fps instead of 16fps (matching alone) on a single GPU. We are working on optimization of the whole flow. Currently we run speckleFiltering (the most time consuming CPU procedure) in parallel with other routines and it gives 1# 5fps with a single GPU.
  • Working on user manual for CVPR demo. Document will be finished soon
  • Small activities:
    – fixed bug in GPU-kernel – sometimes disparity used to round to nearest multiple of 8.
    – experiments with Middlebury 2005 and 2006 datasets.
    – work with config improved (config name received as cmd parameter, class AppSettings created, some settings renamed).
    – tried moving winsize from preprocessor definition to variable (next step is to move to the config). Have got significant performance decrease due to stop of loop unrolling by compiler. So, this activity was cancelled.

Plans:

• Integrate with Joe’s code that works with SI camera.
• Finish user manual.
• Clean DemoCVPR.

Action Items

Gary

• Find out about Texai use at CVPR
• Talk to Matai about integration if we can get Radu’s images working
• Talk to Joe about NVidia demo
• Make sure Marius knows about Patrick’s work

Radu

• Get objects in database to Victor

Victor

• Write Features2D documentation and tutorial

From last time

Gary

• (./) Get Vadim a place to stay
• (./) Complain about Ilya Lysenkov not being able to get commit to SVN

Agenda

• Test system
• User contrib
• Features2D
  • Obj Rec
  • Testing framework
  • VO
• Object Rec stategy/goals
• Generalizing calibration/feature points

Minutes

Features2D

• Lowered threshold and switched off non-max suppression on FAST
• Get 5-10K points per image instead of 10s
• Upped the recognition
  • 100’s of inliers per object up from like 3
• Looking at better clustering techniques:
  • When find correspondence for test descriptors
  • Look for cluster of descriptors corresponding to the same object and near each others
  • Run ransac on the clusters only
• Forward backwards check:
  • Image A and B. Take point in A and find correspondence in B. Do the same thing from B to A. If not, drop.
• Ratio test from SIFT features
• Ransac is the bottleneck right now. Should be fast if number of iterations is low, but for us its high
  • Moped doesn’t do ransac, they randomly chose 5 correspondences, if low inliers they throw away. Do it withing object cluster
• Need some ways of speeding up
  • Prefilter
• Plans
  • Recognition rate test with differing number of ransac and interations
  • Forward backward check
  • Ratio test
• S. in touch last week, replicating Moped
• Alexander doing this, Maria does this but on vacation.

Mono VO

• Victor getting occasional loss of tracking
  • Choice of keyframes is important
  • Again turning down threshold on FAST and using forward back check
  • Smoothing prior will help
    • Helps with small number of inliers too
• Basically VO works on short sequences
• Incorporate changes to SBA that Helen has been making
• Almost ready to play with
• PTAM can get pretty good results on table top tracking
  • PTAM might have a relocalizer in it. They also have trackers on each point
  • We’ll be matching frames instead.
  • Want to compare Mono against

Test system

• Are not reporting what failed
• OpenCV vs ROS: Cannot just report what test failed since we run across several operating systems
• Either remove test temporarily and issue a Ticket
  • Go to Trac and can see which test were failing
• Haven’t implemted a log function to be OS specific
• Hudson would be more convenient
• For now, black list and raise a ticket.

Stereo

• Vadim run on stereo data using Hirshmuller
• Victor: NVidea wants dense stereo

Object Recognition

• Marius is putting together a recognition pipeline to the level where a non-computer vision coder can train objects
• Scaling is an issue
  • Getting it over 10 objects
• Pre-filter, recognizer and verifier
• Large dataset in same format
  • Using Romain’s data format
  • 1Mpix camera, have little data
  • Can deal with messages not being there (pan tilt unit)
  • Make a public data set test bench
• Pairs of line features

General Calibration Object detector

• Victor will send time estimate.

Victor

• Textured object recognition: a considerable improvement in recognition of objects from Radu 640×480 dataset http://vault.willowgarage.com/wgdata1/vol1/objects_pantilt_database/ is caused by lowering the threshold of fast detector. This resulted in thousands keypoints per image instead of hundreds, and it looks like repeatability rate was the main reason for a small number of inliers. A side effect of this is a lower ratio of inliers to the total number of matches so we have to use a much higher number of ransac iterations (10-100K instead of 1K). Below are some examples of recognition and pose estimation. [Maria, Alexander]

Figures

• Image recognition and pose:
  • 
• Inlier points
  • 
  • 
• Pose
  • 

• Shape recognition: PAS features have been integrated into textured_object_detection and first experiments with estimating pose of a textureless object were carried out. While there still are lots of issues (clutter, edges occlusions, a high ratio of inconsistent edge correspondences), there are some promising results (see images below) for objects that have low texture. [Ilya]

• Line pairs for recognition
  • 
  • 

• Visual odometry: brought the code up to date with the latest changes in voSt and sba. There still are issues on the full new_college sequence, the tracking is lost after some time due to a small number of inliers. Several experiments with keyframe selection showed that PTAM method (based on average distance to the camera) is preferable to other empirics. [Victor]

Vadim

The last week was spent in fixing various bugs:

- tickets ## 347, 388, 413
- multiple crashes in the tests because of some CvMat::type and CvSeq::flags bit field reorganization. At the same time, XML/YAML I/O functions for sequences have been modified to avoid dependency on the binary representation of the structures.
- compile errors with GCC # 1 on Ubuntu 8.04 (by disabling precompiled headers)
- incorrect image rendering and memory leak in waitKey() (reported by Tandent engineers)

Plans:

1. do experiments with Hirschmuller algorithms on high-resolution images
2. distribute latex documentation by modules
3. video stitching?

Action Items

Gary

• Write Tamir for clarification about devices in November whether the data is open for others to use. Better vision camera in conjunction with it?

_Vadim_’

Victor

• Get estimates of time for plannar homography calibration patterns.

Kurt

• Find out what Suat is doing

From last time

Gary

Vadim

Victor

Agenda

• Get estimates of time for general_plannar homography calibration patterns. (?)
• VO timeline
• Feature object rec
• NVidia plans
• Latex docs … bugs
• User contrib
• Test system
• Object recognition plans

Minutes

Plannar calibration patterns

• Outlet detection/blob detection from ROS into OpenCV to build a calibration object.
  • Given a pattern, can find calibration pattern
  • Would take 2 days to a week

Bar codes

• 2D barcodes would be nice, James has wrapped a library in Python
  • Don’t know if we should implement

Mono VO

• Testing on camera around table
• Quite a few parameters, trying to understand how they influence results
  • Sometime tracking is lost since all inliers are lost
• Implement regularization?
  • When tracking is lost, we’re lost
• Initializes with SFM and then uses SolvePnP
  • If SolvePnP fails, we stop. We should regularize by going back to SFM (no scale) and regularize the scale by interpolating camera motion then back to

SolvePnP

• We don’t have place recognition/loop closer yet, but it is in place in stereo slam
• Need to unify the external files for vocab trees. They are hard coded, they should be downloadable.
  • Patrick has already put these on Vol1.
• Stereo VSLAM seems to be working now. Good results on outdoor data
• Smoothing paper was sent for motion.

Object Recognition

• Need data
• Create a flexible “tool kit” for future recognition needs
• Currently, our detector-descriptor recognition approach is:

1. The training set consists of object images together with 3d points computed from stereo. Stereo is used to find object mask. Keypoints are extracted from each object ( FAST, threshold=5, nonmax suppression is off) and descriptors (SURF) are computed.
2. For each test image
   a. Find keypoints and compute descriptors ~ 0.2 sec
   b. Find the nearest training descriptors to each test descriptor (using L1 distance) ~ 30-80 sec
   c. Label each test keypoint with an object id of the corresponding training keypoint
   d. For each object id
   1) cluster test keypoints with the same object id. Hierarchical clustering is used with a stop criterion on distance between cluster centers (max distance = 100). Test points with correspondences from different images are not allowed into the same cluster ~ 0.01 sec
   2) Select clusters with keypoint count higher than a threshold (threshold=50)
   3) Run RANSAC SolvePnP on each cluster (iterations count = 100) ~ 0.1-2 sec (depends on clusters count)
   4) Each cluster with the number of inliers higher than a threshold (=50) is recognized as an object with the current object id.

Textured objects

• Object recognition working better, but takes 1 minute
• How to speedup?
  • Indexing by KDtree or FLANN (but FLANN uses Euclidian and Victor uses L1 distance which works better)
    • For keypoints to reference
  • Pose recognition is faster now
  • 13 objects, 5 to 10 poses
• How many reference stereo pairs are we checking. We use clustering so it’s probably around 10 clusters per object
• PrimeSense will be commercially available in December. We

GPU

• Merging block matching on GPU with OpenCV by implementing an API for this
• Make it work with Linux and ROS
• Comparing Hirschmuller with BM
  • Preliminary results is Hirschmuller is better but not critically better
• Decision point of what to do next with Stereo

Stitching

Docs

• Split into modules
• Is there an easier way to temporarily enter changes?

Buy another server? I7

Victor

Textured object recognition: a new test base has been created, consisting of a training set of 13 objects and a test set of the same 13 objects with 20 test images per object http://itseez.com/data/pantilt_640x480.tar.gz. The latest results for this dataset of textured_object_detection are summarized in the table below. Only two objects are not recognized robustly (All and Blue people). Recognition has been speeded up from 5 minutes per test image to 1 minute. The most important change is a new version of clustering algorithm that does not merge test keypoints with correspondences from different training images into one cluster. As a result a lot of small clusters can be filtered out and pose estimation is done on few per test image. The current bottleneck is in matching, a new version of BruteForceMatcher that supports indexing for both L2 and L1 is on the way. [Alexander]

object name recognition count (the number of correct recognitions per 20 images)

Shape recognition: Added many-to-many feature matching that filters correspondences by a distance threshold. Added filtering of correspondences by doing hough transform. This resulted in a better detection of a textureless object from Radu dataset (see the image below). This algorithm is sensitive to a silhouette of the object and thus the current version of pantilt_640x480 dataset cannot be used (it crops object edges). A manual operation will be needed to recreate this dataset with acceptable object masks and then an experiment on this dataset will be performed. [Ilya]

Visual odometry: mono visual odometry is integrated into the full pipeline so that stereo and mono can be run together without code recompiling. A table video sequence has been shot with Canon 500D in HD and resized down to 640×360. There are two issues: 1) after some time tracking is lost, like in new_college, 2) there are quite many points for which the distances to the camera where calculated incorrectly (they are much closer to the camera than they are), because a wrong match is occasionally considered as inlier (there are quite many keypoints used for tracking with FAST threshold varying from 5 to 10) [Victor]

OpenCV: after a period of active development buildbot shows a lot of problems on all systems. All build problems have been located and corrected (thanks to Vadim!), as well as some of the tests. Ubuntu32 builds and runs tests successfully now, some problems still remain in Ubuntu64 and Windows, as well as a technical problem in Rosbuilder. [Alexander, Vadim, Victor]

Vadim

The last week was spent in fixing compile problems and test failures:
- closed tickets ## 376, 415, 431, 440, 447, 449, 461 + the patch for OSX
- made OpenCV compile with GCC # x
- studied papers on belief-propagation-based stereo correspondence algorithms: by P. Felzenswalb and the later “constant-space” modification .

Plans:

1. do experiments with Hirschmuller algorithms on high-resolution images
2. distribute latex documentation by modules
3. try BP-based stereo correspondence

Baksheev

Accomplishments:

• Created OpenCV GPU API draft. Began its implementation and integration with OpenCV.
• Investigated differences in results between OpenCV CPU and GPU implementations of Stereo block matching algorithm on Radu’s dataset with texture light from WillowGarage. Main reasons of the differences are that OpenCV uses Sobel with threshold and SAD instead of SSD. After doing that with GPU implementation, we got very similar results. I attached screenshots that demonstrate Sobel influence. I think we should add options to prefilter with Sobel, when we port to OpenCV GPU API interface. A comparison of block matching and Hirschmuller approaches on various stereo pairs has been performed. The details will be sent out in a separate message.
• DemoCVPR was ported back to Cuda runtime API (for further integration with OpenCV) and to Linux. Created CMake-files for it. So next we can integrate the code in textured_object_detection module in ROS.
• Read papers about Hirschmuller’s papers about its CPU and GPU implementation. Investigated approaches to implementing Hirschmuller’s stereo on GPU. We could parallelize different directions on grid level and inner loop by disparity on threads block level. First dumb implementation might take 3-5 days. After we will see all performance bottle necks and problems, and whether it could be faster on GPU than on CPU. Also we investigate belief propagation based matching method, which is slow but may be parallelized on GPU very good. A lack of BP is that we don’t have prototype that works on CPU with good quality.

Plans:

• Finish implementation of the API draft. (Some functionality requires writing GPU code for different image formats).

• Integrate stereo block matching on GPU into OpenCV.

Action Items

Gary

• Ask James about how well the 2D bar code is matching.
• (./) Ask Joe about GPU priorities?
• (./) Ask Steve about purchasing a new server

Vadim

• Think about is there an easier way to temporarily enter documentation changes

Victor

• (./) Send short written description of the texture feature algorithm and which parts are fast or slow.
• (./) Send quote for additional server

Kurt

• {?} Try VO on New Collage sequence.

From last time

Gary

• (./) Write Tamir for clarification about devices in November whether the data is open for others to use.(Nov_) Better vision camera in conjunction with it? (_Yes, possible)

Vadim

Victor

• Get estimates of time for plannar homography calibration patterns.

Kurt

• Find out what Suat is doing

Agenda

• Android port
• 3 or N camera stereo
• Documentation status
  • Entering temporary changes
• User contrib
• Test system?
• Object recognition
  • Test framework, features2D

Minutes

Android

• If all the make files can be put into a separate directory with instructions to build
• CMake is flexible enough to support Android “out of the box”, but maybe they will create Macros to make it easy and we can then support the Android port with CMake
  3 camera stereo
• We have 3 cameras that we need to calibrate together
• All 3 cameras could see a calibration, but we might want to make it more flexible so that you can calibrate N cameras with only pair-wise calibration
  • But it’s not likely to generalize to N cameras
• Stereo calibration
  • Each camera gets calibrated for intrinsics
  • Then external calibration
  • Then internal
• The image planes have to end up all in the same plane, so there isn’t
• Ideas
  • Take images from 1 pair of stereo and the other pair of stereo and then take rotation and translation to solve
  • Main pair is the one that sees the most views (?)
  • Or take 2 pair with a common camera A with B and A with C. Two sets of R and T.
    • Then A has to be consistent with B and C
    • Fix A camera rotation so that the plane has the image centers of all of cameras
• Final goal is registration so that, let’s say A and B are for depth and C is for color image to be registered
  • All images still need to be rectified so that we can get the correspondences for the pixels at different depths
• Epipolar geometry:
  • A and B have to be stereo rectified to do stereo quickly (line search)
  • Then want to ’’color_it with C
  • Get pixel in A, know epiline in C, know its depth from A+B, find the point in C that corresponds to that depth
    • Parallax will be off for close things
  • Since we have 3 cameras in a row A on left, B on right, C close to A but on its Right
    • To get rectification most easily: A and B set the stereo plane. Then C is set parallel to that plane and a scale has to be set

Victor out to the 26th.

Features2D object recognition

• Haven’t built and tried here
• Looks like it runs under ROS
• Testing framework
• Generic descriptor matcher demonstrates use of abstract interface for comparing keypoints. Just uses the match interface such as ferns
• The general tester is under …/tests/cv/src/adetectordescriptor_evaluation.cpp
• Goal
  • list detectors and descriptors to be used and test them against a new detector or descriptor

Documentation

• Reorg in progress
• Can check in now
• Wiki page link where users can just type in documentation notes
  • We then edit these changes into the Latex

Vadim

The last week most of the time was spent in fixing compile problems and test failures:

• closed tickets ## 99, 119, 334, 375, 377, 434, 435, 438, 448, 452, 45#
  studied the following 2 papers about stereo correspondence estimation using belief propagation:
• Efficient Belief Propagation for Early Vision. P. F. Felzenszwalb and D. Huttenlocher. IJCV Vol. 70, No. 1, October 200#
• Qingxiong Yang, Liang Wang, and Narendra Ahuja, A Constant-Space Belief Propagation Algorithm for Stereo Matching, IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2010.
  • The presentation about the algorithms was presented to Itseez group.
  • The guys from NVidia project started implementing the algorithm on GPU (from the first sight the BP-based algorithms are much more scaleable than Hirschmuller SGBM algorithm).
• By the request from James, I implemented VNG bayer→RGB demosaicing algorithm in OpenCV: https://code.ros.org/trac/opencv/ticket/46# The current C version takes ~16ms to de-bayer 640×480 image (the simple bilinear algorithm takes ~1.5ms on the same image).

Action Items

Gary

Vadim

Victor

Kurt

From last time

Gary

• Ask James about how well the 2D bar code is matching.
• Ask Joe about GPU priorities?
• (./) Ask Steve about purchasing a new server. Just do it.

Vadim

• Think about is there an easier way to temporarily enter documentation changes

Victor

• (./) Send short written description of the texture feature algorithm and which parts are fast or slow.
• (./) Send quote for additional server

Kurt

• Try VO on New Collage sequence.

Agenda

• Documentation
  • Catch up
  • User additions
• User contrib …
• 3D Model capture
• Feature based object recognition
• OpenCV on Android
• General object recognition plans

Minutes

Documentation

• Need user link so that we can add in documentation
• Rate good tutorials to emphasize?
• Set up a how to guides on wiki
• Samples code should have a documentation page which indexes what is there
  • Flann example should be updated to use features2d
• Can put in a help() function in each sample that is also its documentation
• Extra person on documentation would be good …

Bundle Adjustment

• 4 papers at ECCV mostly on speedps

Object recognition

• Working on dual contour fragments as descriptors Schmidt’s technique
• Can’t tell the sensitivity of MOPED from the literature
• Question is whether FLANN hurts performance or not
  • Much faster
  • Didn’t see much drop in recognition accuracy. Maybe 5% drop at worst
  • Need to compare brute force
    • Each training image against the scene
    • Then run RANSAC to see if we can find that object
      • Not so simple because of how to choose inlier outlier
      • But this is for the geometric check, but typically it is not to hard to find a cutoff
• In previous efforts, choosing the best cluster, was nearly 100% recognition
  • Look at wrong detections to see if filter out with geometric check
    • One would think the strong geometric check would eliminate false positives
• MOPED takes a large (50-90) number of nearest neighbors
  • Will partition among the objects
  • Then perform ratio test
  • Then a particular object can get at most one hit from each NN
  • They are handling “match starvation” in this way
  • In Lowe’s paper, the ratio test is used to keep down the false positive matches of features
    • Kurt turned up K in kNN. But when K went beyond 10, the algorithm broke. Probably because no ratio test was used
    • Current algorithm is sensitive to number of features and matches. When K goes from 1 to # We get get better matches, but more false positives because not using ratio test yet.
    • Make sure in ratio test, that you don’t allow spatially overlapping descriptor areas to filter out matches. This will destroy good matches for no reason (they physically overlap).
  • There is a ROS function that allows iterating through bag file images
• Green tea was giving 100% using largest cluster method (one object per image found)
  • But even with geo-filtering, we got a false positive. This shouldn’t happen. Matching to shadow?
• Clustering: How does it work?
  • Turn on showing the clustering, get lots of circles. What does this mean? Distance was defaulted to 100
  • Joined keypoints that give same object view
  • When kNN count went up, it slows down. Cluster size > 50, so it goes back to closer to brute force
    • When k low, cluster size is low and so less checking
  • Clustering is done hierarchically using greedy agglomeration.
    • Moped clusters, does a weak ransac inlier check, split accordingly, and then they do careful ransac
    • but single stage clustering should work when there is no occlusion
    • Moped was trying to solve the problem of several copies of the object in the same image
• Notion: Match starvation may be a problem.
  • Try Gary’s suggestion of no overlap might help
  • We use 10 viewpoints per object. Moped merges to a single object. I’s like we have 10x number of objects because we have this
    • MOPED uses fast, gets lots of features, but they use higher resolution images

3 camera stereo rectification (3 cameras inline)

• Vadim needs data, thinks he can make this work
• Calibrate the inrinsics parameters
• Extrinsics Make them orthogonal to baseline. Same as regular stereo
  • Get a new image plane for the 2 cameras
  • Rotate the third camera into this plane
  • Works if optical centers are on same line
    • OK if it’s off a little as long as the 3rd image plane is rotated to be parallel
      • Then you’ll get a scale offset that will need to be adjusted since the plane will be ahead or behind the other 2 planes

Monocular VO

• Testing on office sequence
  • Other visual slam algorithms filter points out of the point cloud if the points stop being consistent
  • Need to understand why so many points are getting in in the new college sequence

Android

• Took a brief look
• Will do some changes in OpenCV to fix some issues that will make building on Android easier
• This will enable simpler Android

3D model capture

• Trying to scale up BiGG
• Needed for scaling, don’t know where the resources may be
• Just a “head’s up”

Data collection

• Will go massive hopefully with pan-tilt table

Latent SVN

• Can load model and do a prediction using c version of his code
  • 5 second per image
  • original Felzenschalb was about 2 seconds
  • Marius converted Felzenschwalb’s cascade from Matlab to Octave
    • Doing per image.
    • Cascade algorithm could be parallelized
  • His technique is fairly general and doesn’t have to depend on HOG. We might try using BiGG

Server

• Can expense a new server dedicated to object recognition.

Plans

• Gary on vacation Aug 14-23
• ECCV Sept 5-11 (Gary)
• Graph Con Sept 20 (Victor)
• IROS Oct 18-22 (Gary)
• Maybe, end of Sept, Victor comes out
• Texture descriptor work right now

Anatoly

Accomplishments:

1. OpenCV GPU module
   • Updated OpenCV GPU API according to Joe’s comments.
   • Finished the API implementation. Now we can focus on algorithms and optimizations.
   • Added 3 tests cases for opencv_gpu module (GpuMat::convertTo, GpuMat::copyTo, async calls tests)
2. StereoBM_GPU:
   • Added low textureness based disparity filtering.
   • Fixed StereoBM kernel crash that was found during textureness based filtering integration in OpenCV.
3. Implemented first version of BP on GPU. It provides 20-25 times speed up on Fermi comparing with Felzenswalb’s CPU implementation. It can run only on 640×480 now due to high memory usage and shows # 4fps on the resolution. Main bottle neck is huge memory transfers. We have several ideas of optimizations.

Plans:

• Continue work with BP GPU implementation: performance optimizations, trying to minimize memory usage in order to run on high resolutions.
• Compare quality of the algorithm with others stereo matching algorithm implementations.
• Probably investigate and implement “A Constant-Space Belief Propagation Algorithm for Stereo Matching” (http://vision.ai.uiuc.edu/~qyang6/)

This is a variant of BP algorithm that works with sparse belief vector, therefore demands less memory.

Victor

OpenCV:

• Still several test fails show up for Ubuntu32 with gcc# 4 as well as for Windows mingw.
• Python compilations and tests problems persist for mingw too.
• Rosbuilder (compilation of all ros packages depending on opencv, cturtle distro is used for now as latest is broken) is finally up and compiling successfully as well as various flavours of Ubuntu and gcc. Compilation issues (trac #490, #495) have been resolved. [Alexander]

Object recognition:

• 6 new objects have been added to the benchmark dataset
  • (tilex, milk, cascade, numi, pledge, spam).
• Code refactoring of textured_object_detection package.
  • Statistics of experiments with object recognition for SIFT/SURF, ratio test and varying other parameters is attached.
  • So far ratio test was tested with thresholds 0.6 and 0.8 and we see a drop in detection rate accompanied by a drop in false positive rate. The full table is attached. [Alexander]

Note: there is a difference in accuracies reported in my previous report and in several subsequent Alexander’s letters. This is caused by a difference in accuracy definitions.

• In the former case we tried to classify a test image into several classes and the class that got the most inlier won.
• For the latter case we solved a detection problem of finding all object instances in a test image and so any cluster with a sufficiently high number of inliers was recognized as an object, which led to lower accuracy numbers.

Features_2d:

• calonder descriptor integrated, algorithmic and timing tests comparing ROS and OpenCV versions of calonder_descriptor have been implemented.
• Other changes coming from Patrick’s request (windowedMatchingMask, index(), matchImpl()). Maria

Other: Victor was on vacations July 20-26

Vadim

Progress:

• optimized VNG-based image demosaicing using SSE
  • The running time on 640×480 image decreased from ~16ms down to ~6ms – about 3x performance increase.
    color conversion
• rewrote all the color conversion functions in C++, added some new functionality:
  • new color conversion codes RGB2HSV_FULL, RGB2HLS_FULL (where h varies from 0 to 255, not from 0..180) and
  • the reverse ones RGB2Lab, RGB2Luv now treat the input RGB as the most popular sRGB space, and apply the necessary gamma correction the problem was reported long ago by G. Kloss).
  • The original conversion is still available via LRGB2Lab and LRGB2Luv codes. Despite the extra gamma correction,
  • the performance of RGB→Lab conversion has been greatly improved.
  • RGB2YUV conversion added.
• The internals of the new color conversion engine are now more flexible, so cvtColor can be extended in the future to handle various RGB profiles (like sRGB, Adobe RGB etc.)

Interpolation

• A useful cubic spline interpolation function has been added for very fast approximation of various functions (like gamma correction)
  • all the tests pass.

Bugs

• The related to color conversion tickets ## 328, 450 have been closed.

New inline stereo

• studied 2 papers on trinocular stereo rectification:
• “An Efficient Trinocular Rectification Method for Stereo Vision” by Y. K. Baik, J. Choi and K. M. Lee.http://cv.snu.ac.kr/newhome/publication/pdf/conf/ic050.pdf
• “Trinocular Rectification for Various Camera Setups” by M. Heinrichs and V. Rodehorst.
  http://srv-43-200.bv.tu-berlin.de/publications/pdf/P_0# pdf
• Neither of the algorithms will work on our configuration, since ours is “collinear stereo” where all the camera optical centers are on the same line (if they have the same lenses).
• On the other hand, it appears that the existing cvStereoRectify can be modified to handle such a case.
  • The current algorithm first makes the optical axises of both cameras parallel, and then rotates both cameras by the same angle to make the optical axises orthogonal to the baseline.
  • The first part can be extended to bring the third camera axis parallel to the first two.

The prototype implementation is ready. It would be very useful to have some data to test the algorithm before putting it to SVN.

Action Items

Gary

• Will collect data
  • Try to collect with pan-tilt table

Vadim

Victor

Kurt

• Get higher res data to Victor … this time for sure!
• Get Vadim 3 inline camera data

From last time

Gary

• Ask James about how well the 2D bar code is matching.
• (./) Ask Joe about GPU priorities?
• (./) Ask Steve about purchasing a new server

Vadim

• Think about is there an easier way to temporarily enter documentation changes

Victor

• (./) Send short written description of the texture feature algorithm and which parts are fast or slow.
• (./) Send quote for additional server

Kurt

• {?} Try VO on New Collage sequence.

Agenda

• Spherical and cylindrical projections
• Features2D
• Object recognition
• OpencV on Android

Minutes

• Spherical first pass is done
  • Cylindrical: y is not modified.
  • What is the final purpose from this function
    • Pose estimation
    • 3D reconstruction
    • Display
    • Feature based object detection and tracking
      • — it’s for display here
  • With 3 cameras, use middle one’s
• Working on general pictorial 2D barcodes Gary
  • Descriptor matcher is a good reference example for features 2D
• Projection stuff in its own pipeline?
  • Calib3D
  • Image stitching
    • Projection and Stitching
      • Use keywords
  • Question remains of where to put this
• Features2D depends on image processing
  • Dependency graph. Uses Features2D and image processing
• Android port, look at this week
  *GSOC, Vadim’s people
  • Qt bindings for HighGUI
    • Save currently displayed image
    • Zoom, scroll, measure pixel values
    • Flexible controls
    • is partially in HighGuI already
      • Trackbar, address of variable

Vacations

• Gary on vacation Aug 14-23
• Vadim out Aug 9-23
• Victor out to Aug 19
• ECCV Sept 5-11 (Gary)
• Graph Con Sept 20 (Victor)
• IROS Oct 18-22 (Gary)
• Victor on vacation
• Vadim Monday

Vadim

Here is the progress made last week:

• 8 more trac tickets have been closed: ##295, 296, 310, 321, 348, 396, 408, 501
• By request from WG (Kurt, Gary) implemented a draft version of spherical projection function (when the image from a calibrated camera is projected to a sphere rather than a plane). See below

Vadim: start of spherical projection

We have a point

(u,v)

From the camera matrix we compute

(x, y, 1) = ((u – cx)/fx, (v – cy)/fy, 1)

We find intersection of a ray

(X, Y, Z) = k*(x, y, 1)

X**2 + Y**2 + (Z – alpha)**2 = 1

we get

(kx)**2 + (ky)**2 + (k - alpha)**2 = 1.

from that we find

k

then we project the intersection point from a sphere back to plane using orthogonal projection:

http://www.uwgb.edu/dutchs/structge/sphproj.htm

stereographic projection, as well as cylindrical projection, is not difficult to implement either, I just wanted to make sure that this is what you need first.

i.e. the forward mapping

(u, v) → (kx, ky)

• 
  • Figure 1: Spherical projection with alpha = 0
• 
  • Figure 2: Spherical projection with alpha = 0.5
• 
  • Figure 3: Spherical projection with alpha = 1

Comments on This

From the reference you give, translating to image-based terms, I think:

• Gnomic projection → planar projection
• Equiangular projection → equirectangular panoramic image

At any rate the other projections are interesting. I kind of like the
orthographic projection, so let’s keep that to work with. But I think
the standard projection is the equiangular one (for panoramic imagery)

See these references:

• http://mathworld.wolfram.com/CylindricalProjection.html
• http://mathworld.wolfram.com/EquirectangularProjection.html

Typical panoramic viewers expect cylindrical or equirectangular images.

I especially recommend this small tutorial:

• http://www.cambridgeincolour.com/tutorials/image-projections.htm

Cylindrical projections look good for modest vertical FOV (e.g., 45
deg); otherwise the spherical projection is probably best.

• Note that Mercator is a compromise between cylindrical and spherical.
• The planar projection looks odd past 120 deg or so.
• Note also that they stitch together three photos with approximately the same FOV as we
  have with the 3-cam setup, and show a cylindrical projection which looks nice.

Ok, action item: generate equirectangular warp with variable viewpoint
(from the center of the sphere to close to the surface).

Action Items

Gary

• 2D Bar codes

Vadim

• Look over android-opencv
• generate equirectangular warp with variable viewpoint (from the center of the sphere to close to the surface).

Victor

Kurt

Get back to Vadim on spherical projection effort.

From last time

Gary

• Will collect data
  • Try to collect with pan-tilt table

Vadim

Victor

Kurt

• Get higher res data to Victor … this time for sure!
• (./) Get Vadim 3 inline camera data

Agenda

• Documentation
• Spherical stitching
• TOD — Textured object detection
• 3 camera stereo

Minutes

• Features2D:
  • OpenCV transition done
  • Ratio test in a separate matching method
  • FAST detector a bit different in OpenCV
    • - Maria to check, Helen to send info

• Object Detection:
  • FLANN seems to work differently, reversion of performance in TOD
    • - Migration of ROS FLANN to OpenCV? Check with Marius
  • TOD baseline: single-object matching [Alexander]

• Other detectors
  • PAS detector (Schmid et al.) and Oriented chamfer matching (Shotton)
  • Scaling linear in oriented chamfer matching
  • GSoC (Chatfield) – color categories

• 3-cam stereo
  • Working on algorithm Vadim
    • - Disparity used to get pixel transform
    • - Target date: end of this week

• Spherical calibration
  • Do some work looking at papers and designing an algorithm
  • Send papers to Vadim [Kurt]

• Issues:
  • Finishing off projects and app pipelines in OpenCV
    • - good work on Features2D, should be a template for other apps
    • - FLANN reversion in TOD
    • - documentation
  • Camera coordinate system in OpenCV / Compare to Hartley/Zisserman and OpenGL

Action Items

Gary

• Talk to Marius about migrating ROS FLANN to OpenCV FLANN

Vadim

• Review techniques for spherical calibration

Victor

• TOD baseline: single-object matching [Alexander]

Kurt

• Ping Pascal Fua about megapixel stereo
• Send spherical calibration papers to Vadim

From last time

Gary

Vadim

Victor

Kurt

• Get higher res data to Victor … this time for sure!
• (./) Get Vadim 3 inline camera data

Agenda

• Context:
  • Overview of recent focus: Distribution centers
    • Critical importance of object recognition
• Object recognition:
  • Victor’s arrival
  • recognition_pipeline
    • object_recognition
      • textured_object_detection (TOD)
      • Features2D status
    • object_recognition_experimental
      • BiGGPy
      • binary_pairs
    • Other features, detectors
      • Status of PAS (Schmid), Oriented chamfer (Shotton), Color Categories (Chatfield)
• Calibration
  • Spherical calibration progress report
  • Wide angle calibration
  • 3-cam stereo
• GSoC Aftermath
  • Status/plans to integrate
    • New HighGUI
    • Android port
    • Panoramic stitching
    • Detectors
    • Others?
• Documentation:
  • Need to make a plan to absorb and then document.
• Foundation and/or non-profit status for OpenCV

Minutes

• Described distribution job
• Object recognition
  • TOD
    • Dataset with images of different resolutions
      • Add more objects
      • Add objects from Cannon dataset
      • Compared resolution results. Recognition for HD is better
      • Take calibrated stereo pair, record scan of rotated object (written by Romain and supported by Radu, now this package is gone).
    • PAS, Oriented Chamfer, Colored Features (and wrappers to PASCAL VOC. Cascaded HoG not done … trying to finish)
    • Felzenshwalb is in the pipeline
• Calibration
  • 3 camera is ready, being debugged right now. 3rd camera scale is too magnified. Debugging ~80%
  • On wide angle, try (x,y) distortion model, don’t use polynomial since wide angle has too large of error there and polynomials don’t handle this well
    • Want fractional function to represent the distortion Poly/Poly might work better
    • Might need different calibration — big chessboard with lots of corners.
      • Might make sense to make a partial view chessboard detector
• GSoC aftermath
  • HighGUI QT is now in trunk :-)
  • VOC is important, to make it nice to allow people to run on PASCAL very easily
  • Color features (SIFT features over 3 channels and combine into higher dimension descriptor)
  • Android port, now done with cmake
  • Image stitching
• Documentation
  • Getting behind
  • Next openCV release in October. Need to find a code stop point.

• Victor Sept 22nd→NVidia GTC. Willow Sept 23rd. to Oct. 14th

Victor

Object detection: a problem with low accuracy under cturtle was fixed by adjusting the ratio threshold (the problem was caused by flann returnning wrong L1 distances in the earlier versions of ROS). All unnecessary dependencies, namespaces and headers were removed from textured_object_detection package. A tool for visualization of inliers for a pair of train and test samples has been implemented. solvePnPRansac has been parallelized using TBB. The speedup on 4 HT cores Intel Core i7-960 # 2GHz, 12Gb DDR3 is # 5×.

The current training base (640×480 images shot by Radu) has been enlarged up to 18 objects and merged with the canon dataset (9 objects, 1280×853, taken with a couple of 40D cameras) resulting in a dataset of 27 object in the training set and 22 in the test set (there were few test images in the canon dataset so we had to take new ones, and 5 out of 9 objects will be available only tomorrow). The recognition results for the new and old bases are summarized in the attached pdf file. [Alexander]

A proposal for adjusting features2d interface has been developed. A new interface incorporates all older functionality together with matching against several images, ratio test and cross-check test. The proposed interface will be sent out after team review on Wed. Maria

Mono vslam: A problem with lots of points appearing behind the camera along its trajectory has been solved — goodPts flag was not taken into account by voSt engine. The current problem is loss of tracking in a certain place of new_college. Changes of ransac PnP parameters did not solve the problem, as well as filtering out matches with large distances. Repeatability of FAST on a sample image from new_college sequence (measured using sample from OpenCV) is 0.98 — so the most of the points are tracked across a video. However precision-recall curve for SURF for the same image shows much worse results: for precision 0.1 recall is equal to 0.0# This means that if we choose 4% of matches with minimum distances, only 10% among these matches will be correct. Of course these are lower numbers than what we have in posest because of windowed match. But still the lack of correct matches is the main hypothesis for an observed loss of tracking. [Victor]

Figure # Current TOD Results, 640×480

Vadim

1. 3-camera rectification algorithm has been written and now is being debugged.
2. Papers on wide-angle cameras calibration have been reviewed. Currently it is planned to reuse the existing calibration engine to calibrate such cameras, but the lens distortion will be represented as a rational function of (x,y) instead of a polynomial.
3. Several functions from CLapack, embedded into OpenCV distribution, have been tweaked to get better SVD performance. As a result, large matrix decomposition is now ~20% faster, and the small (2×2 – 4×4) matrix decomposition is up to 100% faster.

The plans:

1. finish with 3-camera rectification
2. implement rational lens distortion model: undistortion + calibration functions
3. check Android port of OpenCV

Anatoly Baksheev GPU

1. OpenCV GPU module.
   • Implemented function for BGR2GRAY, RGB2GRAY, GRAY2BGR conversions.
   • Fixed bug in BP with user allocated disparity
   • Implemented async versions of drawColorDisp, reprojectPointsTo3D functions.
   • Implemented remap with bilinear interpolation for color images. It does not use texture references, because uchar3 type is not supported for the references.
2. Stereo On GPU demo.
   • Added point cloud visualization.
   • Added possibility to run BP, CSBP on color images (because now we can rectify color images too)
   • Found several crashes:
     • Connected with context managing.
     • Probably connected with host code that support ‘double’ and device code for Compute capability 1.1 that do not support it. OpenCV is compiled for all GPU architectures now.
     • A crash with unknown reason.

We spent several days trying to find reason, but still without success.

1. We looked into NPP. We think best way to integrate OpenCV GPU with NPP is to create OpenCV style wrapper functions in the module that will call NPP. The wrappers won’t support 3-channel images because NPP does not, but in future we may implement it. NPP binaries for all platforms are too heavy, so we are not going to include in OpenCV SVN . In order to compile OpenCV with NPP support user will have to install NPP separately and specify the installation directory in Cmake. If OpenCV is compiled without NPP, the wrappers will throw an exception. Complete list of functions to implement such wrappers will be created later.

Plans:

1. Finish updating DemoCVPR (fix the crashes)
2. Integration OpenCV with NPP.
3. Finish studying GraphCut based methods. Decide if we are implementing them.

Action Items

Gary

• Get new scanning from Radu
• Draft OpenCV release include list
• Send Victor foils
• Make sure Victor has a place to stay on visit.

Vadim

• Draft OpenCV release include list

Victor

• Draft OpenCV release include list

Kurt

From last time

Gary

• (./) Talk to Marius about migrating ROS FLANN to OpenCV FLANN
  • {*} ‘’On Marius’s “todo” list to migrate latest FLANN to OpenCV

Vadim

• Review techniques for spherical calibration

Victor

• TOD baseline: single-object matching [Alexander]

Kurt

• {o} Ping Pascal Fua about megapixel stereo
• {o} Send spherical calibration papers to Vadim

Agenda

• Gary and Kurt are in Crete, just reports today
• See Release # 2 plans

Minutes

Vadim

The last week was spent on debugging the 3-camera rectification algorithm and on implementing the extended
distortion model that handles wide-angle cameras. Both algorithms have been finished and committed to SVN:

1. the new function

float rectify3( const Mat& cameraMatrix1, const Mat& distCoeffs1,

const Mat& cameraMatrix2, const Mat& distCoeffs2,

const Mat& cameraMatrix3, const Mat& distCoeffs3,

const vector<vector<Point2f> >& imgpt1,

const vector<vector<Point2f> >& imgpt3,

Size imageSize, const Mat& R12, const Mat& T12, const Mat& R13, const Mat& T13,

Mat& R1, Mat& R2, Mat& R3, Mat& P1, Mat& P2, Mat& P3, Mat& Q,

double alpha, Size newImgSize,

Rect* roi1, Rect* roi2, int flags );

has been added. It takes 3 camera matrices and the corresponding distortion coefficients, as well as
the (R[otation], T[translation]) pairs from the camera 1 to camera 2 and from the camera 1 to the camera #
Here, camera 1 is the left-most camera, camera 2 is the right-most camera and the camera 3 is the camera in
the middle. The algorithm computes such rectification transformations for each of the 3 cameras, as well as
the new projection matrices.

The function returns the disparity ratio (0 < ratio < 1), i.e. the disparity value d, computed from the
camera 1 and camera 2, should be multiplied by the ratio to get the corresponding pixel in the camera 3 view:

img1(x,y) ~ img2(x-d,y) ~ img3(x-d*ratio,y).

optionally, the function takes the chessboard corners projections for the cameras 1 and 3 – imgpt1 and imgpt3,
and then adjusts R3 and P3 to minimize the reprojection error.

The sample image is attached.

1. the distortion model has been extended from
   

   
   
   xd = x*(1 + k1*r^2 + k2*r^4 + k3*r^6) + 
   
   yd = y*(1 + k1*r^2 + k2*r^4 + k3*r^6) + 
   
   

   
   to
   

   
   
   xd = x*(1 + k1*r^2 + k2*r^4 + k3*r^6)/(1 + k4*r^2 + k5*r^4 + k6*r^6) + 
   
   yd = y*(1 + k1*r^2 + k2*r^4 + k3*r^6)/(1 + k4*r^2 + k5*r^4 + k6*r^6) + 
   
   

   
   the functions calibrateCamera, stereoCalibrate, projectPoints, undistortPoints, initUndistortRectifyMap
   and others have been modified accordingly.
   The new model appears to handle wide-angle cameras much better than the previous model.

The sample images are attached. As you can see, the rational distortion model results in the straight lines in the
undistorted images, despite that the usual pinhole camera model is used, not the special fisheye lens model.
Besides, the average reprojection error dropped from 1.79 pixels to 0.47 pixels on the tested dataset.

Plans:
- try & integrate Android port of OpenCV
- continue to clean the bug tracker
- start revising the documentation

See

Victor

Object recognition: the algorithm for automatic creation of a training base from a bag file or a set of stereo pairs has been created. We follow the ideas from ¹ iteratively adding samples that are recognized poorly to the training base. The test base had also to be re-created in order to eliminate crossing of training and test sets. Experimental results are on the way. [Alex]

Features2d: we did several iterations on the interface for matching an image vs several images. The main issue is how to convert a global descriptor index (descriptors from different images have to be stored together in a single matrix for search with flann) to image and keypoint indices. The implementation is is progress together with usage samples, that will be reviewed by the team this week. Maria

Mono visual slam: several bugs have been fixed, a full circle of new_college is tracked now. The main issue is in accumulation of camera trajectory errors and instead of a cycle we get a spiral. The test system has been modified to allow more general camera trajectory and the new_college scene has been simulated as a flat ring of points and a camera circling above them. However the problem observed on real data is not reproduced on simulated (see the image below). [Victor]

Shape recognition: a clustering algorithm for creating object taxonomy from a confusion matrix has been created. We follow the idea of ², iteratively merging object classes that are mostly confused with each other. An example of a taxonomy is shown here. This will become essential as we scale up the number of objects. This is observed, for example, in ³ where they deal with 10K object categories. [Ilya]

Other: Alexander was on a sick leave for the most of the week.

Best Regards, Victor

¹ Shape matching and object recognition using shape contexts, Belongie, S. and Malik, J. and Puzicha, J., IEEE Transactions on Pattern Analysis and Machine Intelligence, 2002

² Learning and Using Taxonomies For Fast Visual Categorization, Gregory Griffin and Pietro Perona, CVPR 2008

³ What does classifying more than 10,000 image categories tell us?, Jia Deng, Alex Berg, Kai Li and Li Fei-Fei, ECCV 2010

CUDA Anatoly Baksheev

Accomplishments:

1. Most time we spent trying to find reasons or workaround for the crash described before (or stereo kernel launch failure with unknown error). We worked in following way: we got whole OpenCV and began excluding code from it step-by-step while crash was reproduced. If crash disappeared we reverted changes, and continued excluding another code. As result we have 2 little samples that demonstrates the crash. We submitted this bug on NVidia extranet (partners.nvidia.com).
2. OpenCV GPU module.
   • meanShiftFiltering was updated to return output in RGBA format. We started updating the module in order to support color in RGBA by all algorithms.
   • Done small refactoring in the module and corresponding tests.

Plans:

1. Integration OpenCV with NPP.
2. Finish updating Stereo Demo

Action Items

Gary

Vadim

Victor

Kurt

From last time

Gary

• Get new scanning from Radu
• (./) Draft OpenCV release include list
• Send Victor foils
• Make sure Victor has a place to stay on visit.

Vadim

• (./) Draft OpenCV release include list

Victor

• (./) Draft OpenCV release include list

Kurt

Agenda

• We missed 2 weeks due to vacations and conferences, see ’’Prior Reports:_below
• Finalize what is in the release
  • OpenCV # 2 ideas

Working idea’s list:

Key:(./) = Done; :\ = Ongoing, probably make it; :( = Behind; {X} = Cut from list/will not make it.

1. (./) Improved camera calibration (wide-angle lens support …)
   • This is now the default, flags can be used to fix the new parameters of the ratio model
2. :\ Reorganized documentation (that reflects the new OpenCV structure)
3. (./) Google test support, sample tests (so that users can add new tests together with the new functionality)
4. :\ Python support extended to the new C++ functionality & MLL
   • Need to look at pyopencv
   • This might be easiest
5. (./) Android support in the build system, (and optionally iphone)
6. :\ More bugfixes (reported in the bug tracker).
7. (./) Finish features2d interface for one to many image matching (in progress by Maria)
8. :\ Include interface to PASCAL Visual Object Challenge and bag of words matching done by Ken Chatfield
9. :\ Include latent SVM detector done by Nizhny U (in progress, this depends on U folks, they are aware of tight schedule, (./) agreed to ratify external API with us by Sep 20)
10. (./) Add algorithmic/timing tests and documentation on features2d
    • Part of this awaits ratification (cmatch 1 image against multiple images)
11. (./) Replace inclusions of old OpenCV headers by new ones (obtained after OpenCV restructuring) throughout the library to optimize building time.
    • Some tests still include old style headers … plan to fix by release
12. Release date is mid to end of October

Victor will be onsite Sept 23rd. to Oct. 14th

Minutes

• Android — part of opencv in Android.
  • Python is not built because it runs through Java
  • HighGui is only partial (camera support is from Android SDK)
• Features2D
  • Needed to change dmatch structure to allow for 1 to many or 1 to 1
  • Need to talk to Patrick to make sure he’s OK with this, until now it is in branch
    • dmatch now has “imageID” to allow many to 1
    • Purpose is when you have several different images for an object class (object class recognition)
  • Question is where is Features2D used? VO and object rec stack
• Maria is working on the PASCAL VOC interface (code from Ken Chatfield)
  • Also has bag of words for recognizing object categories from descriptors
• TOD + dense stereo (currently in ’’textured_object_detection’’)
  • I can scan them
  • Send bag files to Alexander — get rid of stereo optionally
    • Change pose to homography
  • It’s in object recognition stack
  • It is not in REIN
  • Put an object on a table, use dense stereo
    • Want to add use with selectObj3D
• latentSVM now ratified and on track for integration.
  • Sliding window based search

Vadim

1. With the help from Ethan Rublee added Android support to OpenCV. That includes build scripts for Crystax NDK (http://www.crystax.net/android/ndk-r# php), Swig-based Java wrappers for OpenCV and the sample code. The current build system does not use CMake, but all the makefiles reside in a separate subdirectory opencv/android, and thus do not affect the rest of OpenCV. Also, the NDK makefiles for all OpenCV modules, except for highgui, have been rewritten not to require manual editing when new files are added or deleted to OpenCV.
2. Fixed couple of bugs in the recently optimized lapack subset that caused test freezes.
3. Rewrote matrix expressions to avoid templates use, moved it from headers to the source file matop.cpp. While no measurements have been done yet, subjectively it decreased OpenCV build time quite a bit.
4. Added Google test framework googletest (http://code.google.com/p/googletest/) to OpenCV. Started implementation of a sample test based on this framework.

Plans:

• implement a few tests using googletest.
• start documentation reorganization
• start Python wrappers extension to include the C++ part of OpenCV.

Victor

• OpenCV:
  • buildbot maintenance. Lots of problems arose that were solved by rebooting the server (Mac Pro) as well as all virtual systems (usually they are not rebooted, they just go to suspended mode all the time). Modified scripts for buildbot (fixed some problems with incorrect svn update). Found and removed failed tests from OpenCV (issues #564, 565, 566, 567, 568, 569, 570, 572, 575, 576, 577, 578, 579). Most of these problems related with MinGW compiler under Win3# Increased size of RAM for virtual machine with Windows XP. [Alexander]
• _TOD: _
  • Discovered an issue with pose estimation related to incorrect poses of piecewise-planar objects with features found only on one plane. Then if the object is symmetric, solvePnP returns incorrect pose. A new version of the training base with manually selected objects was created to get more accurate point clouds [Alexander]
• Features2d:
  • implemented algorithmic tests on FeatureDetector and DescriptorExtractor. A sample using a new interface for matching a single image to a set of images was implemented. Fixed a problem with read/write keypoints Maria
• Shape recognition:
  • Experimented with COIL100 dataset ( http://bit.ly/9qREPH ). Chamfer detected 50% of objects, PAS – 28% and combined classifier – 61%. Low results can be explained partly by a lot of objects that are indistinguishable by shape (e.g. green Bulgarian pepper and red Bulgarian pepper). [Ilya]
• Other:
  • Victor gave an invited talk at Graphicon’2010 on new features of OpenCV. Also Victor, Kirill and Anatoly gave an invited talk about stereo on GPU. Victor is on the way to US now.

Last Week Reports: Victor

Object detection:

The training base built by an automatic algorithm results in higher accuracy than the manually created training base. In order to avoid train/test overlap, we used odd frames from a bag files for training set, and even — for the test set. The amount of images chosen for the training set for each object is shown in the table below.

Name   Views count in new base      Views count in old base
100tea      40          			10
all         63          			8
bp          21          			10
coke        29          			12
cs          14          			10
gdo         43          			7
gt          9            			9
gtwl        38           			9
jp          15          			8
naked       33          			10
pc          21          			10
rrtea       9           			13
ts          9           			4

Recognition results are in the table below. There are 200 images per object, OLD and NEW correspond to the old and new training bases, the first number in the column is the amount of images with correct detections, the second is the number of false positives).

		   OLD                        NEW
100tea 	        131 	2                  198    2
all 	        148 	3                  198    0
bp 	        151 	0                  200    4
coke 	        178 	1                  198    2
cs 	        198 	1                  199    1
gdo     	151 	3                  198    0
gt 	        193 	3                  198    0
gtwl 	        177 	2                  200    0
jp 	        188 	3                  200    1
naked 	        173 	1                  198    0
pc 	        187 	4                  198    1
rrtea 	        198 	3                  198    1
ts 	        183 	2                  199    0

The accuracy is so high because even views in a bag file are like odd views that were used to create a training set. However the detection rate on other bag files containing several objects at once, is also higher for the new base. Below is a table for a bag file with 826 views and three objects, showing the number of correct detections:

OLD                           NEW

Numbers of views with all object             520                           750

Numbers of views with gdo object             400                           654

Numbers of views with gt object              675                           741

False positives                                1                             1

The video demonstrating detection on this bag file is uploaded here:

• http://www.youtube.com/watch?v=Jq3w8_XGjXc , or:
• http://www.youtube.com/watch?v=R-GscZPp070 .

Colored set of points indicate a successful detection of an object. [Alexander]

Features2d:

Another iteration over the interface of one-to-many image matching. A sample for this interface has been added. Minor changes to DMatch structure are proposed. Maria

Shape matching:

Several improvements in the PAS matching algorithm fused together with chamfer matching has resulted in high detection rate for textureless stapler (11 detections out of 20 stapler images with only one false positives per 160 non-stapler images). Need more textureless objects to experiment with — we are looking at COIL100 dataset. Also, we discovered a CVPR 2010 paper on chamfer matching that claims to speed it up by a factor of 100. [Ilya]

vslam:

a lot of time spent on a merge of cturtle_trunk version into stacks/vslam version. There still are some subtle differences but they are negligible. [Victor]

Action Items

Gary

Vadim

• Add build instructions to Android so that Alexander can add to build bot
• Renew Visa

Victor

• Send email on … I forgot what you were trying to say to Kurt and I when skype was garbled

Kurt

From last time

Gary

• (./) Get new scanning from Radu
• (./) Draft OpenCV release include list
• (./) Send Victor foils
• (./) Make sure Victor has a place to stay on visit.

Vadim

• Draft OpenCV release include list

Victor

• Draft OpenCV release include list

Kurt

Agenda

• Finalize what is in the release
  • OpenCV # 2 ideas

Working idea’s list:

Key:(./) = Done; :\ = Ongoing, probably make it; :( = Behind; {X} = Cut from list/will not make it.

1. (./) Improved camera calibration (wide-angle lens support …)
   • This is now the default, flags can be used to fix the new parameters of the ratio model
2. :\ Reorganized documentation (that reflects the new OpenCV structure)
3. (./) Google test support, sample tests (so that users can add new tests together with the new functionality)
4. :\ ’’Active work_Python support extended to the new C++ functionality & MLL
   • Need to look at pyopencv
   • This might be easiest
5. (./) Android support in the build system, (and optionally iphone)
6. :\ More bugfixes (reported in the bug tracker).
7. (./) Finish features2d interface for one to many image matching (in progress by Maria)
8. :\ Include interface to PASCAL Visual Object Challenge and bag of words matching done by Ken Chatfield
9. :\ Include latent SVM detector done by Nizhny U (in progress, this depends on U folks, they are aware of tight schedule, (./) agreed to ratify external API with us by Sep 20)
10. (./) Add algorithmic/timing tests and documentation on features2d
    • Part of this awaits ratification (cmatch 1 image against multiple images)
11. (./) Replace inclusions of old OpenCV headers by new ones (obtained after OpenCV restructuring) throughout the library to optimize building time.
    • Some tests still include old style headers … plan to fix by release
12. (./) 3 inline camera stereo.
    • Needs discussion with Kurt about when the camera is too far out of left<=>right image planes. Then there is a problem with scale factors
13. Release date is mid to end of October

Victor is onsite to Oct. 14th

Minutes

• Vadim: Python bindings for C++ (keep James for C)
  • Difficulty is C++ automatically allocates the arrays, but Python must know their size
    • Implemented a conditional allocater function. If the pointer is null, traditional malloc a free is used
      • Else use custom use compatible with numpy
    • Gain the ability to use numpy automatically — will send a report later
    • In python, don’t have to worry about allocation of output arrays, it will be automatic
  • Probably spend another week on this to extend to the rest of C++
• Android: Build system changed to CMake
  • Ethan is heading out now to be a Willow Intern, so will be online next week, Oct 4th.
  • Only image load and save is implemented in HighGUI
• Pascal VOC
  • Bag of words in
  • Code needs re-write. Couple of days
• GSOC
  • In # 2
    • Android
    • Improved HighGUI, now using QT
  • Later
    • Object category recognition and detection, with application to automatically work with PASCAL VOC
      • Easy reading of PASCAL data
      • Color descriptors and bag of words
      • Calculation of ROC curves
        • Maria is working on refining this code for inclusion
    • Spherical image stitching (maybe in October release if Ethan gets in shape)
      • Image blending
    • {X} Cascaded HOG … but Intel is doing this, might get from them
    • Blob tracking (not with Features2D) ’’by_Dat Chu (Nicolas Saunier was mentor)
      • but need to simplify API, get rid of Boost dependency
    • Trajectory analysis (Nicholas)
      • Code is on bitbucker, have to ask Nicholas about the status
    • Region Covariance Descriptor Stephen McKeague under Mark Asbach
      • May go into samples, now in
        • http://bitbucket.org/mark.asbach/opencv-covariance-features
  • Victor relationship with NNU (Nihzniy Novgorod University)
    • Might do an object recognition with OpenCV course
    • Gets 3 students who would work on OpenCV related things
      • Solve PnP improvement, try new solvers by LePetit’s group
      • Object scanner: Blob based calibration pattern.
      • Help with above. Alternative algorithm for stereo correspondence, ray intersection, trinocular stereo

Victor

• Object detection:
  • Captured 10 objects for binpicking project and created a training set for them.
  • Recognition was tested on three objects (clean_concept, coffee_filter and can_opener), the system is capable of recognizing all of them.
  • Pose estimation was tested on 8 test bags (images of each individual bin with lots of objects of the same class, captured by Matei) and the results are promising.
    • Objects with more texture such as can_opener are recognized robustly (as far as we can tell from a single image — meaning the recognition is robust against algorithm parameters changing), others such as egg_poacher are not so robust.
  • The training base loading has been speeded up by precomputing descriptors and storing them in an xml file.
  • Fixed a problem with incorrect camera parameters. [Victor, Alexander]
• Shape recognition:
  • Implemented the first version of Fast Directional Chamfer Matching (FDCM) without algorithmic optimizations.
  • Evaluated FDCM on the COIL100 dataset. It detects 75% of objects and it outperforms our previous detectors
    • (PAS – 47%, Chamfer matching – 50%, PAS+Chamfer – 61%).
  • Implemented a simple framework to test FDCM on the ETHZ Shape Classes dataset or other similar datasets. It scans an image by a sliding window, performs non-maxima suppression of hypotheses, evaluates detections by Intersection-over-Union criterion, visualizes detections and plots results as false postitve per image vs. detection rate.
  • Experimented with FDCM on the ETHZ Shape Classes dataset. The current implementation has low recognition results on this dataset. [Ilya]
• features2d:
  • Integrated bag of words and color descriptors from Ken Chatfield’s code into features2d. Maria
• OpenCV:
  • Installed Mingw TDM for experiments. Plan to test OpenCV under it, maybe will change mingw on virtual machines to TDM. [Alexander]
• Other:
  • Victor gave two invited talks at Graphicon 2010 (OpenCV and stereo) and a talk (together with Radu and Joe Stam from nVidia) on stereo with GPU at nVidia GTC. Lots of

Vadim

The whole last week was spent in experiments with Python bindings. Below is a short ’’"technology preview"_of the new bindings:

• I use the existing wrappers created by James, and added the new functions that call C++ API, so the new functionality will be available side by side with the existing C API wrappers.
• The new bindings are only available when Numpy is installed. This limitation can be removed in the future.
  • Thus, no cv::Mat/cv::MatND etc. are mapped to Python. Instead, numpy arrays are directly processed by OpenCV.
• OpenCV multi-dimensional dense arrays ( cv::MatND, cv::Mat ) are binary-compatible with Numpy arrays, so input numpy arrays can be converted to cv::MatND/cv::Mat without copying data, just the OpenCV headers need to be constructed.
• Similarly, the output arrays, when they are of correct size and type, can be easily handled. However, when OpenCV needs to reallocate the output array using Mat[ND]::create , we need some sort of callback to reallocate array using Numpy API.
  • Such a callback has been added as the new Mat[ND]::allocator member. When it is not NULL, allocator→allocate() and allocator→deallocate() are invoked to allocate/deallocate the array data, otherwise the normal OpenCV functions are used.
• Because in C++ API we can have several overloaded variants of the same function, like cv::add:
  

  
  
  cv::add(Mat, Mat, Mat&);
  
  cv::add(Mat, Scalar, Mat&);
  
  cv::add(Mat, Mat, Mat&, Mat mask);
  
  cv::add(Mat, Scalar, Mat&, Mat mask);
  
  cv::add(MatND, MatND, MatND&, MatND mask);
  
  cv::add(MatND, Scalar, MatND&, MatND mask);
  
  

• I have to implement wrappers for such functions by hand, where the wrappers choose the correct OpenCV function based on the parameter types.
• By now I have implemented the basic functions from core: arithmetical, logical and statistical operations, linear algebra functions, dft, drawing operations etc.

Plans:

• implement wrappers for the most important functions from other opencv modules.
• extend wrapper generation script to support the new API and, probably, C++ classes as well.

Anatoly Baksheev

Accomplishments:

1. NPP integration – switched to Cuda # 2
2. Updated build system according to changes in NPP for Cuda# 2 and Cuda toolkit, Added search in $CUDA_NPP_ROOT.
3. Verified most bugs with NPP (now some functions work fine, ex. copyMakeBorder). Prepared samples and submitted 4 bugs of NPP. Several issues are being investigated.
4. Implemented via NPP:
   o graphcuts, , boxFilter, cvtColor (RGB <→ YCrCb, BGR5x5 <→ BGR, BGR5x5 <→ Gray), exp, log, magnitude.
   o Implemented overloads with cv::Scalar of functions add, subtract, multiply, divide, absdiff (new NPP features)
5. Test were updated and refactored.
6. Stereo DoubleBP – meanShift & BP based stereo matching algorithm.

Now we have working prototype. meanShift & BP are run on GPU. All other pars on CPU. Our implementation now takes 43rd place on Middlebury rating. In contrast with authors’ implementation that has 3rd place. (But difference in percentage of bad pixels is not more than 8%).

We are going to investigate results from it, may be will tweak parameters, maybe we missed something. The algorithm is very slow (2min/frame) now because of implementation. It could be made much faster on CPU, some part could be ported to GPU.

1. Stereo based on GraphCuts.

New NPP has GraphCut labeling function. So now we can very quickly implement stereo matcher that uses it. We started doing it and have finished approximately 50% of work.

Fortunately, we have not started its implementation before, because in this case Itseez and NPP teams would do the same work.

Plans:
· Continue NPP integration (left: canny, histograms, some filters and some issues)
· Work with DoubleBP and StereoGraphCuts.
· Update Stereo Demo, if our ‘megabug’ is fixed, or prepare another repro cases.

Other:
We was on GraphiCon conference 21 – 23 sep 2010.

Action Items

Gary

• Ask Nicholas about the status of Trajectory Management

Vadim

• Write up python report and send to James and myself

Victor

Kurt

From last time

Gary

Vadim

• Add build instructions to Android so that Alexander can add to build bot
• Renew Visa

Victor

• Send email on … I forgot what you were trying to say to Kurt and I when skype was garbled

Kurt

Agenda

• Track the release
  • OpenCV # 2 ideas
  • On schedule?
  • Code freeze when?
  • General status
• If Victor calls in, we can talk some about foundation status for OpenCV
• Post release, should discuss migrating the ROS Recognition Infrastructure to OpenCV. Want an easy way to use different classifiers etc.
• Status of QT/HighGUI
  • Docs
• Doc status
  • User annotate
  • List of what needs docs
  • More and more useful example code

Working idea’s list:

Key:(./) = Done; :\ = Ongoing, probably make it; :( = Behind; {X} = Cut from list/will not make it.

1. (./) Improved camera calibration (wide-angle lens support …)
   • This is now the default, flags can be used to fix the new parameters of the ratio model
2. :\ Reorganized documentation (that reflects the new OpenCV structure)
3. (./) Google test support, sample tests (so that users can add new tests together with the new functionality)
4. :\ :\ ’’Active work_Python support extended to the new C++ functionality & MLL
   • Need to look at pyopencv No, Vadim is writing a universal Python
5. (./) Android support in the build system, (and optionally iphone)
6. :\ More bugfixes (reported in the bug tracker).
7. :\ Finish features2d interface for one to many image matching (in progress by Maria)
8. :\ Include interface to PASCAL Visual Object Challenge (VOC) and bag of words matching done by Ken Chatfield
9. :\ Include latent SVM detector done by Nizhny U (in progress, this depends on U folks, they are aware of tight schedule, (./) agreed to ratify external API with us by Sep 20)
10. (./) Add algorithmic/timing tests and documentation on features2d
    • Part of this awaits ratification (cmatch 1 image against multiple images)
11. (./) Replace inclusions of old OpenCV headers by new ones (obtained after OpenCV restructuring) throughout the library to optimize building time.
    • Some tests still include old style headers … plan to fix by release
12. (./) 3 inline camera stereo.
    • Needs discussion with Kurt about when the camera is too far out of left<=>right image planes. Then there is a problem with scale factors

• Release date is mid to end of October

Victor is onsite to Oct. 14th

Minutes

• Vadim working on modifying OpenCV headers with extra markers so it will know which parameters are output and input to automatically parse python
  • This is one step further from James to generate headers automatically. His parser created a description from C headers but need to manually edit a file
    • New technique automates this
• Documentation
  • Doxygen has (had?) problems with overloaded function.
  • Doxygen is actively developed, maybe they are slow on bug and features, but they are releasing code
  • Maybe we should have links hard coded in to user bugs/notes page
  • Need to add example documentation in samples
    • Use frames(?)
      *
• PASCAL VOC
  • Bag of words and color descriptor integrated by Maria
  • In progress, hopefully next week to integrate the PASCAL VOC
• Latent SVM
  • Some commits from NNU, have to see what their status is
• GPU module
  • Issue: Changed the way we call the code through CUDA.
    • Found a bug in CUDA
      • Memory gets overwritten, but shows only when linked to OpenCV. boiled it down to a simple example and it’s been accepted by the CUDA team, but CUDA won’t release a fix until Nov. Hard to work around
  • This runs outside of OpenCV
• HighGUI
  • If built with QT, will be extra functions for buttons, checkboxes and other controls
  • Script attempts to find QT, if it succeeds it will use, if not. But had problems on some platforms to for now
    • User needs to explicitly enable QT in CMake “use_QT” or “with_QT”.
      • If this behaves well, we will turn on by default
• Want to make contacts with Google streetview and Android teams.
  • Streetview: Want stitching to work with their dataformats
  • Android: Want OpenCV to work well on ARM (Atom is probably OK), good infrastructure for VR, tracking, calibration …
• Schedule:
  • 1 week more of Python
  • 1 week of doc/bug fixes
  • 1 week testing
    • Maria free after a week
    • Alexander needs to be on TOD
    • Victor is onsite until Oct 14
    • Ilya might be able to help with html

Vadim

The short summary progress:

1. fixed couple of bugs in cv::rectify3() and 3calibration.cpp sample. Thanks to Kurt for the test data!
2. Continued working on the extended Python bindings. Wrote parser (~90% complete) for the OpenCV headers that produces the API description files, very similar to manually created by James “api” file. It is planned to modify cv.py wrapper generation script to handle the automatically generated description files for C++ API.

Victor

• Object recognition:
  • Support for prosilica camera has been implemented in TOD.
  • Several objects were scanned with prosilica camera that was calibrated with regard to narrow stereo pair using a scanned checkerboard.
  • The algorithm calculates descriptors form prosilica image and finds 3D coordinates of the keypoints by projecting the narrow stereo point cloud into prosilica image with the calibration transformation.
  • The algorithm was tested on several objects (coffee filter, can opener) and it finds correct poses. Since algorithm parameters have a strong dependence on resolution (5MP vs VGA), they were put into a separate file. [Victor, Alexander]
• Features2d:
  • Problems with running detector/descriptor testbench (reported by Patrick) were fixed, computation of region intersection area was parallelized with TBB.
  • One-to-one keypoint matching with region overlap computation is 6x faster now on a large keypoint dataset.
  • Integration of Ken’s VOC sample is in progress. Maria
• Shape recognition:
  • Support for FDCM has been added into TOD. An additional score from chamfer matching helps to find correct poses of objects.
  • FDCM has been tested against the code provided by the authors and gives similar results.
    • However high recognition scores on Zurich shape dataset reported in the paper are not reproduced by both versions of the code.
    • A dependency on chamfer_matching (ROS package) breaks TOD (probably an older version of FLANN gets used) so this dependency has been temporarily removed. [Ilya]

Anatoly

Accomplishments:

1)      NPP integration
·         Implemented via NPP convertTo (depth convert), sumWindowColumn, sumWindowRow, Sobel, GaussianBlur, Canny.
·         Verified and updated some filters code according to NVidia comments for submitted bugs. One is not a bug, just docs misinterpretation. All work correctly.
·         We are still working on some OpenCV functions, that can be implemented via NPP filtering primitives.
2)      GPU module
·         Added new cvtColor functionality (RGB <→ YUV, RGB <→ XYZ)
3)      Double BP
·         We re-implemented some parts of DBP prototype and tweaked parameters, after we got 9th  place in Middlebury rating for Tsukuba (3rd place by authors).
·         DoubleBP was tested on several Middlebury’s datasets (Tsukuba, Venus, and Aloe) for small resolutions. Some screenshots and info could be found here https://docs.googlecom/present/edit?id=0AZqfgw3H-ZOjZGRocHNwcDlfMjIyY3FteHp2Z2o&hl=en&authkey=CI3p_rkN). We used small resolutions because it very slow (example, up to # 5hours per frame for 950×800 on CPU + GPU).
·         We realized that current most consuming part (99.9% of time, CPU) can hardly be ported to GPU effectively. But we can deviate from the paper and implement it this with less precision but faster and on GPU. May be we will do it.
4)      Stereo Graph Cuts
I harried to claim that NPP could be used for this task. I started implementing the algorithm. When I reached graph generation step, I realized that in contrast with other GC based algorithms, this graph is not a regular pixel grid graph, so NPP function is useless.
Because I had done some parts, I started leaning min GC algorithms and theirs GPU implementations. Maybe I will be able to implement GC for stereo on GPU (like in NPP but with more complex data structure). But now I am not sure that it its possible.
Plans:

·         Continue NPP integration (left: histograms, some functionality based on NPP filters)
·         Work with DoubleBP and StereoGraphCuts.
·         Update Stereo Demo, if our ‘megabug’ is fixed, or prepare another repro cases.

Action Items

Gary

• Start adding documentation to examples
• Make contacts with Google streetview and Android teams.
  • Streetview: Want stitching to work with their dataformats
  • Android: Want OpenCV to work well, good infrastructure for VR, tracking, calibration …

Vadim

• Check on QT status in OpenCV
• Find way to add user notes to OpenCV docs. Create pages off of our wiki at http://opencv.willowgarage.com/wiki/Welcome/Support
• Visa

Victor

• Need 1 to many image matching in Features2D. Talk to Patrick about what this might break.
• Find status of interface to PASCAL VOC
• Find status of LatentSVM

Kurt

From last time

Gary

• (./) Ask Nicholas about the status of Trajectory Management

Vadim

• Write up python report and send to James and myself

Victor

Kurt

Agenda

• Track the release
  • OpenCV # 2 ideas
  • On schedule?
  • Code freeze when?
  • General status
• Doc status
  • User annotate
  • List of what needs docs
  • More and more useful example code
• TOD

Working idea’s list:

Key:(./) = Done; :\ = Ongoing, probably make it; :( = Behind; {X} = Cut from list/will not make it.

1. (./) Improved camera calibration (wide-angle lens support …)
   • This is now the default, flags can be used to fix the new parameters of the ratio model
2. :\ Reorganized documentation (that reflects the new OpenCV structure)
3. (./) Google test support, sample tests (so that users can add new tests together with the new functionality)
   • Google test is integrated and works, but tests are not written in it yet
4. :\ :\ ’’Active work_Python support extended to the new C++ functionality & MLL
   • Need to look at pyopencv No, Vadim is writing a universal Python
5. (./) Android support in the build system, (and optionally iphone)
6. :\ More bugfixes (reported in the bug tracker).
7. :\ Finish features2d interface for one to many image matching (in progress by Maria)
8. (./) Include interface to PASCAL Visual Object Challenge (VOC) and bag of words matching done by Ken Chatfield
9. :\ Include latent SVM detector done by Nizhny U (in progress, this depends on U folks, they are aware of tight schedule, (./) agreed to ratify external API with us by Sep 20)
   • We need external API fixed and a sample written
10. (./) Add algorithmic/timing tests and documentation on features2d
    • Part of this awaits ratification (cmatch 1 image against multiple images)
11. (./) Replace inclusions of old OpenCV headers by new ones (obtained after OpenCV restructuring) throughout the library to optimize building time.
    • Some tests still include old style headers … plan to fix by release
12. (./) 3 inline camera stereo.
    • Needs discussion with Kurt about when the camera is too far out of left<=>right image planes. Then there is a problem with scale factors
13. (./) Updated FLANN is incorporated into OpenCV.

• Release date is end of October

• Victor is onsite to Oct. 14th
• Gary in Taiwan next week

Minutes

• Python status
  • All of opencv can be now be parsed, similar to what James did
  • Fully automated what James did, which takes a manual api description files
    • Vadim wrote a tool that automatically produces api description files
    • Could be passed to James’s tool, but since we now have classes etc, Vadim is modifying the script to handle this more complex output
      • Needs a few more days
• OpenCV now has ~2000 functions (probably 1000 distinct functions)
• Test coverage has not been tracked for awhile, tests continue to be added but we need to regenerate the statistics
  • Now we have broken up cv into “stacks”, so the comparison isn’t quite the same, cxcore will be comparable
  • The problem right now is with build bot, but Alexander is on TOD. He will get back to this to get it working by the release.
• Android
  • Ethan may write the installation for Android
    • Trying to improve the process
  • We need Android # 2 (Nexus One)
• Dft
  • mulSpectrum computes element wise computation of 2 complex arrays
  • If you want power, phase, or both, use mag, phase, CartToPolar respectively
    • polarToCart and merge and then idft
• For documentation, write a script to do a difference before and after on the html output to check whether things are screwed up
  • We might think, post release, maybe we should move completely to sphinx since the tool chain is complex and sphinx can produce pdf and html as required

Marius Muja (FLANN – Fast Approximate Nearest Neighbor)

• Upgraded FLANN in OpenCV to version 1.5 (‘’revision 3737’’).
  • Since the new FLANN is mostly templated, the header files have to be accessible for inclusion to the client code, so
  • FLANN was moved from ‘’3rdparty/flann_to ’’modules/flann’’.
    • The main index class is now

      cv::flann::Index_

      and there is a typedef from cv::flann::Index_ to cv::flann::Index so the existing code that uses the flann from OpenCV works the same.
• The only change that needs to be done to the existing client code is adding

  • #include <opencv2/flann/flann.hpp>

    in case flann is not indirectly included through some other header files (such as the old form cv.h) and linking against the libopencv_flann.so library.

Vadim

The summary progress:

1. Finished and debugged OpenCV header parser. The parser is able to parse all the OpenCV headers, extract all the class and functions, and produce the output very similar to the manually created “api” file in the current Python wrapper. The sample output from the whole OpenCV, as well as the reference “api” from the current wrapper is attached.
   • Now the wrapper generator is updated to handle the automatically extracted API data.
2. In order to simplify Python wrappers code, MatND and Mat are merged into a single class Mat. Now MatND is defined as:
   • typedef Mat MatND;
   • The new n-d capabilities of the Mat class brought very small extra overhead space-wise (the new header is just 16 bytes larger than the previous one) and performance-wise (a single extra comparison “m.dims > 2” is added to the basic array processing functions). Also, compatibility with the old code is preserved. All the tests pass after the modification.
3. Fixed OpenCV build on MacOSX after LatentSVN integration.

Victor

• TOD:
  • Implemented frame pose registration by a chessboard that is scanned together with each object.
    • All 3d points are projected into a single reference frame and inliers from different frames can now exist in the same hypothesis.
    • The recognition results with prosilica are only slightly better — the most of the inliers still come from a single image.
  • Coarse to rough pose search strategy was implemented:
    • first we generated poses from each cluster and look for inilers with large reprojection error (50),
    • then each pose is refined on a new set of inliers and the quality of the pose is estimated with a number of inliers that have very low reprojection error (~8).
      • Poses that result in overlapping projections of objects into a test image are filtered out — we leave the one with the largest number of inliers.
  • The algorithm was tested on several items from binpicking collection (coffee_filter, can_opener, egg_poacher) and several items from household collection (tide, mop, tilex).
    • The results show that the recognition works stable most of the time, while pose estimation is unstable for objects with small textured surface (perspective is too weak to estimate both distance and orientation).
  • Alexander is experimenting with 3D to 3D matching for pose estimation.
  • A bug in solvePnP resulting in poses behind the camera was re-discovered and submitted to Vadim. [Victor, Alexander]
• Object category recognition:
  • Finished to integrate BOW image classification sample. The main sample function is refactored,
  • made the sample cross-platform, vocabulary trained.
  • SVMs and plotted recall-precision curves for Ken’s sample and integrated OpenCV sample are only slightly different because of several sources of randomness in the code. Maria
• features2d:
  • Merged features2d support for one to many image matching (implemented earlier in “features2d” brunch) with trunk revision.
  • Started to test these modifications using some existing OpenCV tests. The changes are not committed yet. Maria

Anatoly Baksheev (GPU)

Accomplishments:
1)      NPP integration

·         Created filters engine – set of classes that help to create convolution kernels for different filtering operations and to execute the operations using NPP in most optimal way.

·         Implemented Laplacian, Scharr, GaussianBlur, sepFilter2D. All other filtering functions were switched to the engine. Tests were updated.

·         Implemented histogram wrappers: histRange, histEven, evenLevel. Now we are checking what other OpenCV functionality could be implemented via the histogram  wrappers.

·         Reported NPP bug (742990 ) “nppiCanny returns NPP_TEX_BIND_ERROR”. But NVidia had known about the bug before.


2)      MeanShift segmentation – integrated into OpenCV. (it was a separate part of DoubleBP before). It uses MS filtering on GPU and, after, does 5D point classification on CPU. Next it generates image with labeled segments. MeanShiftFiltering was updated to return range coordinates that are used by segmentation. Added ability to eliminate segments with size lower than given threshold . CPU part is ~25% of time and not portable to GPU. We plan to speed up several loops with SEE/TBB. Test is in process.



3)      Created ROS package with StereoBM GPU code, interface and initialization infrastructure were re-implemented without OpenCV GPU. So it is stable. (NVidia said that fix of our ‘megabag’ will be available in November’s release). Victor are going to demonstrate it on a robot with GPU. The only disappointing fact is that the GPU has only 64 cores (I guess it is not a Fermi), and performance won’t be higher than on CPU.


4)      Double BP. Implemented fast correlation volume calculation on GPU, now we are debugging it (returns image with artifacts). The implementation uses modified integral images based approach that we created during CVPR preparation.
It is only one part of DoubleBP algorithm. Its performance is less than 1 sec for tsukuba_384x228  (~160 sec before). We need a couple of days to fix bugs and estimate quality + 1 day if we decide to integrate it in OpenCV.



Plans:
·         NPP integration: check is we missed something.

·         Work with DoubleBP.

·         Switch to implementation of SURF or HOG algorithms.

Action Items

Gary

• Get Victor a Nexus One
• Ask Ethan about Android build scripts
• Contract with Steve

Vadim

Victor

Kurt

From last time

Gary

• Start adding documentation to examples
• Make contacts with Google streetview and Android teams.
  • Streetview: Want stitching to work with their dataformats
  • Android: Want OpenCV to work well, good infrastructure for VR, tracking, calibration …

Vadim

• Check on QT status in OpenCV
• Find way to add user notes to OpenCV docs. Create pages off of our wiki at http://opencv.willowgarage.com/wiki/Welcome/Support
• Visa

Victor

• Need 1 to many image matching in Features2D. Talk to Patrick about what this might break.
• Find status of interface to PASCAL VOC
• Find status of LatentSVM

Kurt

Agenda

• Track the release
  • OpenCV # 2 ideas
  • On schedule?
  • General status
• Doc status
  • User annotate
  • List of what needs docs
  • More and more useful example code
• TOD

Working idea’s list:

Key:(./) = Done; :\ = Ongoing, probably make it; :( = Behind; {X} = Cut from list/will not make it.

1. (./) Improved camera calibration (wide-angle lens support …)
   • This is now the default, flags can be used to fix the new parameters of the ratio model
2. :\ Reorganized documentation (that reflects the new OpenCV structure)
3. (./) Google test support, sample tests (so that users can add new tests together with the new functionality)
   • Google test is integrated and works, but tests are not written in it yet
4. :\ :\ ’’Active work_Python support extended to the new C++ functionality & MLL
   • Need to look at pyopencv No, Vadim is writing a universal Python
5. (./) Android support in the build system, (and optionally iphone)
6. :\ More bugfixes (reported in the bug tracker).
7. :\ Finish features2d interface for one to many image matching (in progress by Maria)
   #(./) Include interface to PASCAL Visual Object Challenge (VOC) and bag of words matching done by Ken Chatfield
8. :\ Include latent SVM detector done by Nizhny U (in progress, this depends on U folks, they are aware of tight schedule, (./) agreed to ratify external API with us by Sep 20)
   • We need external API fixed and a sample written
9. (./) Add algorithmic/timing tests and documentation on features2d
   • Part of this awaits ratification (cmatch 1 image against multiple images)
10. (./) Replace inclusions of old OpenCV headers by new ones (obtained after OpenCV restructuring) throughout the library to optimize building time.
    • Some tests still include old style headers … plan to fix by release
11. (./) 3 inline camera stereo.
    • Needs discussion with Kurt about when the camera is too far out of left<=>right image planes. Then there is a problem with scale factors
12. (./) Updated FLANN is incorporated into OpenCV.

• Release date is end of October

• Victor is onsite to Oct. 14th
• Gary in Taiwan next week

Minutes

Vadim

1. Python wrapper generator is semi-completed. The code for standalone functions is generated (the debugging is in progress). The classes, ML classifiers in particular, are not wrapped yet.
2. Ilya started the work on OpenCV documentation reorganization (the new documentation structure should reflect the new modular code structure).
3. 2 build problems have been fixed.
4. Many overload variants of OpenCV functions has been removed after special ArrayArg introduction. ArrayArg is type-safe C++ analogue of CvArr. It can be used to pass Mat&, std::vector<>, CvMat*, Vec<> or Mat<> to the function, as input or output parameter. It has been done to simplify Python wrapper generation, but for C++ users it provides more convenience as well.

Victor

TOD:

• A method for calculating object pose from stereo has been implemented and is being tested now.
  • The results are comparable to solvePnP because the current algorithm allows only clusters that cover a large part of the object.
  • We will try tuning various parameters to enable higher detection rate while still having robust pose from stereo.
• A set of ~25 objects has been scanned. [Victor, Alexander]

stereo_gpu:

• Several problems with using stereo on gpu function in ROS have been fixed.
  • A package stereo_gpu has been created to try the method on PR2 (bugs in the current version of CUDA do not allow opencv gpu calls).
• The module was provided to Kevin Watts for benchmarking. [Alexander]

features2d:

• Worked on integration of features2d modifications.
• Finished the test features2d modification using existing OpenCV tests.
• Implemented algorithmic tests for descriptor matchers (BruteForce and FlannBased).
  • They cover all match methods from new interface.
• Did some fixes of FlannBasedMatcher.
• Implemented BruteForceMather L2-specializations using Eigen library for all match methods (not only one-to-one as before).
• match() and knnMatch() are faster up to 4 times, radiusMatch() – up to 2 times (because we do not need to look for the nearest).
• Prepared the opencv branch “f2d_ros_chack” to switch ROS on this and check backward compatibility. Maria

Shape recognition:

• Considerable improvement in the implementation of FDCM. Key steps:
• Improved orientation estimation and removed points with hard-to-estimate orientation,
• Implemented manual templates scaling instead of linear interpolation:
  • scaled points from contours, approximated them using ApproxPolyDP and then interpolated orientation,
• Used findContours from the chamfer_matching package instead of this function from OpenCV,
• Used precise Distance Transform,
• Implemented processing of closed contours. [Ilya]

Anatoly Baksheev (CUDA Implementation

Accomplishments:

1)      OpenCV GPU module

·         implemented gpu version of magnitude, magnitudeSqr, phase, cartToPolar, polarToCart (that is needed for HOGs)

·         Added new cvtColor functionality (RGB <-> HSV, RGB <-> HLS)

·         implemented via NPP magnitudeSqr and rectStdDev. We've covered all NPP functionality useful for OpenCV.

·         Sobel filter for signed output types is in progress (needed for HOGs). It can’t be implemented using NPP.

2)      HOG. We switched to HOG development. To the moment we have read 3 papers about HOG, played with people detection sample (OpenCV CPU HOGs + Linear SVM), started learning its sources. So now we are preparing to port them to GPU.



3)      SURF, SIFT and Fast. We decided to work on SURF in parallel with HOGs.

·         Learned paper about FAST point detector algorithm. Our conclusion that is could be ported to GPU with good speed-up. Implementation will take ~1 day. But fast FAST is not useful without fast SURF.

·         Now we are learning papers about SUFR and investigating different SUFR CPU and GPU implementations. The sources have GPL license so we have to create our own implementation.



4)      DoubleBP. Fixed bugs and tested DoubleBP with BM-style correlation volume (fast replacement of approach from the paper). With it the algorithm has gone to the bottom of Middlebury’s rating. So the very slow part which we replaced contributes significantly in DoubleBP’s quality. Also even now total work time of  the algorithm is about 5-10 times slower than BP with approximately the same quality. Therefore we froze its further development.

Plans:

·         Work on HOGS, SURF, SIFT, FAST GPU implementations.

·         Finish Sobel for signed output types.

Action Items

Gary

Vadim

Victor

Kurt

From last time

Gary

• (./) Get Victor a Nexus One
• (./) Ask Ethan about Android build scripts
• (./) Contract with Steve

Vadim

Victor

Kurt

Agenda

• Track the release
  • OpenCV # 2 ideas
  • On schedule?
  • General status
• Doc status
  • User annotate
  • List of what needs docs
  • More and more useful example code
• TOD
  • Plans
  • Status?

Working idea’s list:

Key:(./) = Done; :\ = Ongoing, probably make it; :( = Behind; {X} = Cut from list/will not make it.

1. (./) Improved camera calibration (wide-angle lens support …)
   • This is now the default, flags can be used to fix the new parameters of the ratio model
2. (./) Reorganized documentation (that reflects the new OpenCV structure)
3. (./) Google test support, sample tests (so that users can add new tests together with the new functionality)
   • Google test is integrated and works, but tests are not written in it yet
4. :\ ’’Active work_Python support extended to the new C++ functionality & MLL
   • Need to look at pyopencv No, Vadim is writing a universal Python
5. (./) Android support in the build system, (and optionally iphone)
6. :\ More bugfixes (reported in the bug tracker).
7. :\ Finish features2d interface for one to many image matching (in progress by Maria)
8. (./) Include interface to PASCAL Visual Object Challenge (VOC) and bag of words matching done by Ken Chatfield
9. :\ Include latent SVM detector done by Nizhny U (in progress, this depends on U folks, they are aware of tight schedule, (./) agreed to ratify external API with us by Sep 20)
   • We need external API fixed and a sample written
10. (./) Add algorithmic/timing tests and documentation on features2d
    • Part of this awaits ratification (cmatch 1 image against multiple images)
11. (./) Replace inclusions of old OpenCV headers by new ones (obtained after OpenCV restructuring) throughout the library to optimize building time.
    • Some tests still include old style headers … plan to fix by release
12. (./) 3 inline camera stereo.
    • Needs discussion with Kurt about when the camera is too far out of left<=>right image planes. Then there is a problem with scale factors
13. (./) Updated FLANN is incorporated into OpenCV.

• Release date is end of October
  • Release is delayed 2 weeks due to delay in completing python interface (for missing C functions but mainly for ML functions)

Minutes

• Documentation has been re-org’d to reflect new structure
  • It now has new feedback link
    • Users will have to register with willow wiki
      • Then start the wiki page and click on the template to start it
      • How to get an automatic empty page
  • Need to get up
• State of Hudson build
  • Hudson does not test trunk of opencv
• Build bot is improved http://argus-cv.dnsalias.org:8010/waterfall
  • ROS fails
• Kurt
  • ESM (Efficient Second order Minimization ESM, see second paper “Real-time image-based …” http://www-sop.inria.fr/icare/personnel/sbenhima/index.html
    • Matlab and binary version on the web
    • Lab automation, vslam, sub-pixel accurate point finding
    • Put Ilya on this
• Release
  • Python wrappers
    • Getting ML learning methods to work is a problem right now. It has never worked and want to add
      • Can this be put off to the next release?
    • As soon as this is done, we can release
  • Need to clear out bugs
  • Fixed build bot problems