Arxiv 2017

[1705.05301] Back to RGB: 3D tracking of hands and hand-object interactions based on short-baseline stereo [PDF] [notes]

Paschalis Panteleris, Antonis Argyros

read 22/05/2017

Objective

Real time hand tracking as a optimization problem that maximizes color consistency between calibrated stereo RGB cameras

Generative approach with asumption of 3D model of the hand(s) and objects

General idea : Correct model ==> photo-consistent views

Synthesis

Optimizes a model hypotheses using Particle Swarm Optimization (PSO) to maximize color consistensy

Pipeline

Preprocessing

• camera calibration (centers, relative position, focal lengths, orientation, distorsion...)
• images undistorted

Distinctiveness maps

• points which are more distinctive are more important (details about used function given page 4)
• points less distinctive then given threshold not considered at all (threshold)
• render distinctiveness maps for the two images
• using corner detection inspired from Harris but simpler compute
• color consistency between two points given computed as min of distinctiveness * $\exp(-\beta*||color difference||)$
• total color consistency computed as a sum over the image

Iterations

• corresponding points are determined using the current 3d hand model
• objective function is computed
• PSO is used to produce new hand model position proposistions

Results

metric : tracking error (mean distance of the estimated joints to ground truth) of joints for hands, on three anchor poins for object

Comparable to RGBD methods. (Compared to Efficient model-based 3d tracking of hand articulations using kinect. BMVC 2011)

Dataset

Rendered realistic dataset with libhand provided hand texture

Notes

• uses libhand model (22 bones) for 3d model adapted to have 26 DoF
• top of page 2 a good classification of previous work classified in Generative, discriminative, hybrid RGBD-multi RGB Stereo RGB mono RGB
• page 5 good definition and explanation of PSO