First-Person Hand Action Benchmark 2017

[first-person-benchmark-dataset] First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotations [PDF] [project page] [dataset not available here] [notes]

Guillermo Garcia-Hernando, Shanxin Yuan, Seungryul Baek, Tae-Kyun Kim

read xx/05/2017

Synthesis

Benchmark hand pose estimation and action recognition from RGB-D data in ego-centric setting

No prior 3D model

New Dataset : Daily Hand-Object Actions Dataset

100.000 RGBD annotated with 3D hand poses

3D pose acquired using magnetic 6D sensors + inverse kinematics over 21 joint hand model

1175 action samples : 45 categories of hand actions (write, scratch, wash, squeeze,...) manually annotated, 25 objects in 3 scenarios

for 4 objects in 10 actions : 6-D object pose ground truth and mesh

camera on the shoulder

RGB : 1920x1080 (contaminated with magnets and tapes) Depth : 640x480 30 fps

Pipeline

Preprocessing

compensate for anthropomorphic differences by normalizing hand poses (same distance between pairs of joints)

Network training

• Shallow LSTM
• Deep LSTM

Results

purely depth-based results: poor using hand poses produces the best performances with best-performer Gram Matrix and Lie group

Hand object interaction has to be present in training set