GMDL (Workshop at ECCV) 2016

[arxiv 1603.08152] How useful is photo-realistic rendering for visual learning? [PDF] [notes]

Yair Movshovitz-Attias, Takeo Kanade, Yaser Sheikh

read 17/04/2018

Objective

Assess effect of realistic lighting and rendering on generalization to real data Show that realism is important towards object viewpoint estimation

Synthesis

Dataset generation

Rendering settings

• Randomized light:

• temperature

• intensity

• elevation on a sphere

• Randomized Shutter speed and F-stop

• Lens vignetting

• Backgrounds from VOC where no car is present

• All images are JPG compressed

Training augmentation

• Color jittering
• Color channel swapping
• Image degradation by downsampling
• place rectangular patches from uniform color or Pascal dataset on the renders

Train-test split

819.000 images for training, 1.800 for testing (leave one car model out)

Experiments

Test datasets

• CMU-Car (camera matrices for 3.240 cars, manual annotations of landmark points to determine view point)
• PASCAL3D+

Viewpoint prediction

• Two synthetic datasets:
  • RenderScene : one car model shot from 1800 angles
  • RenderCar: use entire set of 3D CAD vehicles models
• Use ground truth bounding boxes Using RenderCar is on par or outperforms training on small real datasets alone, combining leads best results

Results

Lighting and Material complexity

Complex Material and Directional lighting > Complex material, ambient lighting > Simple material and ambient lighting

Mixture of synthetic and real

At given train size, Mixing 25% to 50% synthetic with real performs best, slightly above (1% absolute diff in angular error) using only real and significantly better then using only synthetic (8% absolute diff in angular error)