Other Papers

2017

Arxiv (ICLR 2018 submission)

[1708.05349]

PixelNN: Example-based Image Synthesisn [PDF] [notes]

Aayush Bansal, Yaser Sheikh, Deva Ramanan

read 15/11/2017

Objective

Create plausible reconstructions from degraded representations of this image (for instance blurred, edges, surface normal map...)

Control image synthesis explicitly to produce, for instance by restricting the set of images it should resemble or by providing a guiding image.

Synthetis

Pipeline

Image regression

A CNN is used to map from the degraded input to the reconstruction. This reconstruction is usually over-smoothed (reconstructs only low to middle frequencies)

Pixel nearest neighbour mapping

The reconstruction is used to map each pixel to a pixel from a candidate image. This mapping takes into account different intermediate features extracted from a neural network. The cosine distances between features are used to evaluate the proximity between the context of the pixel of the intermediate reconstruction and the one of the training set. The mapping is then operated independently for each pixel. To speed-up the matching, K global nearest neighbours are first selected using high level features (from conv5 layer), and the pixel matches are looked for in a restricted window (not clear exactly how the center of the window is selected).

The output can be controlled by restricting the subset of the images in the training set which are used for the NN matching.

Experiments

Looks similar in quality to Pix2Pix with different shortcomings (sometimes sharper results for PixNN)