General Adversary Networks (GANs)

GANs consists of two networks coupled together, one is called Generator and the other is called as Discriminator. Generative will create an output and Discriminator will evaluate whether the output generates is considered acceptable or not. While training, the Discriminator will provide a feedback to the Generative and the Generative will generate a new output on the basis of the feedback received. Then the Discriminator will again evaluate the new output and provide another feedback. Thus, the training cycle continues untill the Discriminator is convinced that the output generated by the Generator is acceptable.

Discriminator

The architecture of Discriminator consists of series of Convolution layers without any Pooling but with a stride of 2 or more for downsamping. The output of Discriminator is a probability value between 0 and 1 with 0 being completely not acceptable and 1 means perfectly acceptable. The activation function used in each CNN layer is a leaky ReLU. A dropout between 0.4 and 0.7 between layers to prevent over-fitting and memorization.

A sample code using Keras 2.0 and Tensorflow > 1.0 backend showing how a Discriminator is defined is given below: (taken from here)

self.D = Sequential()
depth = 64
dropout = 0.4
-In: 28 x 28 x 1, depth = 1
-Out: 14 x 14 x 1, depth=64
input_shape = (self.img_rows, self.img_cols, self.channel)
self.D.add(Conv2D(depth*1, 5, strides=2, input_shape=input_shape,\
padding='same', activation=LeakyReLU(alpha=0.2)))
self.D.add(Dropout(dropout))
self.D.add(Conv2D(depth*2, 5, strides=2, padding='same',\
activation=LeakyReLU(alpha=0.2)))
self.D.add(Dropout(dropout))
self.D.add(Conv2D(depth*4, 5, strides=2, padding='same',\
activation=LeakyReLU(alpha=0.2)))
self.D.add(Dropout(dropout))
self.D.add(Conv2D(depth*8, 5, strides=1, padding='same',\
activation=LeakyReLU(alpha=0.2)))
self.D.add(Dropout(dropout))
# Out: 1-dim probability
self.D.add(Flatten())
self.D.add(Dense(1))
self.D.add(Activation('sigmoid'))
self.D.summary()

The final Dense layer has only one output as probability value to whether to accept or reject completely the input image.

Generator

The Generator synthesizes fake output based on random noise input using the inverse of convolution called transposed convolution. Upscaling between layers is used to add realistic effects and batch normalization is used for stabilizing learning. ReLU is used as activation function. Dropout of 0.3 - 0.5 is used in first layer to avoid over-fitting. The following code defines a Generator: (taken from here)

self.G = Sequential()
dropout = 0.4
depth = 64+64+64+64
dim = 7

-In: 100
-Out: dim x dim x depth

self.G.add(Dense(dim*dim*depth, input_dim=100))
self.G.add(BatchNormalization(momentum=0.9))
self.G.add(Activation('relu'))
self.G.add(Reshape((dim, dim, depth)))
self.G.add(Dropout(dropout))

-In: dim x dim x depth
-Out: 2xdim x 2xdim x depth/2

self.G.add(UpSampling2D())
self.G.add(Conv2DTranspose(int(depth/2), 5, padding='same'))
self.G.add(BatchNormalization(momentum=0.9))
self.G.add(Activation('relu'))
self.G.add(UpSampling2D())
self.G.add(Conv2DTranspose(int(depth/4), 5, padding='same'))
self.G.add(BatchNormalization(momentum=0.9))
self.G.add(Activation('relu'))
self.G.add(Conv2DTranspose(int(depth/8), 5, padding='same'))
self.G.add(BatchNormalization(momentum=0.9))
self.G.add(Activation('relu'))

-Out: 28 x 28 x 1 grayscale image [0.0,1.0] per pix

self.G.add(Conv2DTranspose(1, 5, padding='same'))
self.G.add(Activation('sigmoid'))
self.G.summary()
return self.G