GoogleNet - rugbyprof/5443-Data-Mining GitHub Wiki

What is 1x1 Convolution filter?

The 1x1 convolutional filter behaves exactly the same as “normal” filters. The only new thing is the size 1x1 of the filter which indicates that the filter does not care at all about the correlation of information in the same feature map. Which mean, there is no visual pattern being learned in here. Instead, the filter pools the information across multi feature maps. The size of the kernel actually is 1 * 1 * k where k is the number of feature maps. This is one way to compress these feature maps into one. The 1×1 convolution is key because it will be used to reduce the dimensionality of its feature map.

Used for

In GoogLeNet architecture, 1x1 convolution is used for two purposes To reduce the dimensions inside the “inception module”. To add more non-linearity by having ReLU immediately after every 1x1 convolution.

http://iamaaditya.github.io/2016/03/one-by-one-convolution/

Inception Module

The inspiration comes from the idea that you need to make a decision as to what type of convolution you want to make at each layer: Do you want a 3×3? Or a 5×5? And this can go on for a while. So why not use all of them and let the model decide? You do this by doing each convolution in parallel and concatenating the resulting feature maps before going to the next layer.

The module basically acts as multiple convolution filter inputs, that are processed on the same input. It also does pooling at the same time. All the results are then concatenated. This allows the model to take advantage of multi-level feature extraction from each input.

Pooling Layer

It is common to periodically insert a Pooling layer in-between successive Conv layers in a ConvNet architecture. Its function is to progressively reduce the spatial size of the representation to reduce the amount of parameters and computation in the network, and hence to also control overfitting. The Pooling Layer operates independently on every depth slice of the input and resizes it spatially, using the MAX operation. The most common form is a pooling layer with filters of size 2x2 applied with a stride of 2 downsamples every depth slice in the input by 2 along both width and height, discarding 75% of the activations. Every MAX operation would in this case be taking a max over 4 numbers (little 2x2 region in some depth slice). The depth dimension remains unchanged.

http://cs231n.github.io/convolutional-networks/

What is dropouts for Regularization in Neural networks

Dropout is a technique where randomly selected neurons are ignored during training. They are “dropped-out” randomly. This means that their contribution to the activation of downstream neurons is temporally removed on the forward pass and any weight updates are not applied to the neuron on the backward pass. You can imagine that if neurons are randomly dropped out of the network during training, that other neurons will have to step in and handle the representation required to make predictions for the missing neurons. This is believed to result in multiple independent internal representations being learned by the network

https://machinelearningmastery.com/dropout-regularization-deep-learning-models-keras/