Deep Learning Dictionary - abdussamad007/deep-learning-autoencoder GitHub Wiki
Deep Learning Dictionary.
Hidden Layers: - Let me explain the hidden layers using cases to get the 360 visualization of hidden layers then we will discuss the math behind it.
Case 1: Facial Recognition of Human - where computer learns to detect human faces. Human face is a complex object, it must have eyes, a nose, a mouth, and to be in a round shape, for computer it means that there are a lot of pixels of different colors that are comprised in different shapes. And in order to decide whether there is a human face on a picture, computer has to detect all those objects.
I think the following diagram worth a thousand words.
Basically, the first hidden layer detects pixels of light and dark, they are not very useful for face recognition, but they are extremely useful to identify edges and simple shapes on the second hidden layer. The third hidden layer knows how to comprise more complex objects from edges and simple shapes. Finally, at the end, the output layer will be able to recognize a human face with some confidence.
Basically, each layer in the neural network gets you farther from the input which is raw pixels and closer to your goal to recognize a human face.
Case 2** How to to tell a computer that a picture of a bus is a bus?**
If you want a computer to tell you if there's a bus in a picture, the computer might have an easier time if it had the right tools.
So your bus detector might be made of a wheel detector (to help tell you it's a vehicle) and a box detector (since the bus is shaped like a big box) and a size detector (to tell you it's too big to be a car). These are the three elements of your hidden layer: they're not part of the raw image, they're tools you designed to help you identify busses.
If all three of those detectors turn on (or perhaps if they're especially active), then there's a good chance you have a bus in front of you.
Neural nets are useful because there are good tools (like backpropagation) for building lots of detectors and putting them together
Case 3: How do we see using our eyes?
Lets look at the picture first
In this picture, the flow of information runs from top to bottom. Sensory neurons — the eyeballs at the top — take input from the external world and encode it into a convenient form (which is typically electrical pulse trains for biological neurons, and numerical data for the computer "neurons" of artificial neural networks). They distribute this encoded information to other neurons, in the next layer below. Effector neurons — the stars at the bottom — send their signals to output devices (which are typically muscles for biological neurons, and computer terminals for artificial neurons). In between are neurons that neither see nor act upon the outside world directly. These inter-neurons communicate only with other neurons. They are the hidden layers.
Now, How do we represent layer in deep learning neural network?
Each neuron in the hidden layer (and output layer ) can be split into two parts : pre-activation and activation
pre-activation - It does the aggregation (How and which function is used?)
Mathematics : ai(x) = bi + Wi hi-1(x) i=1,2,3.... N , 1st, 2nd ,3rd layer etc.
ai-preactivation
bi-bias
Wi- weight , hi-1(x) is the function which we will will describe below called the activation function
Let's call the input vector x, the hidden layer activations h, and the output activation y. You have some function f that maps from x to h and another function g that maps from h to y
So the hidden layer's activation is f(x) and the output of the network is g(f(x))
Why have two functions (f and g) instead of just one?
If the level of complexity per function is limited, then g(f(x)) can compute things that f and g can't do individually.
Activation - It does the non linearity (How and which function is used?)
h(x) = g(f(x) called the activation function (for example logistic, tanh,linear etc. )
Reference: NPTEL Week 3: week 3
Quora: https://www.quora.com/What-is-the-role-of-a-hidden-layer
Stackexchange: https://stats.stackexchange.com/questions/63152/what-does-the-hidden-layer-in-a-neural-network-compute