Neural Network Advanced - utkaln/machine-learning GitHub Wiki

Shorcut to Neural-Network-Basics

Gradient Descent replaced with Adam Optimizer

• Gradient Descent can be further optimized to automatically choose better Learning Rate aka alpha
• This is done by choosing Adam Optimizer instead of traditional Gradient Descent equation with a manual input for learning rate
• This is represented in Tensorflow with a minor modification in the compile step that computes the loss (cost) function

model.compile(
  optimizer=tf.keras.optimizers.Adam(learning_rate=1e-3), 
  loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True))

Convolutional Neural Network

• Instead of each layer of Neural Network trying to read into the all the training data or output of the previous layer, another alternative is to only look into a subset of data to make more efficient prediction and reduce the risk of overfitting. This type of layers are called Convolutional Layers.
• Convolutional is a strong alternative to Dense layer