Activation Functions

Constant (Fixed)

Pre-defined learning rate schedule

function FIXED() {
  return 0.3; // Learning Rate
}

Time-Based Decay

Pre-defined learning rate schedule

Step Decay

Pre-defined learning rate schedule

Exponential Decay

Pre-defined learning rate schedule

Momentum

Adaptive learning rate method

Adagrad

Adaptive learning rate method

Adadelta

Adaptive learning rate method

RMSprop

Adaptive learning rate method

Adam

Adaptive learning rate method

Bibliography

• Introduction to Cyclical Learning Rates
• Finding Good Learning Rate and The One Cycle Policy