Learning Rate Scheduling -> python plot https://www.kaggle.com/code/maniklakherwal/learning-learning-rate-scheduling/notebook

Question
On beta with RMSProp ... this sentence from:
https://towardsdatascience.com/gradient-descent-with-momentum-59420f626c8f [In other words, ]$\beta$ represents the degree of weighting increase and acts as a constant smoothing factor. A lower $\beta$ discounts older observations faster.

Does it make sense? yeah

Geron: You can easily verify that if the gradient remains constant, the terminal velocity (i.e., the maximum size of the weight updates) is equal to that gradient multiplied by the learning rate η multiplied by 1 (ignoring the sign). For example, if β = 0.9, then the 1−β terminal velocity is equal to 10 times the gradient times the learning rate, so Momen‐ tum optimization ends up going 10 times faster than Gradient Descent!

Lecture 8, Cornell https://www.cs.cornell.edu/courses/cs6787/2019fa/lectures/Lecture8.pdf

A brief history of learning rate schedulers and adaptive optimizers https://spell.ml/blog/lr-schedulers-and-adaptive-optimizers-YHmwMhAAACYADm6F

from sklearn.model_selection import GridSearchCV

param_grid = [
    {'n_estimators': [3, 10, 30], 'max_features': [2, 4, 6, 8]},
    {'bootstrap': [False], 'n_estimators': [3, 10], 'max_features': [2, 3, 4]},
]
forest_reg = RandomForestRegressor()

grid_search = GridSearchCV(forest_reg, param_grid, cv=5,
                           scoring='neg_mean_squared_error',
                           return_train_score=True)

grid_search.fit(housing_prepared, housing_labels)

_{From Aurélien Géron, Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow (Second Edition)}

Why? found stuff on Scikit-Learn that is better.

Learning Rate Schedulers: https://d2l.ai/chapter_optimization/lr-scheduler.html

good graphset here on under/overfitting yes/no algorithmic: https://datascience.foundation/sciencewhitepaper/underfitting-and-overfitting-in-machine-learning

• StandardScalar
• Perceptron doc

• Keras examples on under/over fitting: https://curiousily.com/posts/hackers-guide-to-fixing-underfitting-and-overfitting-models/
• k-fold https://machinelearningmastery.com/deep-learning-models-for-multi-output-regression/

Playground https://playground.tensorflow.org