Machine Learning Tutorial

Ridge Regression

Ridge regression is also a linear model for regression, so the formula it uses to make predictions is the same one used for [Ordinary Least Squares]]. In ridge regression, though, the coefficients (w) are chosen not only so that they predict well on the training data, but also to fit an additional constraint. We also want the magnitude of coefficients to be as small as possible. In other words, all entries of w should be close to zero. Intuitively, this means each feature should have as little effect on the outcome as possible (which translates to having a small slope), while still predicting well. This constraint is an example of what is called regularization. Regularization means explicitly restricting a model to avoid overfitting. The particular kind used by ridge regression is known as [L2 regularization.

The Ridge model makes a trade-off between the simplicity of the model (near-zero coefficients) and its performance on the training set. This is configured by the alpha parameter. The optimum setting of alpha depends on the particular dataset we are using. Increasing alpha forces coefficients to move more toward zero, which decreases training set performance but might help generalization.

from sklearn.linear_model import Ridge

ridge = Ridge(alpha=10).fit(X_train, y_train)

Check out the Linear Regression page to see how to get the coefficients.