In regression, we are trying to predict a continuous value from the features. Models are fit to minimize the difference between predicted and actual values.

As with classification, there are many different regression methods. We will go through some of the most common ones that are available through scikit-learn.

Linear regression is fit by minimizing the residual sum of squares.
Residual sum of squares:
$$\underset{\omega}{min}||X\omega-y||^2_2$$
This is the same as linear regression that you've learned before (e.g. fitting y=mx+b) but here we have multiple ms ($\omega$) and multiple x values.

Support vector regression tries to get all predicted values within a set threshold ($\varepsilon$) of the data. Only the data points on the threshold or beyond it will affect the fit, these are the support vectors. The model will prioritize a flat function (even weights) if all data fits within the threshold. As with a support vector classifier, different kernels can be chosen to predict values from transformed features.

Trees can act as regression models instead of classification. Instead of predicting the class, the leaf will predict the output value. The regression can be done with a single tree or multiple trees in a random forest.

Metrics for regression include mean absolute error (MAE), root mean squared error (RMSE), R2 score, and pearson's correlation. R2 score can be calculated with the formula:
$$R^2=1-\frac{SS_{res}}{SS_{tot}}$$
$SS_res$ is the residual sum of squares and $SS_tot$ is the total sum of squares (between actual data points and mean across data points).

Ridge and lasso penalties can help mitigate overfitting, they can be applied to any classification or regression model that uses weights (also called coefficients) and minimizes an objective function. Ridge (or l2) penalties add a sum of the squared weight terms to the objective function and lasso (or l1) penalties add a sum of the absolute values of the weight terms to the objective, in both cases they are weighted by a hyperparameter $\alpha$. The ridge penalty will prevent any weight from being dominant and leads to weights being more even while lasso penalty rewards some weights being set to zero. Both of these can reduce overfitting - ridge prevents one feature that might be correlated with the target value by chance in the training set from being weighted too strongly while lasso can remove less predictive features by setting their weight to zero. ElasticNet combines both with an additional hyperparameter to define how the ridge and lasso penalties are weighted relative to each other. Some scikit-learn classifier and regression models already apply one or both of these by default. Check the documentation to see what is available for each classifier and what is applied by default.

https://colab.research.google.com/github/aswalker-lab/AIChemBiochem/blob/main/4_Regression/Regression%20for%20Chemistry.ipynb
You will need to upload the delaney dataset.
https://colab.research.google.com/github/aswalker-lab/AIChemBiochem/blob/main/4_Regression/Regression%20for%20Biochemistry.ipynb
You will need to upload the promelt dataset