Scikit-learn has some support for neural networks, especially the type of basic fully connected feed forward network we will discuss today. But customization is very limited so it isn't useful if you want to create a neural network that is very optimized for your data. Good alternatives are PyTorch, TensorFlow, and Keras. All three are valid options. PyTorch and TensorFlow offer the most customizability. Keras is more user friendly for beginners and is built on top of PyTorch or TensorFlow. It offers less customizability but you can always do the customization in TensorFlow/PyTorch and then use that in Keras. PyTorch is very commonly used in biology or chemistry applications so we will use PyTorch in this class.

Fully connected feed forward neural networks are made up of an input layer, one or more hidden layers, and an output layer where the outputs of every node in a layer feeds into every node in the next layer. Sometimes these layers are also referred to as dense or linear layers. The input layer is made up of your features and the output layer is the size of what you are trying to predict. With neural networks you can predict multiple things at the same time, which can be useful and improve performance if the tasks are related. For now we will stick to a single prediction for either binary classification or a single regression task so our output layer will have a size of 1. The hidden layers are responsible for processing the information in the data so that the representation in the hidden layers is useful for predicting the target.
Each node will have weights associated with it, each weight is multiplied by it's corresponding input term and the values are summed. The nodes also optionally include a bias term which is added to this sum without being multiplied by an input, this is similar to an intercept value in a linear regression. A bias term is included by default for linear layers in PyTorch. Therefore if we have a node with three inputs this sum will be:
$z = w_1X_1 + w_2X_2 + w_3X_3 + b$
The weights and bias terms are parameters because they are what will be fit to optimize our neural network's performance. Both weights and bias terms are fit during training, but often we just say weights rather than parameters.

If there are multiple hidden layers, you must include a nonlinear activation function between hidden layers. The reason for this is that each node does a linear combination of inputs. If multiple linear combinations are stacked on top of each other they can be reduced to a single linear combination, so effectively adding additional layers does not allow the model to learn any more than a single layer. A nonlinear activation function prevents this from happening. Common activation functions include ReLU which outputs z if z > 0 and 0 otherwise or the sigmoid function which outputs the value of the sigmoidal equation applied to z. There are other activation functions as well, see here for a list and plots of common functions: https://www.v7labs.com/blog/neural-networks-activation-functions

Backpropagation is the method by which our model will be fit to our data. It works by calculating the gradients for the parameters across the network, which is essentially asking how the loss (an objective function used to score our predictions) change as the model parameters change. Below the steps for fitting the network are discussed.

• To prepare the data for training it is often necessary to split the data into batches. This is because if we have large datasets and models with many parameters it will take a lot of memory to train on all the data at once. Therefore data is often split into batches, parameters are updated for one batch before moving on to the next. One fully run of all batches is an epoch. Training often needs to be repeated for multiple epochs.
• For each batch a forward propagation is done. This means predicting the result with the current parameters.
• The loss is computed. The loss is an objective function where a lower value indicates better predictions. The goal of training the model is to minimize the loss. Custom loss functions can be defined to get the model behavior you want. For now, we will just focus on simple loss functions that determine classification or regression quality.
• To update the parameter we need to know how the loss will change with respect to each parameter. If decreasing a weight will decrease the loss we want to decrease it, if increasing a weight will decrease the loss then we want to increase it. To do this we want to calculate the gradient, or the partial derivatives of the loss with respect to each parameter. Tracking how the loss changes with weights as we move backward through the network is the backpropagation part. The gradients are calculated using the calculus chain rule. The parameters are then updated based on the learning rate and gradient. The learning rate will determine how large the updates are with each batch of training (see below for more details on the learning rate).

One challenge in neural network training is that gradients can either be very small (vanishing) or large (exploding). Gradients in early layers in the network involve the product of many gradients from the later layers. If these are less than one they can compound to result in very small gradients for early layers, approaching zero. If gradients are too small, parameters will not change significantly with each batch of training and will be stuck, making it difficult to optimize the early layers. Derivatives of the activation functions are included in the gradients. Therefore, activation functions that have a maximum value less than one contribute to vanishing gradients (e.g. the maximum possible derivative of the sigmoid function is 0.25). For ReLU the maximum derivative is 1, so you are less likely to get vanishing gradients with ReLU. However, a disadvantage of ReLU is it can lead to dead neurons that always output a zero value and are therefore not useful in making predictions. Leaky ReLU is a good compromise where the function is max(ax, x) where a is a hyperparameter between 0 and 1. Leaky ReLU only outputs 0 if x=0 but has a max derivative of 1 like ReLU, if a is very small however it could still contribute to vanishing gradients if inputs to the activation function are often negative.
Exploding gradients can occur if weights, inputs, or loss are too large because they also contribute to gradients. Scaling input features can avoid exploding gradients due to large inputs. Exploding gradients cause the model to be unstable with large changes to parameters with each batch. This often becomes a worsening problem because large changes in parameters may make the loss higher and also make the parameters larger which then in turn makes the gradients even larger.

Weights need to be initialized randomly because if neurons in the same layer are given identical weights they will not be able to learn different information. They cannot be too large or too small to avoid the exploding or vanishing gradient problem discussed above, respectively. There are different strategies for setting weights based on the number of nodes feeding into and out of a node. PyTorch uses these methods, the defaults are generally good but you can change this as a hyperparameter if you choose.

The learning rate determines how much the weights change with each update while the optimizer is the algorithm that determines the weight updates. Adam in some of its derivatives are among the most commonly used, where gradient descent is the most simple to understand (parameters are updated by subtracting the product of the learning rate and the gradient from the current value). If the learning rate is too small parameters will not change significantly between batches and the model will take longer to train, there is also a higher risk of getting stuck in a in the region around a local minima, where a larger learning rate may cause the model to be able to jump out with a large parameter update. If the learning rate is too large the model can become unstable and may not converge at a minima which has improved performance. Some training strategies change the learning rate throughout training which can avoid these problems, and other than changes that are built into the optimizer, we will focus on constant learning rates for now. In addition to small learning rates, getting stuck at a saddle point or local minima can also prevent the model from being updated because the gradients at these locations are zero. Optimizers like Adam include momentum (based on average gradients from recent steps) and changes in step size in response to gradient size, this can sometimes avoid the model getting stuck in a saddle point or local minima. Mini batches can also help with this because gradients will be more variable between batches.

The output layer is the final layer in the network and outputs what you are trying to predict. The size corresponds to how many tasks the model is being trained for. The activation function and loss function are chosen for the specific task. In PyTorch, the loss function includes the proper activation function so only the loss function needs to be set and you should not have an activation function on your final layer. Other libraries may work differently. See the table below for the built in activation functions and names of the proper loss functions. Some tasks have multiple possible loss functions.

Task	Built-in Activation	Loss
Regression	None	MSEloss, L1Loss, SmoothL1Loss
Single binary classification	Sigmoid	BCEWithLogitsLoss
Multiclass	Softmax	CrossEntropyLoss
Multilabel	Sigmoid	BCEWithLogistsLoss

You generally want to track the training progress for each epoch for both training and validation data. You can track the loss for training data and metric(s) of your choice for both the training and validation data. You can see when the model converges for the training data when improvements in loss get close to zero between epochs, this likely indicates your model has found a minima or saddle point and is unlikely to improve further therefore there is no point in running additional epochs. If loss remains steady from the beginning it could indicate a vanishing gradient problem or if loss gets larger it could indicate an exploding gradient problem. Comparing training and validation allows you to assess overfitting, if the training has much higher metrics than validation then your model is overfit. Sometimes your training loss and metric will continue to improve while the validation stays the same or even gets worse. If this occurs you might want to add early stopping where the model stops at an epoch where validation improvement has been below a specified threshold for the a specified previous number of epochs. Training data will generally have better metrics than validation data but sometimes you will see validation data do better, especially at early epochs because metrics are calculated throughout training for training data and after all batches has been run for validation data.

Dropout can help with overfitting. It involves randomly dropping a set of neurons (how many is determined by the dropout rate hyperparameter). This prevents overfitting by keeping a model from relying too much on one or a few neurons. Dropout layers should be added after the activation layer of every hidden layer.

The hyperparameters for a fully connected neural network include the number of layers, the number of neurons in each of those layers, the activation function for each layer, and if dropout layers are included and if so what the dropout rate is. They also include things like the learning rate and the optimizer, along with any optimizer-specific hyperparameters. But we will focus on the hyperparameters specific to the layers themselves here. The number of parameters for the network will be determined by the size of the input and output layers as well as the how many hidden layers there are and their respective sizes. Larger datasets can generally support larger models without overfitting so the size of the dataset should be kept in mind when choosing the number of hidden layers and their sizes. A general rule for fully connected layers is that the hidden representation size should remain the same or get smaller further in the network. If these representations get smaller it can be viewed as compressing the data where the model is trained to retain the most predictive information in the deeper layers. The exact optimal set of hyperparameters depends on the data.
To optimize hyperparameters, GridSearchCV can be used for combinations of a a few hyperparameters, but it quickly becomes impractical because hyperparameter choice is much more open and contingent for neural networks than it is for models we have previously discussed. For example we have to choose the number of hidden layers, and then for each hidden layer that exists we have to choose the dimensions. The more layers, the more combinations. There are statistical methods for hyperparameter optimization that search hyperparameter space more efficiently than testing every combination. There are also emerging methods that use agentic AI built on LLMs either alone or in combination with a statistical method that can reason about what changes to make to the model (e.g. by detecting overfitting by comparing training and validation metrics). We will discuss these in the future when we talk about other neural networks but they can be applied to fully connected feed forward networks.

Chemistry example: https://colab.research.google.com/github/aswalker-lab/AIChemBiochem/blob/main/7_Neural_networks/Chemistry_neural_networks.ipynb
Biochemistry example: https://colab.research.google.com/github/aswalker-lab/AIChemBiochem/blob/main/7_Neural_networks/Biochemistry_neural_network.ipynb