Recurrent neural networks (RNNs) process input sequences, including of text-based inputs like natural language, SMILES strings, and protein or nucleic acid sequences. For these text based inputs it is often helpful to tokenize them, which involves breaking the text sequencing into parts. The most basic way to do this would be to tokenize for each individual character but then the model has to learn how that character's meaning changes depending on what other characters are next to it. For example in SMILES strings a "C" can indicate a carbon atom or it can be the first character in the symbol for chlorine, "Cl". To make it so that our network does not have to learn this we can tokenize our SMILES string so that there is a token for all two character atom symbols. This would be considered an atom-level tokenization of a SMILES string. We can also go further and tokenize common substrings, for example "C(=O)O" is a carboxylic acid group. For simplicity in this class we will focus on atom-level tokenization, if you want to read more about tokenization for common substructures, see: https://pubs.acs.org/jcisd8/article/61/4/1560/849860/SMILES-Pair-Encoding-A-Data-Driven-Substructure as an example.

For nucleic acid and protein sequences, these are more commonly tokenized at the character level. However you could tokenize common motifs or set a specific length for each token, this would be tokenization by k-mer where k is the length of the token. For this class we will use tokenization and the individual base or amino acid level.

Once we have our tokenization strategy determined, we create a vocabulary which is the full set of tokens. In addition to tokens from our actual dataset this can include special tokens such as tokens to make all sequences the same length of which we can use for rare tokens or unrecognized tokens in the test set that were not present in the training set and therefore not included in our vocabulary. Each token in the vocabulary is assigned an index. These indices are arbitrary (for example the token assigned index 1 might not have a similar meaning to the token assigned index 2 but it might be similar to index 5). To make a more meaningful representation we can use an embedding layer to process our tokens before inputting them to our neural network. The embedding layer is a matrix of dimensions vocabulary size * embedded dimension, where the embedded dimension is a hyperparameter chosen by the user. All of the entries in the matrix are parameters that will be fit when we train our network. Each row corresponds to the token with the same index of the row (1st token goes with the first row, etc.). After fitting these often represent the meaning of the tokens, tokens that mean similar things are likely to have similar representations in the embedding matrix, for example we might expect Cl and Br to have similar embeddings because they are both halogens or in a protein example for S and T to have similar representations because they are similar amino acids.

A recurrent neural network (RNN) reuses nodes along the sequence, where the hidden representation of a node can be passed back to that same node for the next element in the sequence. Because RNNs can be used for time series as well as sequences inputs are often referred to as the input at time t, but we can also think of this as the t-th item in the sequence ($x_t$). At that point the input will be $x_t$ and the hidden representation from the previous character in the sequence $h_{t-1}$. We can train our RNN to predict the next element in the sequence or some other value associated with that position in the sequence. The former case can be used to generate text but RNNs are much worse as generative AI methods than transformers, and we will discuss generative AI and transformers more in future lectures. RNNs can also be used to predict some value at the end of the sequence, for example some property of a molecule or protein from the SMILES or sequence, respectively.

RNNs often do not perform well for longer sequences, they will forget information from early in the sequence which may be important for prediction and are also prone to vanishing gradients because the recurrent transformation is applied many times. Long short-term memory (LSTM) networks were designed to mitigate this and we will focus on LSTMs in this class. LSTMs have additional operations in the recurrent part of the network, referred to as gates and they also have a cell state that is meant to keep track of information over longer parts of the sequence. The gates are the forget, input, and output gates. The forget gate is determined from the input and previous hidden state and goes through a sigmoid activation function so that values are close to zero or one, this is then multiplied element-wise by the cell state and therefore determines what from the cell state is forgotten (anything multiplied by near-zero value). The input gate is similar and decides what from the input is added to the cell state for long term memory storage. Finally, the output gate determines what information from the cell state is used for the new hidden representation.
LSTMs are often just a single hidden layer, but they can have multiple layers. If they have multiple layers, the hidden representation from the previously layer at that position is used as the input to the next layer. LSTMs can be bidirectional (BiLSTM), these are made up of two separate LSTMs which each process the sequence in opposite directions. For input to the fully connected portion for classification or regression we can take the final hidden representation from each of these LSTMs and concatenate them.
The ability to remember information from earlier in the sequence is especially important for SMILES. To understand the molecule, we want our network to be able to represent atoms properly based on what they are bound to. That means the network must remember an atom at a branching point, which are indicated by (), until the end of the branch when we reach the next atom it is bound to. If our branch is very long this becomes difficult. We also want our model to remember atoms at the start of a ring until it reaches the end of the ring if the ring is large or has branching off the ring this also becomes difficult. For protein sequences, positions that are distant in the sequence might be close together in the folded protein so we might want the LSTM to remember amino acids over long distances for that reason. This will be difficult for our LSTM because it has no structural information to help inform to remember something that might be in contact with a later amino acid.

Chemistry example: https://colab.research.google.com/github/aswalker-lab/AIChemBiochem/blob/main/9_Recurrent_neural_networks/9a_Chemistry_LSTM.ipynb
Biochemistry example: https://colab.research.google.com/github/aswalker-lab/AIChemBiochem/blob/main/9_Recurrent_neural_networks/9b_Biochemistry_LSTM.ipynb
These notebooks, especially the biochemistry one are slow on CPU, so you may want to use a GPU runtime to run these on Google colab.