In unsupervised learning there are no labels for the data. Therefore, the goal of unsupervised learning is usually to cluster data into groups that are similar. Unsupervised learning is challenging because there is no ground truth, making it difficult to have metrics to assess clustering results. In addition molecular and protein structures are often modular, made up of substructures or protein domains that can be mixed and matched. In this situation there is no good way to cluster molecules/proteins that may share some components and differ in others. The best clustering may depend on what the label for the data ends up being, but we are usually doing unsupervised learning in cases where we do not know the label.

It can be helpful to visualize the data so that we can assess the size and shape of any clusters. However our data is usually multidimensional and not possible to visualize. We previously discussed how PCA can reduce dimensionality to a user determined number of components, with 2 or 3 components we can plot our data for visualization. When the data is high dimensional, either due to many features or many datapoints, PCA visualization will often not show clear clusters making it difficult to use PCA to visualize clusters. tSNE and UMAP are alternatives to PCA for transforming data into two dimensions that address poor cluster separation in PCA. To accomplish this they use non-linear transformations that result in data being more separated than it is in PCA, often showing clustering. However because this nonlinear transformation distorts distances from the original space and because there is some randomness in the results, tSNE and UMAP should only be used for visualization and not over interpreted.

There are many different clustering methods. Below are some of the most commonly used methods.

In K-means clustering the user specifies the number of clusters (k). The method identifies clusters that minimize the within cluster sum of squares, or the square distances between the points assigned the the cluster and the cluster centroid (mean of the cluster). K-means works best for more circular clusters and worse for elongated clusters or those with irregular shapes.

In affinity propagation the number of clusters is set by the clustering algorithm based on what clusters the data best rather than the user, although the preference hyperparameter will influence how many clusters are chosen. It tries the find the best "exemplars" or most representative points for each cluster an then assign points to the clusters based on their similarity to the exemplars. Exemplars are determined by passing messages between the points and finding those that best represent the other points.

Hierarchical clustering groups points according to a hierarchy, so you can have subclusters within clusters. This helps deal with some of the modularity of chemical and biochemical data, although when there are many different substructures or domains that can be shared across molecules or proteins it becomes more difficult even for this type of clustering. Agglomerative clustering is a type of hierarchical clustering available in scikit-learn that starts from the bottom-up with individual points being merged into clusters. It involves finding the most similar clusters and merging them and then repeating the process. The full hierarchy is determined and then the clusters are determined from there. Different methods can be used to calculate the distance between clusters which can be set with the metric and linkage hyperparameters in scikit-learn. The number of clusters can either be set by the user, in which case the clustering algorithm runs until that number of clusters has been reached or a distance can be set and used to cut the dendrogram representing the hierarchy so that clusters are kept separate if they are above a specified distance threshold from each other.

DBSCAN clusters points based on density. It is a good alternative for data where the clusters are irregularly shaped. It will cluster so that points assigned the the cluster core have enough other points in their surroundings. Core points near each other are grouped into clusters until the cluster is complete. DBSCAN also allows for singleton, or noise, points that are not assigned to a cluster. A weakness of DBSCAN is that it assumes a single density for all clusters but you might have some clusters that are more diffuse than others and these might all be assigned as noise points. HDBSCAN addresses this by assessing local density and gradually relaxing density thresholds to merge low density clusters.

Balanced Iterative Reducing and Clustering using Hierarchies (BIRCH) clustering is designed to use less memory and therefore can work on very large datasets that require a lot of computational power to cluster using the other methods. Locally dense groups are clustered first, then these groups are treated as a single point for subsequent rounds of clustering, avoiding having to do computation on the individual points. The clusters are represented as a tree and points are added one at a time, either being added to an existing cluster or as a new branch on the tree. Once the final tree is constructed, subclusters can be further grouped. BIRCH performs best on compact clusters and does not handle irregularly shaped clusters well.

Butina clustering was designed specifically for clustering small molecules (original reference: Butina. JCICS, 1999. 39. 747-750.). It uses the Tanimoto similarity to compare molecules and then clusters similar molecules together. The Tanimoto similarity is calculated from molecular fingerprints.

While it is difficult to assess the quality of clusters in the absence of class labels, there are a few metrics for doing so. These include the Silhouette Coefficient (ranges from -1 to 1, higher is better) which measures the distance between a sample and other points in the class compared to the distance between a sample and all points in the next nearest class, the Calinski-Harabasz Index (higher is better) which measures class definition such that points within a class should be close together and the classes should be far apart, and the Davies-Bouldin Index (lower is better) which looks at cluster separation by looking at the worst case similarity between a cluster and a neighbor that it is most difficult to distinguish from that cluster. All three metrics may unfairly penalize non-convex clusters like those found by DBSCAN or HDBSCAN so this should be considered when comparing clusters from these techniques to others.

Chemistry example: https://colab.research.google.com/github/aswalker-lab/AIChemBiochem/blob/main/6_Unsupervised_learning/Chem_Unsupervised_Learning.ipynb
Biochemistry example: https://colab.research.google.com/github/aswalker-lab/AIChemBiochem/blob/main/6_Unsupervised_learning/Biochem_unsupervised_learning.ipynb