In this chapter we will introduce data visualization using matplotlib and seaborn, we will also discuss some statistical analysis using scipy.

First we will discuss how to compare two independent datasets, for example data collected from an experiment conducted with two different conditions. For visual comparison of the data and their distributions, making a histogram can be helpful. With matplotlib, histograms can be plotted with:
plt.hist(data, bins=n_bins)
where n_bins is the number of bins you want your data to be segmented into. See the example code for how to plot two different histograms.
Before comparing two datasets, you want to know if your data is normally distributed. If you have sufficient number of datapoints, there are statistical tests you can use to check if your data is normally distributed. You likely need at least 20-30 datapoints to start detecting deviations from the normal distribution, at lower datapoints your test will likely be underpowered and can give a p-value >> 0.05 even if the data is not normally distributed. Also with too many data points, the test may be oversensitive to some slight deviations and give a low p-value for data that is only slightly off from normally distributed. Tests for normality include Shapiro-Wilk (better for smaller datasets) and D'Agostino-Pearson (better for larger datasets). To run these:
W, p = shapiro(data)
stat, p = normaltest(data1)
The values returned are the statistic for the test and the associated p-value. If in doubt, you can always use a non-parametric test if you do not know if your data is normally distributed.
Non-parametric tests - those that do not assume the data follows a specific distribution include Kolmogorov-Smirnov and Mann Whitney U test, you can also use these tests for normally distributed data but they will generally have lower power to detect differences. These tests can be imported from scipy and then run:
from scipy.stats import ks_2samp, mannwhitneyu
ks_stat, ks_p = ks_2samp(data1, data2)
u_stat, u_p = mannwhitneyu(data1, data2, alternative="two-sided") Two-sided looks for differences in both directions (higher and lower)
Kolmogorov-Smirnov compares the distributions of the two datasets, while Mann Whitney U ranks all samples from both datasets and then compares those ranks, so it is more to see if one dataset has significantly more samples of a higher rank than the other.
If your data is normally distributed, you can use a t-test. Student's t-test can be used if the variances are equal and Welch's t-test if they are not. Levene's test can be used to test if two samples have significantly different variances. As with testing normality, if you do not have enough samples you may get a p > 0.05 but that does not mean that the variances are the same. Levene's test:
lev_stat, lev_p = levene(data1, data2)
T test:
t_stat, t_p = ttest_ind(data1, data2, equal_var=False) This is for Welch's set equal_var=True for Student's

We can also make bar graphs in matplotlib to show comparisons, to make a bar graph in matplotlib
ax.bar(xpos, means, yerr=sems, capsize=6) xpos is where on the x-axis you want each bar to show, means is the height of each bar, yerr is the error bars which could be standard error or standard deviation.
See example code for details on how to add indicators of significance.

To visualize correlation we can use matplotlib to make a scatter plot:
plt.scatter(x, y)
We can calculate Pearson's correlation or Spearman's correlation, Spearman's correlation is better for nonlinear correlations:
pearson_corr, pearson_p = pearsonr(x, y)
spearman_corr, spearman_p = spearmanr(x, y)
We can also fit the data to a linear regression using scipy's linregress:
slope, intercept, r_value, p_value, std_err = linregress(x, y)

Example code can be found in the github or accessed through the following Colab" https://colab.research.google.com/github/aswalker-lab/AIChemBiochem/blob/main/2_Visualizing_data_and_stats/Visualizing%20Data%20and%20Stats.ipynb
You will need to upload the two_datasets_example and correlation_dataset csv files

We will now apply what we've learned to look at correlations between molecular features and solubility (Chem exercise) or correlations between different elements of x-ray crystallography structures of proteins. Choose one of these exercises to complete.

Material will be added later