Exploratory Data Analysis - clizarraga-UAD7/Workshops GitHub Wiki

(Image Credit: Wikimedia Commons, CC)

• Describe main characteristics of dataset: number of rows/columns, missing data, data types, preview.
• How to clean corrupted data, handle missing data, invalid data types, incorrect values.
• Visualize data distributions using the Seaborn Library: bar plots, count plots, histograms, box plots, violin plots, and more
• Calculate and visualize correlations (relationships) between variables with the help of a heat map.

Exploratory Data Analysis is a Statistics approach of analyzing data sets in order to quickly summarize their main characteristics, and may be supported with simple data visualization like box plots, histograms,scatter plots, cummulative distribution functions, quantile-quantile (Q-Q) plots, among others.

John W. Tukey wrote the book Exploratory Data Analysis in 1977, where he held that too much emphasis in statistics was placed on statistical hypothesis testing and more emphasis needed to be placed on using data to suggest hypotheses to test. Exploratory Data Analysis does not need any previous assumption on the statistical distribution of the underlying data.

Tukey suggested computing the five number summary of numerical data: the two extremes (maximum and minimum), the median, and the quartiles since they are defined for all empirical distribution.

Turkey also gives a criteria for defining outlier data. If Q₁, and Q₃ are the first and third quartile positions, the interquartile range IQR = Q₃ - Q₁ , then an outlier value will fall below Q₁ - 1.5 IQR or above Q₃ + 1.5 IQR.

(Image credit: UF Biostatistics Open Learning Textbook, CC)

Histograms summarize the distribution of the data, by placing observations into intervals (bins) and counting the number of observations in each interval.

Boxplots are a box and whisker plot, which provides a compact summary of the distribution of a variable. A standard boxplot consists of:

• a box defined by the 25th and 75th percentiles,
• a horizontal line or point on the box at the median, and
• vertical lines (whiskers) drawn from each hinge (quartile) to the extreme value.

The cumulative distribution function (CDF) is a function F(X) that is the probability that the observations of a variable are not larger than a specified value.

A quantile-quantile (Q-Q) plot, or probability plot, is a graphical means for comparing a variable to a particular, theoretical distribution or to compare it to the distribution of another variable. One common application of the Q-Q plot is to check whether a variable is normally distributed.

Scatterplots are graphical displays of matched data plotted with one variable on the horizontal axis and the other variable on the vertical axis.

We enlist some functions which are useful in a EDA. We will find some function previously used.

!pip install sidetable

conda install -c conda-forge sidetable

import pandas as pd
import sidetable

import sidetable
import pandas as pd
import seaborn as sns

# Load the Titanic dataset
df = sns.load_dataset('titanic')

# Compute the frequency of one variable
df.stb.freq(['column1'], style=True)

# Build a frequency table for one or more columns
df.stb.freq(['column1', 'column2'], style=True)

# See what data is missing
df.stb.missing()

# Group data and add a subtotal
df.groupby(['column1', 'column2'])['col3'].sum().stb.subtotal()

Automated EDA packages can perform EDA in a few lines of Python code.

Here is a small list of them:

• Pandas Profiling
• sweetviz
• autoviz
• dataprep
• Lux API
• dataprep

See Low-code EDA Tools

Please see Slides.

The Seaborn Library is based on the general visualization library Matplotlib. Seaborn makes visualization of a dataset statistical properties more easier to use.

There are several types of graphics that we can produce with Seaborn, we will only show a small set of them, that can be used in performing an Exploratory Data Analysis.

import seaborn.objects as so

• Exploratory Data Analysis with Seaborn & Seaborn.objects
• Exploratory Data Analysis Jupyter Notebook (Pandas/Seaborn).

• 10 automated EDA libraries in one place. Implementation of Exploratory Data Analysis libraries with a few lines of Python code.
• Exploratory Data Analysis Wikipedia article.
• Exploratory Data Analysis. Causal Analysis/Diagnosis Decision Information System (CADDIS), U.S. Environmental Protection Agency.
• Exploratory Data Analysis. Engineering Statistics Handbook. NIST.gov.
• The Essential Data Science Reference Notebook. Eric Onofrey.

• Introduction to Data Visualization. Coding for Economists, Arthur Turrell.
• Pandas Cookbook
• Seaborn Visualization Library Examples
• Introduction to Seaborn Tutorial
• Seaborn.objects.Plot
• Matplotlib Visualization Library

Created: 01/30/2022 (C. Lizárraga); Last update: 02/20/2023 (C. Lizárraga)

CC BY-NC-SA