🔍 Exploratory Data Analysis (EDA)

What is EDA?

EDA: Example Dataset

Example Data Set: titanic

import seaborn as sns
titanic = sns.load_dataset('titanic')
titanic.head()

• The titanic dataset contains information about passengers on the Titanic.
• Collection of both categorical (e.g., sex, class) and numerical (e.g., age, fare) variables.

EDA: Univariate Analysis

What is Univariate Analysis?

• Univariate analysis focuses on analyzing and summarizing a single variable.
• Aim to understand:
  • Location (mean, median, mode).
  • Variability (standard deviation, interquartile range).
  • Distribution shape (skewness, kurtosis).
  • Count and frequency of categorical variables.

Categorical Variables: Frequency and Proportions

Statistics

• For categorical variables, we produce a table summarizing:
  • Frequency: The count of each category.
  • Proportion: The relative frequency of each category (frequency divided by total count).

pd.DataFrame({
  'Frequency': titanic['class'].value_counts(),
  'Proportion': titanic['class'].value_counts(normalize=True),
  'Percentage': titanic['class'].value_counts(normalize=True).mul(100).round(2).astype(str) + '%'
}).sort_index()

Categorical Variables: Visualizations

Visualization Options

• Our visualization options essentially just display this same information:
  • Count Plot: Displays the frequency or proportion of each category.
  • Pie Chart: Shows the proportion of each category as a slice of a pie.
    • Not generally recommended for accurate comparison, but can be useful for showing relative proportions.

colors = sns.color_palette('pastel')[0:3]
fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(15, 4))
sns.countplot(
  data=titanic, 
  x='class', 
  ax=ax[0]
  )
ax[0].set(title='Count Plot', xlabel='Class', ylabel='Count')
ax[1].pie(
  titanic['class'].value_counts(), 
  labels=titanic['class'].unique(), 
  colors=colors, 
  autopct='%.0f%%')
ax[1].set_title('Pie Chart of Passenger Class')
plt.tight_layout()
plt.show()

Numerical Variables: Location Measures

Example: Location Measures

• The mean is sensitive to outliers, while the median is more robust.
  • What if we add a value of 100?
• The mode can be useful for understanding the most common value, especially in categorical data.
  • Why might this stop being useful with float data?

Numerical Variables: Spread Measures

Variance and Standard Deviation

• The sample variance is computed as: \[s^2 := \frac{1}{n-1} \sum_{i=1}^n (x_i - \bar{x})^2.\]

  • This can be intuitively be thought of as the average squared distance of the data points from the mean.
  • We divide by \(n-1\) instead of \(n\) to get an unbiased estimate of the population variance (Bessel’s correction - discussed further next week).

• The sample standard deviation is the square root of the variance: \[s := \sqrt{s^2}.\]

Example: Sample Variance

Min, Max, Range, and Interquartile Range

• It is often useful to understand the range of values in the data, as well as the spread of the middle 50% of the data.

• The minimum and maximum values provide the range of the data: \[\text{Range} := \max(x_i) - \min(x_i).\]

• A quantile is a cut point that divides a sorted dataset into equal-sized groups.

  • Percentiles are quantiles that divide the data into 100 equal parts.
  • Quartiles are quantiles that divide the data into 4 equal parts.
    • The first quartile (Q1) is the 25th percentile.
    • The second quartile (Q2) is the 50th percentile (the median).
    • The third quartile (Q3) is the 75th percentile.

• The interquartile range (IQR) is the difference between the third and first quartiles: \[\text{IQR} := Q3 - Q1.\]

Numerical Data Summary and Box Plots

Data Summary

• We can use the describe() method in pandas to compute most of the common summary statistics for a numerical variable.
• A box plot (or box-and-whisker plot) is a graphical representation of the summary statistics, showing the median, quartiles, and potential outliers.

titanic['age'].describe()

count    714.000000
mean      29.699118
std       14.526497
min        0.420000
25%       20.125000
50%       28.000000
75%       38.000000
max       80.000000
Name: age, dtype: float64

fig, ax = plt.subplots(figsize=(10, 5))
sns.boxplot(data=titanic, x='age', ax=ax)
ax.set_title('Box Plot of Passenger Age')
plt.tight_layout()
plt.show()

Histograms

Histograms

• Box plots are useful for summarizing the distribution of a numerical variable.
• Histograms are useful for visualizing the shape of the distribution.
  • A histogram divides the range of the data into intervals (bins) and counts the number of observations in each bin.
  • The choice of bin width can affect the appearance of the histogram and our interpretation of the data.

fig, ax = plt.subplots(figsize=(10, 5))
sns.histplot(data=titanic, x='age', bins=20, kde=True, ax=ax)
ax.set_title('Histogram of Passenger Age')
plt.tight_layout()
plt.show()

Histogram with Summary Statistic Overlays

Skewness

Skewness

• Skewness measures the asymmetry of the distribution of a numerical variable.
  • A distribution is positively skewed (right-skewed) if it has a long tail on the right side.
  • A distribution is negatively skewed (left-skewed) if it has a long tail on the left side.
• The skewness can be calculated using the formula: \[\text{Skewness} = \frac{1}{n} \sum_{i=1}^n \left( \frac{x_i - \bar{x}}{s} \right)^3.\]

Kurtosis

• Kurtosis measures the “tailedness” of the distribution of a numerical variable.
  • A distribution is leptokurtic if it has heavy tails and a sharp peak.
  • A distribution is platykurtic if it has light tails and a flat peak.
  • A distribution is mesokurtic if it has tails and a peak similar to the normal distribution.
• The kurtosis can be calculated using the formula: \[\text{Kurtosis} = \frac{1}{n} \sum_{i=1}^n \left( \frac{x_i - \bar{x}}{s} \right)^4.\]

Example: Skewness and Kurtosis

Skewness and Kurtosis Calculation

• There are built-in functions in libraries like scipy.stats to calculate skewness and kurtosis, but we can also implement these calculations manually using the formulas provided.

def calculate_skewness_kurtosis(data):
    n = len(data)
    mean = np.mean(data)
    std_dev = np.std(data, ddof=1)  # Sample standard deviation
    skewness = (1/n) * np.sum(((data - mean) / std_dev) ** 3)
    kurtosis = (1/n) * np.sum(((data - mean) / std_dev) ** 4)
    return skewness, kurtosis

Histogram with Kurtosis and Skewness

skewness, kurtosis = calculate_skewness_kurtosis(titanic['age'].dropna())
print(f'Skewness: {skewness:.2f}, Kurtosis: {kurtosis:.2f}')

Skewness: 0.39, Kurtosis: 3.16

Quick Note on the Normal Distribution

What is the Normal Distribution?

• The normal distribution (or Gaussian distribution) is a continuous probability distribution that is symmetric around its mean, with a bell-shaped curve.
  • We will cover probability distributions in more detail next week but it is helpful to have an idea about the normal distribution when discussing skewness and kurtosis.
• It is defined by its mean (\(\mu\)) and standard deviation (\(\sigma\)).
• Many natural phenomena and measurement errors tend to follow a normal distribution, making it a fundamental concept in statistics.

Bell Curve

Bell Curve Example

Cross Tabulation

What are our aims?

• Find and measure relationships between variables.

pd.crosstab(
  titanic['class'], titanic['survived'], 
  margins=True, normalize='index')

Tabulation of Categorical and Quantitative Variables

What about when one variable is quantitative?

• We can use grouped summary statistics to examine the relationship between a categorical variable and a quantitative variable.
  • For example, we can compute the mean age of passengers in each class in the Titanic dataset.

titanic.groupby('class')['age'].describe()

Covariance

Multiple Quantitative Variables

• For two quantitative variables we are primarily interested in the sample covariance and correlation.

Correlation

Sample Correlation

• The sample correlation between two variables \(X\) and \(Y\) is calculated as: \[r_{XY} := \frac{s_{XY}}{s_X s_Y},\] where \(s_X\) and \(s_Y\) are the sample standard deviations of \(X\) and \(Y\), respectively.
  • The correlation coefficient ranges from -1 to 1.
  • Values close to 1 indicate a strong positive linear relationship.
  • Values close to -1 indicate a strong negative linear relationship.
  • Values close to 0 indicate no linear relationship.

Correlation Visualization Example

Warnings about Correlation

Non-Linear Relationships

Look at the following scatter plot:

• The correlation is nearly 0, but there is clearly a strong relationship.

Covariance and Correlation Matrices

titanic[['age', 'fare']].cov()

titanic[['age', 'fare']].corr()

Checking our Matrices

Let’s look at the scatter plot

fig, ax = plt.subplots(figsize=(10, 5))
sns.scatterplot(data=titanic, x='age', y='fare', ax=ax)
ax.set(
  title='Scatter Plot of Age and Fare',
  xlabel='Age',
  ylabel='Fare'
)
plt.tight_layout()
plt.show()

Example Data: Wine Dataset

• We consider the wine data set provided by Geeks for Geeks in the following note on Exploratory Data Analysis in Python.

wine = pd.read_csv('data/WineQT.csv')
wine.head()

wine.shape

(1143, 13)

Summary Statistics

• We can compute the summary statistics for the wine dataset using the describe method.

wine.describe()

Univariate Analysis

Histograms to inspect the distribution of the variables

for feature in wine.columns:
    fig, ax = plt.subplots(figsize=(10, 5))
    sns.histplot(wine[feature], kde=True, ax=ax)
    plt.title(f"{feature} | Skewness: {round(wine[feature].skew(), 2)}")
    plt.savefig(f"assets/wine_{feature}.png")

plt.tight_layout()
plt.show()

Histogram Plots

← Alcohol Chlorides Citric Acid Density Fixed Acidity Free Sulfur Dioxide Id pH Quality Residual Sugar Sulphates Total Sulfur Dioxide Volatile Acidity →

1 / 13

• Which variables are skewed?

Skewness and Kurtosis

Skewness

wine.skew()

fixed acidity           1.044930
volatile acidity        0.681547
citric acid             0.371561
residual sugar          4.361096
chlorides               6.026360
free sulfur dioxide     1.231261
total sulfur dioxide    1.665766
density                 0.102395
pH                      0.221138
sulphates               2.497266
alcohol                 0.863313
quality                 0.286792
Id                     -0.010419
dtype: float64

wine.kurtosis()

fixed acidity            1.384614
volatile acidity         1.375531
citric acid             -0.714686
residual sugar          27.675366
chlorides               47.078324
free sulfur dioxide      1.932170
total sulfur dioxide     5.098748
density                  0.888123
pH                       0.925791
sulphates               12.017377
alcohol                  0.221179
quality                  0.314664
Id                      -1.216364
dtype: float64

Bivariate Analysis

Pairplots

sns.pairplot(wine.iloc[:, :2])

Quality vs Alcohol

Box Plots

fig, ax = plt.subplots(figsize=(10, 5))
sns.boxplot(x='quality', y='alcohol', data=wine, ax=ax)
ax.set(
  title='Box Plot of Quality and Alcohol',
  xlabel='Quality',
  ylabel='Alcohol'
)
plt.tight_layout()
plt.show()

Multivariate Analysis

Heatmap

plt.figure(figsize=(7, 7))

sns.heatmap(wine.corr(), annot=True, fmt='.2f', cmap='Pastel2', linewidths=2)
plt.title('Correlation Heatmap')
plt.show()

Conclusion

References