Exploratory Data Analysis (EDA)

On This Page

• 16.1  What is EDA?
• 16.2  Understanding the Dataset
• 16.3  Cleaning Data
• 16.4  Handling Missing Values
• 16.5  Detecting Outliers
• 16.6  Feature Understanding
• 16.7  Correlation Analysis
• 16.8  Visualization
• 16.9  Complete Case Study — E-Commerce Sales EDA
• 16.10  Deriving Business Insights
• 16.11  Practical Exercises
• 16.12  Module Quiz (MCQs)
• 16.13  Module Assignment
• 16.14  Interview Questions — Module 16
• 16.15  Student Notes Page

Module 16: Exploratory Data Analysis (EDA)

16.1  What is EDA?

Exploratory Data Analysis (EDA) is the process of investigating a dataset before drawing conclusions or building models — understanding its structure, spotting problems, and finding initial patterns. This is where every real data analytics project actually begins.

16.2  Understanding the Dataset

import pandas as pd

df = pd.read_csv("ecommerce_orders.csv")

print(df.shape)         # rows, columns
print(df.head())        # first 5 rows
print(df.info())        # data types, non-null counts
print(df.describe())    # statistical summary

• shape tells you the size of the dataset (rows × columns)
• info() reveals data types and which columns have missing values
• describe() shows min, max, mean, and quartiles for numeric columns at a glance

16.3  Cleaning Data

df.columns = df.columns.str.strip().str.lower()   # clean column names
df = df.drop_duplicates()                          # remove duplicate rows
df['order_date'] = pd.to_datetime(df['order_date'])  # fix data types

16.4  Handling Missing Values

print(df.isnull().sum())

df['discount'] = df['discount'].fillna(0)
df = df.dropna(subset=['customer_id'])   # drop rows missing critical info

16.5  Detecting Outliers

Outliers are unusually extreme values that can distort averages and mislead analysis. Box plots are the standard tool for spotting them visually.

import matplotlib.pyplot as plt

plt.boxplot(df['order_value'], vert=False)
plt.title("Order Value Distribution - Outlier Detection")
plt.show()

Q1 = df['order_value'].quantile(0.25)
Q3 = df['order_value'].quantile(0.75)
IQR = Q3 - Q1
lower = Q1 - 1.5*IQR
upper = Q3 + 1.5*IQR

df_clean = df[(df['order_value'] >= lower) & (df['order_value'] <= upper)]

16.6  Feature Understanding

Before analyzing, classify each column as numeric or categorical, and understand what each represents in the business context.

16.7  Correlation Analysis

print(df[['order_value','discount','quantity']].corr())

A correlation value close to +1 or -1 indicates a strong relationship; values near 0 suggest little to no linear relationship between the two columns.

16.8  Visualization

revenue_by_category = df.groupby('category')['order_value'].sum().sort_values(ascending=False)
revenue_by_category.plot(kind="bar", color="navy")
plt.title("Revenue by Product Category")
plt.show()

16.9  Complete Case Study — E-Commerce Sales EDA

Business Problem: An online store wants to understand what's driving sales and where to focus marketing spend.

import pandas as pd
import matplotlib.pyplot as plt

df = pd.read_csv("ecommerce_orders.csv")

# 1. Understand
print(df.shape, df.info())

# 2. Clean
df = df.drop_duplicates()
df['discount'] = df['discount'].fillna(0)

# 3. Handle Outliers (IQR method)
Q1, Q3 = df['order_value'].quantile([0.25, 0.75])
IQR = Q3 - Q1
df = df[df['order_value'] <= Q3 + 1.5*IQR]

# 4. Analyze
top_category = df.groupby('category')['order_value'].sum().idxmax()
avg_order = df['order_value'].mean()

print("Top Category:", top_category)
print("Average Order Value:", round(avg_order, 2))

16.10  Deriving Business Insights

• Electronics drives the highest revenue — prioritize ad spend and stock for this category.
• Average order value is around Rs.1,200 — bundle offers above this threshold could boost basket size.
• 3 abnormally large orders were flagged as outliers — worth a manual fraud/B2B review.
• Discount and order value show a weak correlation — discounts alone aren't strongly driving bigger orders.

16.11  Practical Exercises

Basic (5 Questions)

1. Load any CSV dataset and print its shape, info(), and describe().

2. Check for missing values in your dataset and count them per column.

3. Remove duplicate rows from a sample dataset.

4. Create a box plot for one numeric column to check for outliers.

5. List the data type (numeric/categorical) of each column in your dataset.

Intermediate (5 Questions)

1. Use the IQR method to detect and remove outliers from a numeric column.

2. Create a correlation matrix for 3+ numeric columns in your dataset.

3. Group your dataset by a categorical column and visualize the totals with a bar chart.

4. Fill missing values in a numeric column using its median instead of 0.

5. Write 3 business insights based on a groupby() analysis of your own dataset.

Advanced (5 Questions)

1. Perform a complete EDA (understand, clean, handle missing values, detect outliers, visualize) on a real public dataset (e.g. Titanic or a sample sales CSV).

2. Build a correlation heatmap and identify the two most strongly correlated variables.

3. Detect and separately analyze outlier records — are they errors, or legitimate rare cases?

4. Create a complete written EDA summary report (1 page) with at least 5 business insights.

5. Compare two subgroups in your dataset (e.g. two regions or two time periods) and explain any meaningful differences you find.

16.12  Module Quiz (MCQs)

16.13  Module Assignment

Assignment: Complete EDA on a Retail Sales Dataset

• Find or create a CSV with at least 100 rows of sales data (date, category, amount, region).
• Perform full EDA: shape/info/describe, missing values, duplicates, and outlier detection.
• Create a correlation matrix and one heatmap.
• Create at least 2 visualizations showing category and regional performance.
• Write a final 5-point business insights summary.

Expected Output: A complete, well-documented EDA notebook or script covering every stage from raw data to clear business recommendations.

16.14  Interview Questions — Module 16

• Q: What is the purpose of EDA in a data analytics project?
  A: EDA helps analysts understand a dataset's structure, quality, and initial patterns before drawing conclusions or building models, reducing the risk of working with flawed data.
• Q: How do you detect outliers in a dataset?
  A: Common methods include visualizing with a box plot, or calculating the IQR (Interquartile Range) and flagging values beyond 1.5×IQR from the first or third quartile.
• Q: Should you always remove outliers?
  A: Not necessarily — outliers should be investigated first, since some represent genuine rare events (like a bulk order) rather than errors, and removing them blindly can hide important information.
• Q: What is the difference between EDA and data cleaning?
  A: Data cleaning is one part of EDA focused on fixing errors and inconsistencies, while EDA as a whole also includes understanding structure, visualizing patterns, and deriving business insights.

16.15  Student Notes Page

Use this space to write down key points, doubts, and your own examples from today's session.