Percentiles, Data Distributions, Normal Distribution, Scatter Plots & Linear Regression

Before we dive in, let us quickly cover the building blocks you will need.

What is a dataset?

A dataset is just a collection of numbers (or other values). For example:

ages = [12, 15, 18, 22, 30, 45, 60]

That list of seven numbers is a dataset.

What is NumPy?

NumPy is a Python library for working with numbers and arrays. We import it like this:

import numpy as np

np is just a short nickname (alias) for NumPy so we do not have to type numpy every time.

What is Matplotlib?

Matplotlib is a Python library for drawing charts and graphs. We use it like this:

import matplotlib.pyplot as plt
plt.show()

What is SciPy?

SciPy is a Python library for advanced mathematics and statistics. We will use it for the Normal Distribution section.

What Is a Percentile?

Imagine 100 students took a test. If you scored at the 70th percentile, it means you scored higher than 70 out of every 100 students. You are in the top 30%.

🎯 Simple definition: A percentile tells you what percentage of the data falls below a specific value.

Percentiles help answer questions like:

• "Is this salary above average for this city?"
• "Is this patient's blood pressure dangerously high compared to others?"
• "Is this website load time in the slow 10% or the fast 90%?"

The Three Most Common Percentiles

These three together are called the quartiles.

How to Calculate a Percentile in Python

Python's NumPy library has a built-in function:

numpy.percentile(array, percentile_number)

• array = your list of numbers
• percentile_number = which percentile you want (0 to 100)

Example 1 · Simple Percentile Calculation

import numpy as np

ages = [5, 31, 43, 48, 50, 41, 7, 11, 15, 39, 80, 82, 32, 2, 8, 6, 25, 36, 27, 61, 31]

# Find the 75th percentile
result = np.percentile(ages, 75)

print(result)

Expected Output:

43.0

What this means: 75% of the ages in this list are below 43. Only 25% are 43 or older.

Line-by-Line Explanation

import numpy as np

→ Load the NumPy library and give it the short name np.

ages = [5, 31, 43, 48, 50, 41, 7, 11, 15, 39, 80, 82, 32, 2, 8, 6, 25, 36, 27, 61, 31]

→ Create a list of 21 age values. This is our dataset.

result = np.percentile(ages, 75)

→ Ask NumPy: "What value sits at the 75th percentile in this dataset?" NumPy sorts the list internally, then calculates which value 75% of the data falls below.

print(result)

→ Print the answer: 43.0

Example 2 · All Three Quartiles at Once

import numpy as np

ages = [5, 31, 43, 48, 50, 41, 7, 11, 15, 39, 80, 82, 32, 2, 8, 6, 25, 36, 27, 61, 31]

q1  = np.percentile(ages, 25)
q2  = np.percentile(ages, 50)
q3  = np.percentile(ages, 75)

print("25th percentile (Q1):", q1)
print("50th percentile (Q2 / Median):", q2)
print("75th percentile (Q3):", q3)

Expected Output:

25th percentile (Q1): 11.0
50th percentile (Q2 / Median): 31.0
75th percentile (Q3): 43.0

What this means:

• 25% of people are younger than 11
• Half the group is younger than 31 (31 is the middle value)
• 75% of people are younger than 43

🤔 Thinking Prompt

What would happen to the 75th percentile if you added 100 more very old people (ages 90 · 110) to the dataset? Would it go up or down? Why?

Real-World Use of Percentiles

Common Beginner Mistake · Confusing Percentile with Percentage

❌ Wrong thinking: "75th percentile means I scored 75% on the test."

✅ Correct thinking: "75th percentile means I scored higher than 75% of all test-takers."

A person at the 75th percentile may have scored 60% on the test · but most others scored even lower!

What Is Data Distribution?

Distribution describes how data values are spread out across a range.

Think of it like this: if you asked 1,000 people their age, some would be very young, some middle-aged, some old. If you drew a chart of how many people fall into each age group, the shape of that chart is the distribution of your data.

Distribution answers the question: "Where do most of my values cluster, and how spread out are they?"

Two Key Words: Mean and Standard Deviation

Before we generate distributions, you need to know two things:

Mean (Average)

The mean is the sum of all values divided by the count of values.

numbers = [2, 4, 6, 8, 10]
mean = (2 + 4 + 6 + 8 + 10) / 5
# mean = 30 / 5 = 6.0

Standard Deviation

Standard deviation (often written as std or σ, the Greek letter sigma) measures how spread out values are around the mean.

• Small std = values are packed tightly around the mean
• Large std = values are spread far from the mean

# Dataset A: tight cluster
A = [9, 10, 10, 11, 10]   # std ≈ 0.6  (very close together)

# Dataset B: wide spread
B = [2, 5, 10, 15, 18]    # std ≈ 5.8  (far apart)

Generating a Large Random Dataset with NumPy

In real data science, you often need to simulate or generate data for testing. NumPy makes this easy.

numpy.random.normal(loc, scale, size)

Example 3 · Generate a Simple Random Distribution

import numpy as np

# Generate 5 random values with mean=5.0 and std=1.0
data = np.random.normal(5.0, 1.0, 5)

print(data)

Example Output (values will differ each run · random):

[4.73 5.12 6.01 4.88 5.31]

All values are near 5.0, because the mean is 5.0 and the spread (std) is only 1.0.

Example 4 · Generate a Larger Dataset and Plot It

import numpy as np
import matplotlib.pyplot as plt

# Generate 250 random values, mean=5.0, std=1.0
x = np.random.normal(5.0, 1.0, 250)

plt.hist(x, 5)   # Draw a histogram with 5 bars
plt.show()

Expected Output: A histogram (bar chart) showing that most values cluster around 5, with fewer values far from 5.

Line-by-Line Explanation

x = np.random.normal(5.0, 1.0, 250)

→ Generate 250 random numbers. Most will be near 5.0 because that is the mean. The standard deviation 1.0 means most values fall within 1 unit of 5 (so roughly between 4 and 6).

plt.hist(x, 5)

→ Draw a histogram. A histogram is a bar chart where each bar shows how many values fall into that range. The 5 means divide the data into 5 groups (bars).

plt.show()

→ Display the chart on screen.

What Does the Histogram Tell You?

The histogram visually shows the shape of your data:

• The tallest bar is where most values are (near the mean)
• Bars get shorter as you move away from the mean
• With enough data this forms a symmetric bell shape

🤔 Thinking Prompt

What happens if you change 1.0 to 3.0 in np.random.normal(5.0, 3.0, 250)? The std becomes larger · what do you think the histogram will look like now?

(Hint: wider and flatter · more values spread far from 5.0)

Real-World Use of Data Distribution

What Is Normal Distribution?

The Normal Distribution (also called the Gaussian Distribution or Bell Curve) is the most important shape in all of statistics.

It has these key properties:

• It is perfectly symmetrical · left and right sides are mirror images
• The mean, median, and mode are all the same value (the centre)
• It forms a bell shape
• Most values cluster near the mean, fewer values occur far from the mean

🔔 The bell curve is everywhere in nature: human heights, IQ scores, measurement errors, blood pressure, shoe sizes · all follow approximately normal distributions.

The 68-95-99.7 Rule

One of the most powerful properties of the normal distribution:

Example: If the average height of women is 165 cm with a std of 6 cm:

• 68% of women are between 159 · 171 cm (165 ± 6)
• 95% are between 153 · 177 cm (165 ± 12)
• 99.7% are between 147 · 183 cm (165 ± 18)

Generating a Normal Distribution with SciPy

To draw a smooth bell curve, we use SciPy's stats module:

from scipy.stats import norm

norm.pdf(x, mean, std) calculates the height of the bell curve at each x value.

Example 5 · Draw a Simple Bell Curve

import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import norm

# Create 100 evenly spaced x values from -3 to 3
x = np.arange(-3, 3, 0.1)

# Calculate the bell curve height at each x value
# Mean = 0, Standard Deviation = 1
y = norm.pdf(x, 0, 1)

plt.plot(x, y)
plt.title("Standard Normal Distribution")
plt.xlabel("Value")
plt.ylabel("Probability Density")
plt.show()

Expected Output: A smooth bell-shaped curve centred at 0, rising to a peak in the middle and tapering symmetrically toward both sides.

Line-by-Line Explanation

x = np.arange(-3, 3, 0.1)

→ Create a list of x values: -3.0, -2.9, -2.8, ... 2.8, 2.9. These are the positions along the horizontal axis. np.arange(start, stop, step) works like range() but allows decimal steps.

y = norm.pdf(x, 0, 1)

→ For every x value, calculate how tall the bell curve should be at that point.

• 0 = the mean (centre of the bell)
• 1 = the standard deviation (how wide the bell is)
• pdf stands for Probability Density Function · the mathematical formula for the bell curve shape

plt.plot(x, y)

→ Draw a line connecting all the (x, y) points · this creates the smooth curve.

Example 6 · Change the Mean and Standard Deviation

import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import norm

x = np.arange(0, 20, 0.1)

# Three curves: same mean=10, different standard deviations
y1 = norm.pdf(x, 10, 1)   # Narrow bell (std=1)
y2 = norm.pdf(x, 10, 2)   # Medium bell (std=2)
y3 = norm.pdf(x, 10, 3)   # Wide flat bell (std=3)

plt.plot(x, y1, label="std=1")
plt.plot(x, y2, label="std=2")
plt.plot(x, y3, label="std=3")
plt.title("Normal Distributions with Different Std")
plt.legend()
plt.show()

Expected Output: Three bell curves all centred at 10, but with different widths. The std=1 curve is tall and narrow. The std=3 curve is wide and flat.

Key insight: A smaller standard deviation = tighter, taller bell (data is consistent). A larger standard deviation = wider, flatter bell (data is more variable).

Example 7 · Normal Distribution Histogram with Real-Scale Data

import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import norm

# Generate 1000 random heights (cm), mean=165, std=6
heights = np.random.normal(165, 6, 1000)

# Draw histogram
plt.hist(heights, bins=20, density=True, alpha=0.6, color='skyblue', label='Histogram')

# Draw bell curve on top
x = np.arange(140, 190, 0.1)
y = norm.pdf(x, 165, 6)
plt.plot(x, y, color='red', linewidth=2, label='Bell Curve')

plt.title("Simulated Women's Heights")
plt.xlabel("Height (cm)")
plt.ylabel("Density")
plt.legend()
plt.show()

Expected Output: A blue histogram of simulated heights with a smooth red bell curve drawn over the top · both centred near 165 cm.

🤔 Thinking Prompt

If you change the mean from 165 to 180, what will happen to the position of the bell curve? Will it become taller or shorter?

(The curve moves right to centre at 180. Height stays the same because std did not change.)

Real-World Uses of Normal Distribution

Common Beginner Mistake · Confusing pdf with Probability

❌ Wrong: "The output of norm.pdf(x, 0, 1) is the probability of getting exactly x."

✅ Correct: pdf gives the density · the height of the curve. For continuous distributions, the probability of getting exactly one value is technically zero. You calculate probabilities over ranges using the area under the curve.

What Is a Scatter Plot?

A scatter plot is a type of chart where each dot represents one data point with two values: one on the X-axis and one on the Y-axis.

Scatter plots help you answer the question: "Is there a relationship between these two variables?"

For example:

• Does study hours relate to exam score?
• Does engine size relate to fuel consumption?
• Does height relate to weight?

How to Read a Scatter Plot

Example 8 · Your First Scatter Plot

import matplotlib.pyplot as plt

# Study hours
x = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

# Exam scores
y = [45, 52, 58, 65, 70, 73, 79, 85, 88, 95]

plt.scatter(x, y)
plt.title("Study Hours vs Exam Score")
plt.xlabel("Hours Studied")
plt.ylabel("Exam Score")
plt.show()

Expected Output: 10 dots arranged in a clear upward-sloping pattern. Students who study more score higher.

Line-by-Line Explanation

x = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

→ The X-axis values · hours studied (1 through 10).

y = [45, 52, 58, 65, 70, 73, 79, 85, 88, 95]

→ The Y-axis values · exam scores. Note how they increase as x increases.

plt.scatter(x, y)

→ Draw one dot for each (x, y) pair. scatter() is specifically designed for scatter plots.

plt.xlabel("Hours Studied")
plt.ylabel("Exam Score")

→ Label the axes so anyone reading the chart knows what they are looking at.

Example 9 · Scatter Plot with Random Data

This is the W3Schools approach · generating random realistic data:

import numpy as np
import matplotlib.pyplot as plt

# Generate random data for age and speed
x = np.random.normal(0.0, 2.0, 1000)   # X values: 1000 random numbers, mean=0, std=2
y = np.random.normal(0.0, 2.0, 1000)   # Y values: 1000 random numbers, mean=0, std=2

plt.scatter(x, y)
plt.title("Random Scatter Plot (No Relationship)")
plt.show()

Expected Output: 1,000 dots scattered in a roughly circular cloud centred at (0, 0). Since X and Y were generated independently, there is no relationship between them.

Example 10 · Scatter Plot Showing a Strong Relationship

import numpy as np
import matplotlib.pyplot as plt

# Generate x values
x = np.arange(1, 51)   # 1 to 50

# Y is related to x (y ≈ 2x + some noise)
noise = np.random.normal(0, 3, 50)
y = 2 * x + noise

plt.scatter(x, y, color='green')
plt.title("Strong Positive Relationship")
plt.xlabel("X")
plt.ylabel("Y (≈ 2X + noise)")
plt.show()

Expected Output: Dots rise clearly from bottom-left to top-right · a strong positive relationship.

🤔 Thinking Prompt

If you changed y = 2 x + noise to y = -2 x + noise, what direction would the dots go?

(They would go from top-left to bottom-right · a negative relationship.)

Real-World Uses of Scatter Plots

What Is Linear Regression?

Linear regression is the process of drawing a straight line through your scatter plot data in a way that best represents the trend · and then using that line to predict new values.

🏠 Analogy: Imagine you have data on house sizes (in m²) and their prices. Linear regression draws the best straight line through that data. You can then use that line to predict: "If a house is 120m², what price should it have?"

This is one of the most fundamental algorithms in all of machine learning.

The Equation of a Line

Remember from school mathematics: a straight line has the equation:

y = mx + b

Linear regression finds the best values of m (slope) and b (intercept) to fit your data.

What Does "Best Fit" Mean?

The line of best fit is the one that minimises the total error between the actual data points and the line's predicted values.

These errors (vertical distances from each dot to the line) are called residuals. The mathematical method that finds the best line is called Ordinary Least Squares (OLS) · it minimises the sum of squared residuals.

You do not need to calculate this manually · Python does it for you!

How to Do Linear Regression in Python

We use scipy.stats.linregress():

from scipy import stats

slope, intercept, r, p, std_err = stats.linregress(x, y)

Example 11 · Your First Linear Regression

from scipy import stats

# Car age (years)
x = [5, 7, 8, 7, 2, 17, 2, 9, 4, 11, 12, 9, 6]

# Car speed (km/h)
y = [99, 86, 87, 88, 111, 86, 103, 87, 94, 78, 77, 85, 86]

slope, intercept, r, p, std_err = stats.linregress(x, y)

print("Slope:", slope)
print("Intercept:", intercept)
print("R value:", r)

Expected Output:

Slope: -1.7512877115526118
Intercept: 103.52986248931952
R value: -0.7586402890911867

What this tells us:

• Slope = -1.75: For every extra year of age, a car's speed decreases by about 1.75 km/h
• Intercept = 103.5: A brand-new car (age=0) would have a predicted speed of 103.5 km/h
• R = -0.76: A negative value close to -1 indicates a moderate-to-strong negative relationship (older cars tend to be slower)

The R Value (Correlation Coefficient) Explained

The r value (also called Pearson's r) measures how well the line fits the data:

Example 12 · Use the Line to Make a Prediction

Now that we have the slope and intercept, we can predict the speed of any car given its age:

from scipy import stats

x = [5, 7, 8, 7, 2, 17, 2, 9, 4, 11, 12, 9, 6]
y = [99, 86, 87, 88, 111, 86, 103, 87, 94, 78, 77, 85, 86]

slope, intercept, r, p, std_err = stats.linregress(x, y)

# Predict the speed of a 10-year-old car
age = 10
predicted_speed = slope * age + intercept

print(f"Predicted speed for a {age}-year-old car: {predicted_speed:.1f} km/h")

Expected Output:

Predicted speed for a 10-year-old car: 85.5 km/h

How it works: We just applied y = mx + b:

y = (-1.75) × 10 + 103.53
y = -17.5 + 103.53
y = 86.03  (approximately 85.5 km/h)

Example 13 · Full Linear Regression with Scatter Plot and Best-Fit Line

This is the complete, professional workflow combining everything:

import matplotlib.pyplot as plt
from scipy import stats
import numpy as np

# Data: car age vs speed
x = [5, 7, 8, 7, 2, 17, 2, 9, 4, 11, 12, 9, 6]
y = [99, 86, 87, 88, 111, 86, 103, 87, 94, 78, 77, 85, 86]

# Step 1: Calculate the regression line
slope, intercept, r, p, std_err = stats.linregress(x, y)

# Step 2: Create a function to predict y from x
def predict(x_value):
    return slope * x_value + intercept

# Step 3: Generate y values for the regression line
x_line = np.arange(0, 22)          # X values from 0 to 21
y_line = list(map(predict, x_line)) # Predicted Y for each X

# Step 4: Plot scatter + line
plt.scatter(x, y, color='blue', label='Actual data')
plt.plot(x_line, y_line, color='red', label='Best-fit line')
plt.title("Car Age vs Speed — Linear Regression")
plt.xlabel("Car Age (years)")
plt.ylabel("Speed (km/h)")
plt.legend()
plt.show()

print(f"R² = {r**2:.4f}")

Expected Output: A scatter plot of blue dots with a downward-sloping red line through them. The line trends downward from left to right, showing that older cars are slower.

R² = 0.5755

Line-by-Line Explanation

slope, intercept, r, p, std_err = stats.linregress(x, y)

→ Calculate all the regression statistics at once. Python unpacks them into five separate variables.

def predict(x_value):
    return slope * x_value + intercept

→ Define a function that takes any x value and returns the predicted y. This is our line equation y = mx + b as a function.

x_line = np.arange(0, 22)
y_line = list(map(predict, x_line))

→ Create 22 x values (0 through 21). Then use map() to apply predict() to each one, creating the corresponding y values for the line.

plt.scatter(x, y, color='blue', label='Actual data')
plt.plot(x_line, y_line, color='red', label='Best-fit line')

→ Draw the actual data as blue dots, then draw the regression line in red.

print(f"R² = {r**2:.4f}")

→ Print R-squared (R²). This is the square of the R value and tells you what percentage of the variation in Y is explained by X. R² = 0.5755 means 57.55% of speed variation is explained by age. Not perfect, but meaningful.

Understanding R² (R-Squared)

R² (pronounced "R-squared") is one of the most important model quality metrics:

R² = 0.5755 for our car data means: a moderate fit · age explains about 57% of the variation in speed. Other factors (engine condition, brand, maintenance) account for the rest.

🤔 Thinking Prompts

1. If R² = 0.95, does that mean your model will always make perfect predictions?
2. If slope = 0, what does the regression line look like, and what does it mean?
3. If you added more data points far from the line, would R² increase or decrease?

Common Beginner Mistakes in Linear Regression

Mistake 1 · Predicting Far Outside the Data Range

# Our data goes from age 2 to 17.
# This prediction is unreliable:
predicted = predict(100)   # A 100-year-old car?!
print(predicted)  # Output: -71.6  — A negative speed! Nonsense!

✅ Rule: Only predict within or slightly beyond the range of your training data. Predicting far outside is called extrapolation and is often misleading.

Mistake 2 · Confusing Correlation with Causation

❌ "The regression shows age causes lower speed."

✅ Linear regression shows a statistical relationship. Correlation does not prove causation. There may be other factors involved, or the pattern may be coincidental.

Mistake 3 · Using Linear Regression on Non-Linear Data

# If your data curves upward like y = x², a straight line is a poor fit.
# Always plot your data first and visually check if a straight line is appropriate.