z-Score (Standard Score)

The $z$ -score (or standard score) measures how many standard deviations an observation $x$ lies above or below the population mean $\mu$ .

Formula

Population $z$ -Score:

z = \frac{x - \mu}{\sigma}

Where:

$x$ is the raw value.
$\mu$ is the population mean: $\mu = \frac{1}{N}\sum_{i=1}^N x_i$ .
$\sigma$ is the population standard deviation: $\sigma = \sqrt{\frac{1}{N}\sum_{i=1}^N (x_i - \mu)^2}$ .

Sample $z$ -Score:

z = \frac{x - \bar{x}}{s}

Where $\bar{x}$ is sample mean and $s$ is sample standard deviation (using $n-1$ degrees of freedom).

Key Properties

Standardized Distribution: After applying $z$ -score transformation, the new distribution has a mean of 0 and a standard deviation of 1.
Empirical Rule (68-95-99.7 Rule for Normal Distributions):
- $\approx 68.27\%$ of values fall within $|z| \le 1$ .
- $\approx 95.45\%$ of values fall within $|z| \le 2$ .
- $\approx 99.73\%$ of values fall within $|z| \le 3$ .

Common Use Cases in Machine Learning

1. Feature Standardization

Gradient descent and distance-based algorithms (SVM, KNN, PCA, Logistic Regression, Deep Learning) require normalized feature magnitudes to prevent large-scale features from dominating gradients:

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
X_standardized = scaler.fit_transform(X_train)

2. Statistical Outlier Detection

Points with $|z| > 3$ (or $|z| > 2.5$ ) are commonly flagged as potential outliers in normally distributed data:

from scipy import stats
import numpy as np

z_scores = np.abs(stats.zscore(data))
outliers = np.where(z_scores > 3.0)

Robust Alternatives (For Heavy Outliers / Skewed Data)

Because mean ( $\mu$ ) and standard deviation ( $\sigma$ ) are sensitive to extreme outliers, standard $z$ -scores can be distorted. Robust alternatives include:

1. Modified $z$ -Score (using Median Absolute Deviation)

z_{\text{modified}} = \frac{0.6745 \times (x - \text{median})}{\text{MAD}}

Where $\text{MAD} = \text{median}(|x_i - \text{median}(x)|)$ .

2. Robust Scaler (IQR)

x_{\text{scaled}} = \frac{x - \text{median}}{Q_3 - Q_1}

from sklearn.preprocessing import RobustScaler
scaler = RobustScaler()
X_robust = scaler.fit_transform(X_train)