z-Score (Standard Score)

The $z$-score (or standard score) measures how many standard deviations an observation $x$ lies above or below the population mean $\mu$.

Formula

Population $z$-Score:

$$z = \frac{x - \mu}{\sigma}$$

Where:

• $x$ is the raw value.
• $\mu$ is the population mean: $\mu = \frac{1}{N}\sum_{i=1}^N x_i$.
• $\sigma$ is the population standard deviation: $\sigma = \sqrt{\frac{1}{N}\sum_{i=1}^N (x_i - \mu)^2}$.

Sample $z$-Score:

$$z = \frac{x - \bar{x}}{s}$$

Where $\bar{x}$ is sample mean and $s$ is sample standard deviation (using $n-1$ degrees of freedom).

Key Properties

1. Standardized Distribution: After applying $z$-score transformation, the new distribution has a mean of 0 and a standard deviation of 1.
2. Empirical Rule (68-95-99.7 Rule for Normal Distributions):
   • $\approx 68.27\%$ of values fall within $|z| \le 1$.
   • $\approx 95.45\%$ of values fall within $|z| \le 2$.
   • $\approx 99.73\%$ of values fall within $|z| \le 3$.

Common Use Cases in Machine Learning

1. Feature Standardization

Gradient descent and distance-based algorithms (SVM, KNN, PCA, Logistic Regression, Deep Learning) require normalized feature magnitudes to prevent large-scale features from dominating gradients:

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
X_standardized = scaler.fit_transform(X_train)

2. Statistical Outlier Detection

Points with $|z| > 3$ (or $|z| > 2.5$) are commonly flagged as potential outliers in normally distributed data:

from scipy import stats
import numpy as np

z_scores = np.abs(stats.zscore(data))
outliers = np.where(z_scores > 3.0)

Robust Alternatives (For Heavy Outliers / Skewed Data)

Because mean ($\mu$) and standard deviation ($\sigma$) are sensitive to extreme outliers, standard $z$-scores can be distorted. Robust alternatives include:

1. Modified $z$-Score (using Median Absolute Deviation)

$$z_{\text{modified}} = \frac{0.6745 \times (x - \text{median})}{\text{MAD}}$$

Where $\text{MAD} = \text{median}(|x_i - \text{median}(x)|)$.

2. Robust Scaler (IQR)

$$x_{\text{scaled}} = \frac{x - \text{median}}{Q_3 - Q_1}$$

from sklearn.preprocessing import RobustScaler
scaler = RobustScaler()
X_robust = scaler.fit_transform(X_train)