Machine Learning - Model Evaluation & Calibration

Evaluating statistical learning models requires metrics suited to classification thresholds, ranking quality, and probability calibration.

Confusion Matrix & Classification Metrics

For binary classification with positive ($1$) and negative ($0$) classes:

Core Derived Metrics

• Precision (Positive Predictive Value): $$\text{Precision} = \frac{TP}{TP + FP}$$
• Recall (Sensitivity / True Positive Rate): $$\text{Recall} = \text{TPR} = \frac{TP}{TP + FN}$$
• Specificity (True Negative Rate): $$\text{Specificity} = \text{TNR} = \frac{TN}{TN + FP}$$
• False Positive Rate (Fall-out): $$\text{FPR} = 1 - \text{Specificity} = \frac{FP}{TN + FP}$$
• $F_1$ Score (Harmonic Mean): $$F_1 = 2 \cdot \frac{\text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}} = \frac{2 TP}{2 TP + FP + FN}$$
• $F_\beta$ Score: $$F_\beta = (1 + \beta^2) \cdot \frac{\text{Precision} \cdot \text{Recall}}{\beta^2 \cdot \text{Precision} + \text{Recall}}$$ Setting $\beta = 2$ emphasizes recall; setting $\beta = 0.5$ emphasizes precision.

Evaluation Curves

1. Receiver Operating Characteristic (ROC) & AUC

• Axes: x-axis is $\text{FPR}$, y-axis is $\text{TPR}$.
• Properties: Invariant to class imbalance because both $\text{TPR}$ and $\text{FPR}$ evaluate within their respective true class populations.
• Area Under Curve (ROC-AUC): Probability that a randomly chosen positive example is ranked higher than a randomly chosen negative example. Random guessing yields $\text{AUC} = 0.5$.

2. Precision-Recall (PR) Curve

• Axes: x-axis is $\text{Recall}$, y-axis is $\text{Precision}$.
• Properties: Highly sensitive to class imbalance. When negatives heavily outnumber positives (e.g., fraud detection at 0.1% incidence), the PR curve and PR-AUC provide a more realistic assessment than ROC-AUC.

3. Cumulative Gain & Lift

• Lift: Ratio of positive responses found in the top $k\%$ selected population compared to a random sample: $$\text{Lift} = \frac{P(\text{Positive} | \text{Selected})}{P(\text{Positive})}$$

Model Probability Calibration

A model with a high ROC-AUC may still produce output scores that do not reflect true posterior probabilities $P(Y=1|\mathbf{x})$. For example, Gradient Boosted Decision Trees (GBDT) and Support Vector Machines (SVM) frequently produce uncalibrated scores that cluster away from extreme probabilities $0$ and $1$.

A model is well-calibrated if, among samples assigned predicted probability $\hat{p} = 0.80$, exactly 80% belong to the positive class:

Calibration Techniques

1. Platt Scaling (Univariate Logistic Regression)

Fits a scalar sigmoid transformation on raw model scores $s_i$ using an independent validation set:

Parameters $A, B \in \mathbb{R}$ are optimized via maximum likelihood (cross-entropy). Platt scaling is parametric and robust against overfitting on smaller calibration sets.

2. Isotonic Regression

A non-parametric approach that fits a piecewise constant monotonic step function $m(s)$:

Solved via the Pair-Adjacent Violators Algorithm (PAVA). It requires no parametric assumptions but demands larger calibration datasets to avoid overfitting.

3. Piecewise Linear Calibration

Partitions raw predictions into bins and fits monotonic linear slopes within each bin, guaranteeing a continuous, monotonically increasing calibrated curve.

Quantitative Calibration Metrics

• Brier Score: Mean squared error between predicted probabilities $p_i \in [0, 1]$ and actual binary outcomes $y_i \in \{0, 1\}$: $$\text{BS} = \frac{1}{N} \sum_{i=1}^N (p_i - y_i)^2$$ Lower values indicate better calibration and sharpness.
• Expected Calibration Error (ECE): Groups predictions into $M$ equal-width probability bins $B_1, \dots, B_M$ and measures the weighted average discrepancy: $$\text{ECE} = \sum_{m=1}^M \frac{|B_m|}{N} \left| \text{acc}(B_m) - \text{conf}(B_m) \right|$$

Association Rule Metrics

For evaluating co-occurrence in market basket analysis and entity relationships:

• Support: Probability of observing both items $A$ and $B$: $$\text{Support}(A, B) = P(A \cap B)$$
• Confidence (Strength): Conditional probability that item $B$ appears given $A$: $$\text{Confidence}(A \to B) = P(B \mid A) = \frac{P(A \cap B)}{P(A)}$$
• Lift: Ratio of observed joint probability to the expected probability under independence: $$\text{Lift}(A, B) = \frac{P(A \cap B)}{P(A) P(B)}$$
  • $\text{Lift} = 1$: $A$ and $B$ are statistically independent.
  • $\text{Lift} > 1$: Positive association ($A$ and $B$ co-occur more frequently than chance).
  • $\text{Lift} < 1$: Negative association (substitution effect).
• Leverage: Difference between joint probability and independent probability: $$\text{Leverage}(A, B) = P(A \cap B) - P(A) P(B)$$