Machine Learning - Theoretical Tradeoffs & Versus

Understanding statistical learning requires comparing formal mathematical frameworks, objective functions, and structural assumptions.

Three Approaches to Classification

For assigning input vector x∈RD\mathbf{x} \in \mathbb{R}^D to one of KK discrete classes C1,…,CKC_1, \dots, C_K:

1. Discriminant Functions (Geometric)

Directly maps input vector x\mathbf{x} to a class label without estimating intermediate probabilities:

y=f(x)y = f(\mathbf{x})
  • Examples: Perceptrons, hard-margin Support Vector Machines (SVM), Fisher Linear Discriminant.
  • Characteristics: Fast, geometric, but cannot compute predictive uncertainty, risk thresholds, or posterior probabilities for downstream cost-matrix decision making.

2. Probabilistic Discriminative Models

Directly models the posterior class probabilities P(Ck∣x)P(C_k \mid \mathbf{x}):

P(Ck∣x)=f(wTx+w0)P(C_k \mid \mathbf{x}) = f(\mathbf{w}^T \mathbf{x} + w_0)
  • Examples: Logistic Regression, Generalized Linear Models (GLMs), Multi-layer Perceptrons with cross-entropy loss.
  • Characteristics: Focuses model capacity entirely on defining the decision boundary between classes. Does not waste capacity modeling the complex input distribution P(x)P(\mathbf{x}).

3. Probabilistic Generative Models

Models the class-conditional density P(x∣Ck)P(\mathbf{x} \mid C_k) and class prior P(Ck)P(C_k), using Bayes' theorem to compute the posterior:

P(Ck∣x)=P(x∣Ck)P(Ck)∑j=1KP(x∣Cj)P(Cj)P(C_k \mid \mathbf{x}) = \frac{P(\mathbf{x} \mid C_k) P(C_k)}{\sum_{j=1}^K P(\mathbf{x} \mid C_j) P(C_j)}
  • Examples: Naive Bayes, Linear Discriminant Analysis (LDA), Gaussian Mixture Models (GMM).
  • Characteristics: Allows generating synthetic data samples x∼P(x∣Ck)\mathbf{x} \sim P(\mathbf{x} \mid C_k), detecting out-of-distribution outliers via low marginal P(x)P(\mathbf{x}), and handling missing features via marginal integration.

L1 Regularization (Lasso) vs. L2 Regularization (Ridge) vs. Elastic Net

Regularization prevents overfitting by adding a penalty on weight magnitudes to the empirical loss function:

Lreg(w)=12N∥y−Xw∥22+λ1∥w∥1+λ22∥w∥22\mathcal{L}_{\text{reg}}(\mathbf{w}) = \frac{1}{2N} \|\mathbf{y} - \mathbf{X}\mathbf{w}\|_2^2 + \lambda_1 \|\mathbf{w}\|_1 + \frac{\lambda_2}{2} \|\mathbf{w}\|_2^2
Dimension L1 Regularization (Lasso, λ2=0\lambda_2 = 0) L2 Regularization (Ridge, λ1=0\lambda_1 = 0) Elastic Net (λ1>0,λ2>0\lambda_1 > 0, \lambda_2 > 0)
Penalty Term λ1∑j=1D∥wj∥\lambda_1 \sum_{j=1}^D \|w_j\| λ22∑j=1Dwj2\frac{\lambda_2}{2} \sum_{j=1}^D w_j^2 λ1∑∥wj∥+λ22∑wj2\lambda_1 \sum \|w_j\| + \frac{\lambda_2}{2} \sum w_j^2
Weight Sparsity Produces exact zeros (wj=0w_j = 0); performs feature selection Shrinks weights toward zero; rarely sets weights to exact zero Produces sparse models while retaining groups of correlated features
Constraint Geometry Polytope (diamond in 2D) with sharp corners on coordinate axes Hypersphere (circle in 2D) with uniform curvature everywhere Convex combination of diamond and sphere
Multicollinearity Arbitrarily selects one feature from a group of correlated features Distributes weight evenly across correlated features Selects groups of correlated features together ("grouping effect")
Analytic Solution Non-differentiable at 00; solved via coordinate descent Closed-form: w=(XTX+λ2I)−1XTy\mathbf{w} = (\mathbf{X}^T \mathbf{X} + \lambda_2 \mathbf{I})^{-1} \mathbf{X}^T \mathbf{y} Solved via iterative coordinate descent

Matrix Conditioning in Ridge Regression

When features are collinear, the gram matrix XTX\mathbf{X}^T \mathbf{X} becomes singular or ill-conditioned (its minimum eigenvalue approaches zero, causing condition number κ(XTX)→∞\kappa(\mathbf{X}^T \mathbf{X}) \to \infty). Inverting it amplifies data noise exponentially.

Ridge regularization adds λ2I\lambda_2 \mathbf{I}, shifting every eigenvalue by λ2>0\lambda_2 > 0:

λmin⁡(XTX+λ2I)=λmin⁡(XTX)+λ2≥λ2>0\lambda_{\min}(\mathbf{X}^T \mathbf{X} + \lambda_2 \mathbf{I}) = \lambda_{\min}(\mathbf{X}^T \mathbf{X}) + \lambda_2 \ge \lambda_2 > 0

This guarantees that the regularized matrix is strictly positive-definite, well-conditioned, and stably invertible.

Fisher's Linear Discriminant vs. Least-Squares Regression

When applied to binary classification (target y∈{−1,+1}y \in \{-1, +1\}):

  • Least-Squares: Minimizes ∑i=1N(wTxi+w0−yi)2\sum_{i=1}^N (\mathbf{w}^T \mathbf{x}_i + w_0 - y_i)^2. Points located far from the boundary on the correct side still generate large residual penalties, which distorts the decision boundary.

  • Fisher's Criterion: Maximizes class separation by finding a projection vector w\mathbf{w} that maximizes between-class scatter SB\mathbf{S}_B relative to within-class scatter SW\mathbf{S}_W:

    J(w)=wTSBwwTSWwJ(\mathbf{w}) = \frac{\mathbf{w}^T \mathbf{S}_B \mathbf{w}}{\mathbf{w}^T \mathbf{S}_W \mathbf{w}}

    where between-class scatter SB=(m1−m2)(m1−m2)T\mathbf{S}_B = (\mathbf{m}_1 - \mathbf{m}_2)(\mathbf{m}_1 - \mathbf{m}_2)^T and within-class scatter SW=∑n∈C1(xn−m1)(xn−m1)T+∑n∈C2(xn−m2)(xn−m2)T\mathbf{S}_W = \sum_{n \in \mathcal{C}_1} (\mathbf{x}_n - \mathbf{m}_1)(\mathbf{x}_n - \mathbf{m}_1)^T + \sum_{n \in \mathcal{C}_2} (\mathbf{x}_n - \mathbf{m}_2)(\mathbf{x}_n - \mathbf{m}_2)^T.

    Solving ∇wJ(w)=0\nabla_{\mathbf{w}} J(\mathbf{w}) = 0 yields the optimal projection:

    w∝SW−1(m1−m2)\mathbf{w} \propto \mathbf{S}_W^{-1} (\mathbf{m}_1 - \mathbf{m}_2)

Bias vs. Variance Decomposition

For a squared-error loss function, the expected test error decomposes into three additive components:

E[(Y−f^(x))2]=Bias[f^(x)]2+Var[f^(x)]+σ2\mathbb{E}[(Y - \hat{f}(\mathbf{x}))^2] = \text{Bias}[\hat{f}(\mathbf{x})]^2 + \text{Var}[\hat{f}(\mathbf{x})] + \sigma^2
  1. Bias (Bias[f^(x)]=E[f^(x)]−f(x)\text{Bias}[\hat{f}(\mathbf{x})] = \mathbb{E}[\hat{f}(\mathbf{x})] - f(\mathbf{x})): Error introduced by erroneous assumptions in the learning algorithm (e.g., assuming linear relationship when the truth is polynomial). High bias leads to underfitting.
  2. Variance (Var[f^(x)]=E[(f^(x)−E[f^(x)])2]\text{Var}[\hat{f}(\mathbf{x})] = \mathbb{E}[(\hat{f}(\mathbf{x}) - \mathbb{E}[\hat{f}(\mathbf{x})])^2]): Sensitivity of the model to small fluctuations in the training set. High variance leads to overfitting.
  3. Irreducible Error (σ2\sigma^2): Inherent noise in the data generating process that no model can eliminate.

Type I Error vs. Type II Error

Error Type Statistical Term Machine Learning Diagnostic Example Impact
Type I Error False Positive (α\alpha) Rejecting true null hypothesis H0H_0 Legitimate transaction falsely flagged as fraud
Type II Error False Negative (β\beta) Failing to reject false null hypothesis H0H_0 Actual fraud transaction undetected
  • Significance Level (α\alpha): P(Type I Error)P(\text{Type I Error}).
  • Statistical Power (1−β1 - \beta): P(Reject H0∣H0 is false)=Recall=TPRP(\text{Reject } H_0 \mid H_0 \text{ is false}) = \text{Recall} = \text{TPR}.