Machine Learning - Theoretical Tradeoffs & Versus
Understanding statistical learning requires comparing formal mathematical frameworks, objective functions, and structural assumptions.
Three Approaches to Classification
For assigning input vector to one of discrete classes :
1. Discriminant Functions (Geometric)
Directly maps input vector to a class label without estimating intermediate probabilities:
- Examples: Perceptrons, hard-margin Support Vector Machines (SVM), Fisher Linear Discriminant.
- Characteristics: Fast, geometric, but cannot compute predictive uncertainty, risk thresholds, or posterior probabilities for downstream cost-matrix decision making.
2. Probabilistic Discriminative Models
Directly models the posterior class probabilities :
- Examples: Logistic Regression, Generalized Linear Models (GLMs), Multi-layer Perceptrons with cross-entropy loss.
- Characteristics: Focuses model capacity entirely on defining the decision boundary between classes. Does not waste capacity modeling the complex input distribution .
3. Probabilistic Generative Models
Models the class-conditional density and class prior , using Bayes' theorem to compute the posterior:
- Examples: Naive Bayes, Linear Discriminant Analysis (LDA), Gaussian Mixture Models (GMM).
- Characteristics: Allows generating synthetic data samples , detecting out-of-distribution outliers via low marginal , and handling missing features via marginal integration.
L1 Regularization (Lasso) vs. L2 Regularization (Ridge) vs. Elastic Net
Regularization prevents overfitting by adding a penalty on weight magnitudes to the empirical loss function:
| Dimension | L1 Regularization (Lasso, ) | L2 Regularization (Ridge, ) | Elastic Net () |
|---|---|---|---|
| Penalty Term | |||
| Weight Sparsity | Produces exact zeros (); performs feature selection | Shrinks weights toward zero; rarely sets weights to exact zero | Produces sparse models while retaining groups of correlated features |
| Constraint Geometry | Polytope (diamond in 2D) with sharp corners on coordinate axes | Hypersphere (circle in 2D) with uniform curvature everywhere | Convex combination of diamond and sphere |
| Multicollinearity | Arbitrarily selects one feature from a group of correlated features | Distributes weight evenly across correlated features | Selects groups of correlated features together ("grouping effect") |
| Analytic Solution | Non-differentiable at ; solved via coordinate descent | Closed-form: | Solved via iterative coordinate descent |
Matrix Conditioning in Ridge Regression
When features are collinear, the gram matrix becomes singular or ill-conditioned (its minimum eigenvalue approaches zero, causing condition number ). Inverting it amplifies data noise exponentially.
Ridge regularization adds , shifting every eigenvalue by :
This guarantees that the regularized matrix is strictly positive-definite, well-conditioned, and stably invertible.
Fisher's Linear Discriminant vs. Least-Squares Regression
When applied to binary classification (target ):
-
Least-Squares: Minimizes . Points located far from the boundary on the correct side still generate large residual penalties, which distorts the decision boundary.
-
Fisher's Criterion: Maximizes class separation by finding a projection vector that maximizes between-class scatter relative to within-class scatter :
where between-class scatter and within-class scatter .
Solving yields the optimal projection:
Bias vs. Variance Decomposition
For a squared-error loss function, the expected test error decomposes into three additive components:
- Bias (): Error introduced by erroneous assumptions in the learning algorithm (e.g., assuming linear relationship when the truth is polynomial). High bias leads to underfitting.
- Variance (): Sensitivity of the model to small fluctuations in the training set. High variance leads to overfitting.
- Irreducible Error (): Inherent noise in the data generating process that no model can eliminate.
Type I Error vs. Type II Error
| Error Type | Statistical Term | Machine Learning Diagnostic | Example Impact |
|---|---|---|---|
| Type I Error | False Positive () | Rejecting true null hypothesis | Legitimate transaction falsely flagged as fraud |
| Type II Error | False Negative () | Failing to reject false null hypothesis | Actual fraud transaction undetected |
- Significance Level (): .
- Statistical Power (): .

