CHEATSHEETSTOOLSBOOKSHELFABOUT中文

Machine Learning - Mathematical & Theoretical Foundations

Machine Learning (Statistical Learning Theory) provides the mathematical framework for learning functions, patterns, and probability distributions from empirical data.

Unlike traditional algorithmic programming—where explicit deterministic rules are implemented by hand—statistical learning infers an optimal mapping f:X→Yf: \mathcal{X} \to \mathcal{Y} from observations drawn from an unknown joint probability distribution P(x,y)P(\mathbf{x}, y).

The Statistical Learning Framework

Let x∈X⊆RD\mathbf{x} \in \mathcal{X} \subseteq \mathbb{R}^D denote an input feature vector and y∈Yy \in \mathcal{Y} denote the target label.

  • Regression: Continuous output Y=R\mathcal{Y} = \mathbb{R}.
  • Classification: Discrete output Y={C1,…,CK}\mathcal{Y} = \{C_1, \dots, C_K\}.

Empirical Risk Minimization (ERM)

The theoretical goal of a learning algorithm is to find a hypothesis function f∈Hf \in \mathcal{H} that minimizes the Expected Risk (true generalization error) with respect to a loss function L(f(x),y)L(f(\mathbf{x}), y):

R(f)=E(x,y)∼P[L(f(x),y)]=∫X×YL(f(x),y) dP(x,y)R(f) = \mathbb{E}_{(\mathbf{x}, y) \sim P}[L(f(\mathbf{x}), y)] = \int_{\mathcal{X} \times \mathcal{Y}} L(f(\mathbf{x}), y) \, dP(\mathbf{x}, y)

Because the true data-generating distribution P(x,y)P(\mathbf{x}, y) is unknown, the algorithm minimizes the Empirical Risk over a finite training sample D={(x1,y1),…,(xN,yN)}\mathcal{D} = \{(\mathbf{x}_1, y_1), \dots, (\mathbf{x}_N, y_N)\}:

R^N(f)=1N∑i=1NL(f(xi),yi)\hat{R}_N(f) = \frac{1}{N} \sum_{i=1}^N L(f(\mathbf{x}_i), y_i)

To prevent overfitting when hypothesis class H\mathcal{H} has high capacity (high VC-dimension), Structural Risk Minimization (SRM) adds a regularization penalty Ω(f)\Omega(f):

min⁡f∈H{1N∑i=1NL(f(xi),yi)+λΩ(f)}\min_{f \in \mathcal{H}} \left\{ \frac{1}{N} \sum_{i=1}^N L(f(\mathbf{x}_i), y_i) + \lambda \Omega(f) \right\}

Canonical Loss Functions

Loss Function Mathematical Form L(y,y^)L(y, \hat{y}) Primary Application Characteristics
Squared Error Loss 12(y−y^)2\frac{1}{2}(y - \hat{y})^2 OLS Regression, Neural Networks Penalizes outliers quadratically; sensitive to extreme values
Absolute Error Loss ∥y−y^∥\|y - \hat{y}\| Median / Robust Regression Robust against outliers; non-differentiable at zero
Cross-Entropy / Log Loss −∑kykln⁡y^k-\sum_{k} y_k \ln \hat{y}_k Logistic Regression, Softmax Strongly penalizes confident wrong predictions; convex
Hinge Loss max⁡(0,1−yy^)(y∈{−1,+1})\max(0, 1 - y \hat{y}) \quad (y \in \{-1, +1\}) Support Vector Machines (SVM) Zero penalty when margin ≥1\ge 1; induces sparsity in support vectors

Three Inference Paradigms

For classification into KK classes:

  1. Discriminant Functions: Directly maps inputs x→Y\mathbf{x} \to \mathcal{Y} geometrically without intermediate probabilities (e.g., Perceptrons, hard-margin SVM, Fisher Linear Discriminant).
  2. Probabilistic Discriminative: Models conditional distribution P(Y∣x)P(Y \mid \mathbf{x}) directly (e.g., Logistic Regression, GLMs, neural networks with Softmax output).
  3. Probabilistic Generative: Models class-conditional densities P(x∣Y)P(\mathbf{x} \mid Y) and class priors P(Y)P(Y), inverting via Bayes' rule to compute posterior probabilities (e.g., Naive Bayes, Linear Discriminant Analysis, Gaussian Mixture Models).

Core Mathematics & Theory Sitemap