Machine Learning - Information Theory & Entropy
Information theory provides the mathematical foundation for evaluating uncertainty, loss functions, and probabilistic distance in machine learning.
Shannon Entropy
For a discrete random variable with alphabet and probability mass function :
- Units: If the logarithm base is , entropy is measured in bits (or shannons). If using natural log , it is measured in nats.
- Intuition: Quantifies the average amount of "surprise" or uncertainty in .
- When for some outcome and for all others, (complete certainty).
- For a discrete variable with states, entropy is maximized when outcomes are equiprobable (), yielding .
Binary Entropy Function
For a binary variable with parameter :
The maximum occurs at with . As or , .
Maximum Entropy Principle
The Principle of Maximum Entropy states that the probability distribution that best represents the current state of knowledge is the one with the largest entropy, subject to known constraints:
- Discrete with No Constraints: The Uniform Distribution maximizes entropy over a finite state space.
- Continuous on with Fixed Mean and Variance : The Gaussian (Normal) Distribution is the unique distribution that maximizes differential entropy:
- Continuous on with Fixed Mean : The Exponential Distribution maximizes differential entropy.
Cross-Entropy & Kullback-Leibler (KL) Divergence
Cross-Entropy
Cross-entropy measures the average number of bits needed to encode data drawn from a true distribution using an estimated model distribution :
In machine learning classification, minimizing cross-entropy loss is mathematically equivalent to maximizing the log-likelihood of the training data.
Relative Entropy (Kullback-Leibler Divergence)
The KL Divergence quantifies the statistical divergence or information loss when approximating true distribution with model :
Mathematical Properties
- Non-negativity (Gibbs' Inequality):
- Asymmetry: Because it is asymmetric and violates the triangle inequality, KL divergence is a directed divergence rather than a formal distance metric.
- Relation to Optimization: Because the true distribution entropy is constant with respect to model parameters , minimizing cross-entropy is strictly equivalent to minimizing .
Perplexity
In language modeling and sequence prediction, Perplexity (PPL) measures how well a probability model predicts an observed sample sequence:
- Perplexity can be interpreted as the effective branching factor of the model: a perplexity of means the model is as uncertain as choosing uniformly among possible tokens at each step.
- Lower perplexity indicates superior predictive confidence.

