Machine Learning - Mathematical Functions

In statistical learning and neural networks, mathematical transformation functions govern how linear combinations of inputs map to predictions, probabilities, and nonlinear feature representations.

Activation Functions vs. Link Functions

In Generalized Linear Models (GLMs), an input feature vector $\mathbf{x} = [x_1, \dots, x_D]^T$ is combined with weight vector $\mathbf{w}$ and bias $w_0$ into a linear predictor:

$$\eta = \mathbf{w}^T \mathbf{x} + w_0$$

The mathematical relationship between the linear predictor $\eta$ and the conditional expectation of the target variable $\mu = \mathbb{E}[Y | \mathbf{x}]$ is defined through two inverse concepts:

1. Link Function $g(\mu)$: Maps the expected value $\mu$ into the unbounded linear predictor space: $$g(\mu) = \mathbf{w}^T \mathbf{x} + w_0$$
2. Activation Function $f(\eta)$: The inverse of the link function, $f = g^{-1}$, mapping the unbounded linear combination back to the target scale: $$\mu = f(\eta) = f(\mathbf{w}^T \mathbf{x} + w_0)$$

Model	Target Distribution	Link Function $g(\mu)$	Activation Function $f(\eta)$
Linear Regression	Gaussian $\mathcal{N}(\mu, \sigma^2)$	Identity: $g(\mu) = \mu$	Identity: $f(\eta) = \eta$
Logistic Regression	Bernoulli $\text{Bern}(p)$	Logit: $g(\mu) = \ln\left(\frac{\mu}{1 - \mu}\right)$	Logistic Sigmoid: $f(\eta) = \frac{1}{1 + e^{-\eta}}$
Poisson Regression	Poisson $\text{Pois}(\lambda)$	Log link: $g(\mu) = \ln(\mu)$	Exponential: $f(\eta) = e^\eta$

Linear Decision Surfaces

Even when the activation function $f(\cdot)$ is nonlinear (such as sigmoid or step functions), the resulting decision boundary remains a linear hyperplane in feature space:

$$\mathbf{w}^T \mathbf{x} + w_0 = \text{constant}$$

Because $f$ is strictly monotonic, thresholding the probability $P(Y=1|\mathbf{x}) \ge 0.5$ is equivalent to thresholding the linear argument $\mathbf{w}^T \mathbf{x} + w_0 \ge 0$.

Logistic Sigmoid

The logistic sigmoid function maps the real line $\mathbb{R} \to (0, 1)$, interpreting outputs as probabilities:

$$\sigma(z) = \frac{1}{1 + e^{-z}} = \frac{e^z}{1 + e^z}$$

Mathematical Properties

1. Symmetry: $$\sigma(-z) = 1 - \sigma(z)$$
2. Derivative: $$\frac{d\sigma(z)}{dz} = \sigma(z)(1 - \sigma(z))$$ The maximum derivative occurs at $z = 0$, where $\sigma'(0) = 0.25$.
3. Inverse (Logit): $$\sigma^{-1}(p) = \text{logit}(p) = \ln\left(\frac{p}{1 - p}\right)$$
4. Vanishing Gradient: For $|z| \gg 0$, $\sigma'(z) \to 0$, causing backpropagated gradients to diminish across deep layers.

Hyperbolic Tangent ($\tanh$)

The hyperbolic tangent function maps $\mathbb{R} \to (-1, 1)$:

$$\tanh(z) = \frac{e^z - e^{-z}}{e^z + e^{-z}} = \frac{\sinh(z)}{\cosh(z)}$$

Relation to Sigmoid & Advantages

$$\tanh(z) = 2\sigma(2z) - 1$$

• Zero-Centered Output: Outputs have mean close to zero. Unlike sigmoid (whose outputs are strictly positive in $(0, 1)$), zero-centered activations prevent gradient vectors from always pointing in the same direction during backpropagation, improving convergence speed.
• Derivative: $$\frac{d\tanh(z)}{dz} = 1 - \tanh^2(z)$$ Maximum derivative is $1.0$ at $z = 0$ (4x steeper than sigmoid).

Softmax Function

Softmax generalizes the binary logistic function to multi-class classification across $K \ge 2$ mutually exclusive classes:

$$\text{Softmax}(\mathbf{z})_i = \frac{e^{z_i}}{\sum_{j=1}^K e^{z_j}} \quad \text{for } i = 1, \dots, K$$

Properties & Numerical Stability

1. Probability Distribution: $$\sum_{i=1}^K \text{Softmax}(\mathbf{z})_i = 1 \quad \text{and} \quad \text{Softmax}(\mathbf{z})_i \in (0, 1)$$
2. Shift Invariance: For any scalar constant $c \in \mathbb{R}$: $$\frac{e^{z_i - c}}{\sum_j e^{z_j - c}} = \frac{e^{-c} e^{z_i}}{e^{-c} \sum_j e^{z_j}} = \text{Softmax}(\mathbf{z})_i$$ In production implementations, setting $c = \max_j(z_j)$ prevents numerical overflow (NaN from large exponentials).
3. Derivative: $$\frac{\partial \text{Softmax}(\mathbf{z})_i}{\partial z_j} = \text{Softmax}(\mathbf{z})_i (\delta_{ij} - \text{Softmax}(\mathbf{z})_j)$$ where $\delta_{ij}$ is the Kronecker delta (1 if $i = j$, else 0).

Rectified Linear Unit (ReLU) & Smooth Approximations

ReLU

$$\text{ReLU}(x) = \max(0, x)$$

• Computational Efficiency: Evaluated via a single conditional comparison; derivative is either $0$ (for $x < 0$) or $1$ (for $x > 0$).
• No Saturation for $x > 0$: Gradient does not vanish for positive inputs, enabling training of very deep architectures.
• Dying ReLU: Neurons whose pre-activation stays negative across all training examples have zero gradient and permanently deactivate.

Softplus (Smooth ReLU)

$$\text{Softplus}(x) = \ln(1 + e^x)$$

The first derivative of Softplus is analytically identical to the logistic sigmoid:

$$\frac{d}{dx} \text{Softplus}(x) = \frac{e^x}{1 + e^x} = \sigma(x)$$

Leaky ReLU & Parametric ReLU (PReLU)

$$f(x) = \max(\alpha x, x) = \begin{cases} x & \text{if } x \ge 0 \\ \alpha x & \text{if } x < 0 \end{cases}$$

Setting small slope $\alpha > 0$ (typically $\alpha = 0.01$) ensures non-zero gradient everywhere, eliminating the dead neuron issue.