Logistic Regression

Updated: 2019-01-13

Sigmoid/Logistic function

g(z)=11+ezg(z) = {1 \over 1+e^{-z}}

Hypothesis

hθ(x)=g(θx)=11+eθxh_\theta(x) = g(\theta \cdot x) = {1 \over 1 + e^{-\theta \cdot x}}
  • y=1y=1 if hθ(x)0.5h_\theta(x) \ge 0.5 (i.e. θx0\theta \cdot x \ge 0)
  • y=0y=0 if hθ(x)<0.5h_\theta(x) < 0.5 (i.e. θx<0\theta \cdot x < 0)

Logistic Regression Cost Function

J(θ)=1m[i=1myiloghθ(xi)+(1yi)log(1hθ(xi))]J(\theta) = - {1 \over m} [\sum_{i=1}^m y_i \log h_\theta(x_i) + (1-y_i) \log(1-h_\theta(x_i))]

Gradient Descent

minθJ(θ)\min_\theta J(\theta)

Repeat: simultaneously update all θj\theta_j

θj:=θjαθjJ(θ)\theta_j:=\theta_j - \alpha {\partial \over \partial \theta_j} J(\theta) θj:=θjαi=1m(hθ(xi)yi)xi\theta_j:=\theta_j - \alpha \sum_{i=1}^m (h_\theta(x_i) - y_i)x_i

Regularization

J(θ)=1m[i=1myiloghθ(xi)+(1yi)log(1hθ(xi))]+λ2mi=1nθj2J(\theta) = - {1 \over m} [\sum_{i=1}^m y_i \log h_{\theta(x_i)} + (1-y_i) \log(1-h_\theta(x_i))] + {\lambda \over 2m} \sum_{i=1}^n \theta_j^2