Logit
Z = b0 + b1*x1 + b2*x2 + ... + bn*xn
The first column of every logistic-regression sheet. Unbounded, and not yet a probability.
- b0
- Intercept, the value of Z when every predictor is zero
- bi
- Change in Z per one unit of predictor i, all else equal
- xi
- The value of predictor i for this row
Sigmoid (logistic function)
P(y=1) = e^Z / (1 + e^Z)
Excel: =EXP(Z)/(1+EXP(Z))
Immediately after Z. Turns the score into a probability. Z = 0 gives 0.5.
- Z
- The logit for this row
- P(y=1)
- Predicted probability that the row belongs to class 1
Likelihood and log-likelihood
Likelihood = IF(y = 1, P, 1 - P)
Objective = SUM( LN(likelihood) ) -> maximise with GRG Nonlinear
The fitting objective. Solver changes only the coefficient cells; every other column recalculates.
- y
- The actual class of the row, 0 or 1
- P
- The row's predicted probability of class 1
- SUM(LN)
- Log-likelihood of the whole training set, always negative
Classification cut-off
predicted class = IF( P(y=1) >= 0.5, 1, 0 )
On the holdout rows, once the coefficients are frozen. Change the 0.5 to trade precision against recall.
- 0.5
- The cut-off, a business decision rather than a statistical one
The five classification metrics
Accuracy = (TP + TN) / (TP + TN + FP + FN)
Precision = TP / (TP + FP)
Recall = TP / (TP + FN) (Sensitivity)
Specificity = TN / (TN + FP)
F1 = 2 x (Precision x Recall) / (Precision + Recall)
On the test rows only, from the four COUNTIFS cells. Choose which one to report from the cost of each error.
- TP
- Actual 1, predicted 1
- FN
- Actual 1, predicted 0. Type II error, loss of opportunity
- FP
- Actual 0, predicted 1. Type I error, loss of resources
- TN
- Actual 0, predicted 0