Why does logistic regression use a sigmoid instead of a straight line?

Logistic regression uses a sigmoid because the straight line underneath it is allowed to produce any real number, including minus four or eleven hundred, and neither can be a probability. The sigmoid maps that whole line into the interval from zero to one, so the output can be read as the probability of yes rather than as a score that happens to look smooth. The straight-line part is left alone; only how its output is read changes.

Check the arithmetic at three points. At $z=0$, $e^0=1$, so $\hat p=1/(1+1)=0.5$. At $z=2$, $e^{-2}$ is about $0.1353$, so $\hat p=1/(1+0.1353)=0.881$. At $z=-2$, $e^2$ is about $7.389$, so $\hat p=1/(1+7.389)=0.119$. The value $0.5$ is the boundary; positive $z$ pushes the probability above a half, negative $z$ below it; larger magnitude means more confidence, but the result never reaches $1$ or $0$ exactly because $e^{-z}$ is never zero and $1+e^{-z}$ is never $1$.

The natural guess is that the sigmoid is a cosmetic bend applied after the real model has done its job. It is not. A straight line cannot be a probability, and clipping a straight line at $0$ and $1$ is not a harmless repair: outside the clipped range the output is exactly $0$ or $1$ and the slope with respect to the knobs is exactly zero, so an example far on the wrong side of the boundary stops pulling the line back. The sigmoid has slope $\hat p(1-\hat p)$, which is small far out but never zero, so every example still contributes a direction to move. The bend is also what makes the raw score $z$ readable as confidence: $z=0$ is balanced, $z=2$ is strong yes, $z=-2$ is strong no.

Keeping the squared score after the squash is legal but bad. It charges at most one for a confident mistake, so saying $0.999$ when the answer is no costs $0.999^2=0.998001$, while saying $0.6$ when the answer is no costs $0.6^2=0.36$; the catastrophic error is less than three times the mild one. The matching score charges the negative logarithm of the probability given to what happened. For the same two mistakes it charges $-\log(0.001)=6.908$ and $-\log(0.4)=0.916$, so the confident mistake is about $7.5$ times the mild one and can grow without limit as the probability goes to zero.

In that expression only one term is ever active. When $y_i=1$ the second term is multiplied by $0$, so the charge is $-\log(\hat p_i)$; when $y_i=0$ the first term is multiplied by $0$, so the charge is $-\log(1-\hat p_i)$. The formula is the surprise at the answer that actually occurred, averaged over examples. It is the score that belongs with a probability output, not an arbitrary choice.

The pairing is what keeps the fitting loop familiar. The sigmoid has derivative $\hat p(1-\hat p)$. The log score is built so that when you differentiate it with respect to $z$, the $\hat p(1-\hat p)$ from the sigmoid cancels against the denominators from the two logs, leaving $\hat p-y$. Then differentiating $z$ with respect to $w_j$ gives $x_{ij}$, so the slope is the average of error times input. Apart from the factor of two that squared error carried, this is the same shape as the straight-line fit.