The Two Equations That Pin Down the Best Line

Last timeThe Line and the Size of Its Mistake

Setting both partial derivatives of the squared total to zero gives two linear equations in the slope and the intercept, 78w + 12b = 258 and 650w + 78b = 2106, solved here by hand to w = 3 and b = 2.

A score that ranks two candidate lines does not produce the best one. Twelve

measurements, $y_i$ taken at the inputs $x_i$ for $i$ from 1 to $n = 12$ , fix

the total squared error

E(w, b) = \sum_{i=1}^{12} \big(y_i - (w x_i + b)\big)^2

as a function of the slope $w$ and the intercept $b$ , and the task is to find

the pair that makes it smallest out of a continuum of pairs. This is the one

place in the whole chain where that can be done exactly, by solving rather than

by searching, and it is worth seeing exactly why it works, because everything

later is a response to this argument failing.

The score as a function of two numbers

Expand one squared residual before adding anything up. Writing the residual as

$y_i - w x_i - b$ and squaring gives

The lesson stops here

14 more paragraphs to go

You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.

The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.

See the plan The contents

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents

Read alongside

bookAn Introduction to Statistical LearningGareth James, Daniela Witten, Trevor Hastie, Robert Tibshirani, 2021Section 3.1.1 minimises the residual sum of squares for a straight line and gives the closed form for the slope and the intercept in terms of the means, which is the formula checked against 429 over 143 here.statlearning.com

bookThe Elements of Statistical LearningTrevor Hastie, Robert Tibshirani, Jerome Friedman, 2009Section 3.2 derives the same equations in matrix form and reads the solution as the orthogonal projection of the measurements onto the space the model can reach, which is the geometry behind the two residual conditions.hastie.su.domains

bookConvex OptimizationStephen Boyd, Lieven Vandenberghe, 2004Section 3.1.4 gives the second-order condition for convexity and section 4.2.3 states that a vanishing gradient is both necessary and sufficient for a differentiable convex function, which is the step from a level surface to an actual minimum.web.stanford.edu