One Sweep Back Instead of Seven Chains

Seven derivatives came out of the two unit network, one for each of its seven

parameters, and each was written as its own chain of factors. Laid out side by

side those chains repeat themselves. Two numbers appear over and over, and they

are not cheap numbers: one of them contains a squashing function and its

derivative.

That repetition is not an accident of a network with two hidden units. It is the

shape of every network, and deciding to pay for it once is the whole of the

method this lesson names.

The model is the one from the previous two lessons. The input is scaled as

$u = x/12$, two hidden units squash a line each, and the prediction is a

weighted sum of the two unit outputs plus an offset:

$$\hat{y} = v_1 t_1 + v_2 t_2 + c, \qquad t_j = \tanh(w_j u + b_j)$$

Here $w_j$ and $b_j$ are the weight and offset inside unit $j$, $t_j$ is that

unit's output, $v_j$ is the weight the prediction puts on it, and $c$ is the

prediction's own offset. That is seven parameters. For one measurement with

input $x$ and value $y$ the contribution to the total squared error is

$E = (\hat{y} - y)^2$, and $r = \hat{y} - y$ is the miss.

The arithmetic is done at $x = 4$, where $y = 15$ and $u = 4/12 = 1/3$, with the

starting weights $w_1 = 1$, $b_1 = 0$, $v_1 = 20$, $w_2 = -2$, $b_2 = 0.5$,

$v_2 = 10$ and $c = 15$. The forward pass gives

$a_1 = 1/3 + 0 = 0.333333$ and $t_1 = \tanh(0.333333) = 0.321513$, then

$a_2 = -2/3 + 0.5 = -0.166667$ and $t_2 = \tanh(-0.166667) = -0.165140$, and so

$$\hat{y} = 20(0.321513) + 10(-0.165140) + 15 = 6.430255 - 1.651404 + 15 = 19.778851$$

The miss is $r = 19.778851 - 15 = 4.778851$ and the contribution to the error is

$r^2 = 22.837413$.