Answers · How models work
What is the difference between the inputs and the weights in a linear model?
In shortThe input is supplied by each example and changes from row to row; the weights are the model's adjustable numbers, chosen once by fitting and shared by every example.
In a linear model the input is the number that arrives with the example, while the weights are the numbers the model owns and fitting chooses. In a straight line, multiplies the input and is added to produce the prediction: the same weights are used for every example, and the input changes from one example to the next.
Take a model for house prices with pounds per square metre and pounds. For a house of square metres the arithmetic is , then , so the prediction is pounds. For a second house of square metres, , then pounds. The input changed from 80 to 110 and the prediction changed; the weights did not move. If instead the weights are and , the first house gives and pounds, while the input is still 80. Changing the input asks a different question of the same model. Changing a weight changes the model.
| step | input x | weight w | offset b | prediction | what happened |
|---|---|---|---|---|---|
| 1 | 80 | 3000 | 40000 | 280000 | first house with the original weights |
| 2 | 110 | 3000 | 40000 | 370000 | second house, same weights, larger input |
| 3 | 80 | 2500 | 40000 | 240000 | first house again, same input, changed weight |
The natural guess is that because and sit side by side in the same product, both are given. They are not. If came from the example, every house would have its own slope and there would be nothing left for fitting to choose; the model would not be a model with knobs, it would be a different formula for every row. The input is the part you cannot change when you fit, because it is the data. The weights are the part you do change when you fit, and after fitting they are frozen for all future inputs. That is why has units of pounds per square metre and has units of square metres: the multiplication is not symmetric in meaning even though it is symmetric in arithmetic.
The offset is a weight as well, not a special input. It is what the model predicts when . Setting does not turn into an input: was chosen with during fitting and stays with the model. In a larger linear model the input may be a list of numbers and the weights a matching list, with one extra offset, but the division is the same. Numbers attached to the example are inputs; numbers attached to the formula and chosen by fitting are weights.
The sharing is what makes the weights worth estimating. One slope and one offset must serve every example in the table, so each example casts a vote on the same two numbers. If each example had its own weights, there would be no common pattern to learn and no reason to prefer one pair over another. This is also why a single odd example can move the fitted line: it is not changing its own private weight, it is pulling on the shared weights that every other example also uses. The input stays local to its row; the weights are global to the model.
What fitting does
Fitting is the search over the weights, not over the inputs. The inputs are held fixed as a table of examples, and the fitting rule tries different settings of and , computes predictions, and compares them with the observed outputs. In the house example, the sizes are fixed data; the pair with and and the pair with and are two candidate weight settings, and they produce different sets of predictions from the same inputs. Nothing in that comparison changes a size. A weight setting that survives fitting is still a choice made by the fitting rule, not a fact supplied by the example.
Where the distinction stops being visible
After fitting, the weights are fixed, so at prediction time they look like constants in the formula and the input is the only thing that varies. That can make it seem as if the weights are part of the background and the input is the only real variable. The distinction has not disappeared; it has moved. A weight is still a model parameter because it was chosen by fitting and is shared across examples. If the weights are updated online, they change, but they change through the fitting rule, not because a particular example supplied them. The other limit is the shape of the formula. No setting of and can make a straight line bend, so if the true relationship is flat then steep then flat, fitting will still return two numbers and the errors will be patterned rather than random.
Worth keeping
- An input belongs to the example; a weight belongs to the model.
- Fitting changes weights, not inputs.
- A straight line cannot be made to bend by turning its weights, so a wrong shape stays wrong however well the weights are chosen.
Where this goes next
This question is one step of The Simplest Model That Learns, a course that works through its subject from the beginning. The page that teaches this part of it is A Guess With Knobs On It, and it reads the whole way through whether or not you have an account.
What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.
Read that pageThe whole courseOther questions people ask about this
- What is actually new in a neural network compared with linear regression?Only three things are new: the slopes are computed by working backwards, the score is no longer a bowl, and the model builds its own inputs. The prediction, the loss, the step and the held out check are unchanged.
- Why does logistic regression use a sigmoid instead of a straight line?The sigmoid turns an unbounded straight-line score into a probability between zero and one, and it pairs with the log score so fitting keeps the same gradient shape.
- How do I know if I should increase or decrease the slope to reduce the error?The sign of how fast the loss changes with the slope tells you: positive means lower the slope, negative means raise it.
- Why does linear regression square the errors instead of just adding their absolute values?Squaring makes large errors count much more than small ones, so it picks a different line from absolute value; absolute value is a different choice about costs, not a mistake.
- Why does the model get stuck repeating the same sentence over and over?The model is not stuck in a memory loop: once it has written a phrase, that phrase is in its context and makes writing it again more likely, so a decoder that always picks the top word cannot leave.
- Why does diffusion add Gaussian noise instead of any random corruption?Diffusion adds Gaussian noise because Gaussian errors combine exactly: each step keeps the spread fixed and gives a closed-form noisy picture at any later step, which makes the reverse step learnable.