Answers

The mathematics underneath

The arithmetic every model is made of, worked through by hand before any of it is handed to a library.

1. What is the difference between variance and standard deviation?They measure the same spread: variance is the average squared distance from the mean, standard deviation is its square root, so you report the standard deviation and calculate with the variance.
2. How do you make a vague claim precise enough to tell whether it's true?Fix what is being talked about, what is asserted, which cases are covered and the cut-off, then name an observation that would prove it false.
3. How do you find the parabola that best fits a function near a point?Match the function's height, slope and bending at the chosen point; those three conditions single out exactly one parabola.
4. Why does a test for a rare disease give so many false positives if the test is accurate?A positive result means the test fired, not that you are likely to be ill; when the disease is rare, healthy people vastly outnumber ill ones, so a small false-alarm rate on them swamps the few true cases.
5. When should I give up on finding an exact formula and simulate instead?Give up on an exact formula when the quantity can be written as an average of something drawable, since simulation then gives an estimate with a measurable error.
6. What's the difference between a state the chain keeps returning to and one it eventually leaves forever?A state the chain keeps returning to is in a closed group it cannot leave; any other state is left forever after finitely many visits.
7. Why does variance square the deviations instead of just averaging how far each value is from the mean?Variance squares deviations because signed deviations cancel to zero, and squares remove the sign while allowing the algebra, scaling and addition rules that absolute distances do not.
8. What does log base 2 of 8 actually mean?It is the exponent you must put on two to get eight, and that exponent is three.
9. What does a negative dot product tell you about two vectors?A negative dot product means the angle between the vectors is greater than a right angle, so they point broadly apart rather than together.
10. Why do eigenvectors stay in the same direction after a transformation?An eigenvector is defined as a vector that a matrix sends to a scaled copy of itself, so staying on its line is the condition that selects it, not a property of ordinary vectors.
11. Why can't cross entropy loss go below the entropy of the labels?Cross entropy cannot fall below the uncertainty already in the labels because it is that uncertainty plus a non-negative distance from the model to the labels.
12. What's the difference between singular values and eigenvalues?Singular values measure stretching along perpendicular directions and exist for every matrix; eigenvalues measure directions a square matrix leaves on their own line.
13. Why does backpropagation never build the full Jacobian?Backpropagation only needs a vector multiplied by each layer's derivative matrix, so it computes that product directly and never builds the matrix.
14. Why doesn't matrix multiplication work like regular multiplication where order doesn't matter?Matrix multiplication means doing one transformation and then another, and reversing that order usually lands a vector somewhere else.
15. How can there be a slope at a single point if slope needs two points?A slope at one point is the single value that slopes between nearby pairs of points settle on as the pair closes up; it is not a slope measured between two identical points.
16. How does the FFT get the same answer without doing all the multiplications?The FFT gets the same spectrum by splitting the readings into even and odd positions and reusing the two half-length results for every rate, so shared partial sums are computed once instead of once per rate.
17. What does expected value actually mean if I never actually get that number?It is a weighted average over all possible outcomes, not a prediction of the outcome you will see; it can be a number no single result ever takes.
18. How can you add up infinitely many infinitely thin slices and get a finite number?You add small positive pieces, then take the finite total they approach as the pieces shrink; the pieces never all become zero before the adding.
19. Why do we use e as the base of natural logarithms instead of 10?The base e is used because it makes growth calculus clean: the derivative of e to the x is itself, while base ten leaves a factor of about 2.3026 in every derivative.
20. Why does multiplying by a sine and summing pick out only that frequency?Multiplying by a sine and adding over whole cycles cancels every other frequency exactly and leaves a positive total for the matching one.
21. Why is 0! equal to 1?Zero factorial is one because there is exactly one way to arrange nothing: the empty arrangement, with nothing left to place.
22. Is f(x) just f times x, or is it something else?No. f(x) means the rule named f applied to x, giving one output value; it is not f multiplied by x.
23. Why is a function thought of as a rule rather than just a formula?A function is the rule that gives exactly one output for each input; a formula is only one way of writing that rule down.
24. Why is the negative gradient the direction of steepest descent?A unit step goes downhill fastest when it points directly opposite the gradient, because the dot product with the gradient is then as negative as it can be.
25. What does a perplexity of 10 actually mean?A perplexity of 10 means the model's average loss is about 2.30 nats per prediction, equivalent to a uniform choice among ten equally likely options, not a 10 percent accuracy.
26. What does temperature scaling actually change in a model's probabilities?It divides every score by one positive number before the softmax, so the probabilities become sharper or flatter while the ranking and top prediction stay the same.
27. Why does softmax followed by cross entropy have gradient p minus y?Because the derivative of the loss through softmax simplifies to the predicted probability minus the target, one number per class, with the exponentials and fractions cancelling out.
28. Why does subtracting the max from the logits not change softmax?Subtracting the largest score is an exact rewriting of softmax, not a change to the distribution: the same factor cancels above and below.
29. Why does the average of many random draws get closer to the expected value as I take more?Averaging divides by the number of draws, so the spread of the average shrinks with the square root of that number; no draw ever compensates for earlier ones.
30. Why does the average of many random samples get closer to the true value as I take more samples?Because the average of independent draws is centred on the true value at every sample size, and its typical distance from that value shrinks like one over the square root of the number of draws.
31. Why do derivatives multiply along a chain but add when the path splits?Along a chain each step scales the effect that arrived, so the scalings multiply; at a split each use contributes its own effect to the same change, so the contributions add.
32. Why do you multiply derivatives when functions are composed?Because a small change at the input is scaled once by the inner function and again by the outer function, so the two scale factors multiply.
33. How does momentum stop the steps from cancelling each other out?Momentum keeps a running total of past gradients, so directions that keep agreeing build up while directions that keep flipping sign cancel out.
34. How can a finite list of samples be turned into frequencies and back with nothing lost?A finite list can be converted into frequencies and back exactly, because the transform is a reversible change of description, not a lossy summary.
35. Why does computing the gradient cost about the same no matter how many parameters the network has?Because one backward sweep shares intermediate derivatives across all parameters, so the whole gradient costs a few forward passes in total rather than one per parameter.
36. What is the difference between the range and the precision of a float32?Range is the span of sizes a float32 can hold before it becomes infinity or zero; precision is how many significant digits it keeps, so nearby values stay distinguishable.
37. Why does cosine similarity ignore how long the vectors are?Cosine similarity compares direction only because dividing by both vector lengths cancels their size exactly, leaving the angle between them.
38. What stays constant when a system just goes around in a loop forever?A fixed combination of the changing counts stays the same round the loop, and that is what keeps the system from settling or escaping.
39. How come lining up only some of the things still involves factorials?Because you multiply only the choices for the places you fill, so a partial line-up is a factorial with the unused tail divided out.
40. Why does the negation of "A and B" become "not A or not B"?Negating 'A and B' asserts that at least one of the two fails, so the and becomes an or.
41. How do I estimate the error of a statistic when I don't have a formula for it?Resample your own data many times, recompute the statistic each time, and use the spread of those results as the error bar.
42. Why does any number to the power zero equal one?Any non-zero number to the power zero is one because no copies are multiplied, and an empty product is one, not zero.
43. What does the sigma notation with i=1 underneath and n on top actually tell me to add?It tells you to write the expression after the sign once for each whole number from one up to the top number, then add all those copies.
44. Why does cutting the error in half require four times as many samples?The error shrinks in proportion to one over the square root of the sample count, so halving it needs four times the samples.
45. Why does normalisation subtract the mean and divide by the standard deviation?Subtracting the average removes the offset and dividing by the spread removes the size, so the output has zero average and unit spread whatever scale the input had.
46. Why do machine learning models use log loss instead of just multiplying probabilities?Multiplying many probabilities underflows to zero on a computer; adding their logarithms keeps the same score and stays representable.
47. Why does cross entropy loss only use the probability of the correct class?Because a definite label puts all the target weight on the correct class and none on the others, so every other term is multiplied by zero.
48. How do you build a random walk whose long-run distribution is a distribution you choose?Propose a symmetric random move; accept moves to heavier target weight outright and moves to lighter weight with probability equal to the weight ratio, then run the chain.
49. Why does Adam need bias correction at the start of training?Adam needs bias correction because its two running averages start at zero, and without correction the early steps are several times larger than the learning rate asks for.
50. Why does gradient descent still work when each step uses only one minibatch's gradient?A minibatch gradient is wrong at every step but correct on average, so the errors cancel over many steps while the true downhill pull keeps adding up.
51. Why do I divide by k! when the order doesn't matter?You divide by the number of ways to order the chosen items, because the ordered count listed every chosen group that many times.
52. What does it mean when a matrix doesn't have full rank?A matrix lacks full rank when it cannot reach as many independent directions as its shape allows, so it loses information and cannot be reversed.
53. How do I estimate something to the nearest order of magnitude?Round every factor to one digit times a power of ten, multiply the digits roughly, add the exponents, and report the nearest power of ten.
54. How does having more items than slots guarantee that two of them share a slot?If you put more items into fewer slots, at least one slot must hold two, because if every slot held at most one you could not fit them all.
55. Why does induction need a base case if the step works?The step only says that if one case holds then the next does; without a first case there is nothing to start from, so no case is proved.
56. Why does a fast wiggle look like a slow one when I sample too slowly?Because readings taken at regular instants cannot tell a rhythm from the same rhythm sped up by a whole number of cycles per reading, so the fast one is reported at a folded, slower rate.
57. Why do the columns of a matrix tell you where the basis vectors go?A matrix lists where each coordinate axis lands, one destination per column, because feeding in a single axis picks out exactly that column.
58. Why does test error go down and then back up as I make the model more complicated?A more flexible model first removes its systematic miss, then starts fitting the noise in its particular training sample, so test error falls and then rises.
59. What does a computation graph actually let you differentiate?It lets you differentiate a final number with respect to any earlier value in the graph by multiplying local derivatives along each route and adding across routes.
60. Why does adding a layer's input to its output make very deep networks easier to train?Because the addition puts a derivative of exactly one on the path through it, so the gradient reaching the early layers is not a long product of small factors and does not shrink with depth.
61. Why does adding a shortcut connection stop gradients from vanishing?A shortcut gives the gradient a route whose factor is one plus the layer's own factor, so many layers in series no longer multiply the signal down to nothing.
62. Why does Manhattan distance give a different answer from straight-line distance?They measure different things: one is the straight gap, the other is the total along a grid, so they disagree whenever the points differ in more than one coordinate.
63. Why do distances between random points become almost equal in high dimensions?Each coordinate adds a small independent amount to the squared distance, and many such amounts average out, so typical distances grow while their spread does not keep up.
64. What is the difference between a half-life and a decay rate?A half-life is the time for an amount to be multiplied by one half, while a decay rate says how fast that multiplication happens, either continuously or as a per-step factor.
65. Why does a Markov chain eventually forget which state it started in?A finite Markov chain forgets its starting point when it has one closed class and no cycle, because two runs from different starts share chance that the rule cannot separate again.
66. What's the difference between a vector as a point and a vector as a direction?A vector as a point is a location, while a vector as a direction is an arrow from the origin whose length and orientation matter; the same numbers can be read either way.
67. Does a subscript like a_ij mean a times i times j?No, a subscript marks which row and column an entry belongs to rather than multiplying by the letters underneath.
68. What does it mean if a system ends up on an attractor?It means the long run is pulled onto a set rather than a single value: a resting point, a repeating loop, or a spiral closing onto one of those.
69. Why does my test set stop being a fair test if I keep tuning on it?Because every time you use the test set to choose something, information from it shapes the model, so it stops being untouched and its score becomes optimistic.
70. Why can a model get 100% on the training set and still be useless?A perfect training score only shows the model can reproduce the examples it was fitted on; it says nothing about new examples, where it may be no better than guessing.
71. Why do I have to subtract the overlap when I add two groups together?Because anything in both groups gets counted twice, so you take one copy back out to leave it counted once.
72. Why does adding a penalty on the weights stop overfitting?Adding a penalty makes large weights costly, so the fit keeps them only when the training data saves enough error to pay for them, which limits the settings the model can reach.
73. Why does sliding a filter along a signal become simple multiplication after a Fourier transform?Because a filter scales every steady rhythm without changing its rate, so the whole sliding sum collapses into one multiplication for each rhythm.
74. Why do gradients become tiny in the early layers of a deep network?Because the backward pass multiplies one number per layer, and a factor slightly below one, repeated over many layers, collapses towards nothing.
75. Why is an if-then statement true when the condition is false?An if-then claim only forbids one case, the condition holding while the result fails, so when the condition is false there is nothing left for it to forbid and it stays true.

The other subjects