Basis Lab

Answers

One question, answered the whole way through, with the arithmetic actually done rather than named. Every one of them ends at the page of a course that carries it further, because an answer you cannot build on is a fact you will look up again next month.

300 questions so far, page 2 of 6

The mathematics underneath

51 to 75 of 75
  1. Why do I divide by k! when the order doesn't matter?You divide by the number of ways to order the chosen items, because the ordered count listed every chosen group that many times.
  2. What does it mean when a matrix doesn't have full rank?A matrix lacks full rank when it cannot reach as many independent directions as its shape allows, so it loses information and cannot be reversed.
  3. How do I estimate something to the nearest order of magnitude?Round every factor to one digit times a power of ten, multiply the digits roughly, add the exponents, and report the nearest power of ten.
  4. How does having more items than slots guarantee that two of them share a slot?If you put more items into fewer slots, at least one slot must hold two, because if every slot held at most one you could not fit them all.
  5. Why does induction need a base case if the step works?The step only says that if one case holds then the next does; without a first case there is nothing to start from, so no case is proved.
  6. Why does a fast wiggle look like a slow one when I sample too slowly?Because readings taken at regular instants cannot tell a rhythm from the same rhythm sped up by a whole number of cycles per reading, so the fast one is reported at a folded, slower rate.
  7. Why do the columns of a matrix tell you where the basis vectors go?A matrix lists where each coordinate axis lands, one destination per column, because feeding in a single axis picks out exactly that column.
  8. Why does test error go down and then back up as I make the model more complicated?A more flexible model first removes its systematic miss, then starts fitting the noise in its particular training sample, so test error falls and then rises.
  9. What does a computation graph actually let you differentiate?It lets you differentiate a final number with respect to any earlier value in the graph by multiplying local derivatives along each route and adding across routes.
  10. Why does adding a layer's input to its output make very deep networks easier to train?Because the addition puts a derivative of exactly one on the path through it, so the gradient reaching the early layers is not a long product of small factors and does not shrink with depth.
  11. Why does adding a shortcut connection stop gradients from vanishing?A shortcut gives the gradient a route whose factor is one plus the layer's own factor, so many layers in series no longer multiply the signal down to nothing.
  12. Why does Manhattan distance give a different answer from straight-line distance?They measure different things: one is the straight gap, the other is the total along a grid, so they disagree whenever the points differ in more than one coordinate.
  13. Why do distances between random points become almost equal in high dimensions?Each coordinate adds a small independent amount to the squared distance, and many such amounts average out, so typical distances grow while their spread does not keep up.
  14. What is the difference between a half-life and a decay rate?A half-life is the time for an amount to be multiplied by one half, while a decay rate says how fast that multiplication happens, either continuously or as a per-step factor.
  15. Why does a Markov chain eventually forget which state it started in?A finite Markov chain forgets its starting point when it has one closed class and no cycle, because two runs from different starts share chance that the rule cannot separate again.
  16. What's the difference between a vector as a point and a vector as a direction?A vector as a point is a location, while a vector as a direction is an arrow from the origin whose length and orientation matter; the same numbers can be read either way.
  17. Does a subscript like a_ij mean a times i times j?No, a subscript marks which row and column an entry belongs to rather than multiplying by the letters underneath.
  18. What does it mean if a system ends up on an attractor?It means the long run is pulled onto a set rather than a single value: a resting point, a repeating loop, or a spiral closing onto one of those.
  19. Why does my test set stop being a fair test if I keep tuning on it?Because every time you use the test set to choose something, information from it shapes the model, so it stops being untouched and its score becomes optimistic.
  20. Why can a model get 100% on the training set and still be useless?A perfect training score only shows the model can reproduce the examples it was fitted on; it says nothing about new examples, where it may be no better than guessing.
  21. Why do I have to subtract the overlap when I add two groups together?Because anything in both groups gets counted twice, so you take one copy back out to leave it counted once.
  22. Why does adding a penalty on the weights stop overfitting?Adding a penalty makes large weights costly, so the fit keeps them only when the training data saves enough error to pay for them, which limits the settings the model can reach.
  23. Why does sliding a filter along a signal become simple multiplication after a Fourier transform?Because a filter scales every steady rhythm without changing its rate, so the whole sliding sum collapses into one multiplication for each rhythm.
  24. Why do gradients become tiny in the early layers of a deep network?Because the backward pass multiplies one number per layer, and a factor slightly below one, repeated over many layers, collapses towards nothing.
  25. Why is an if-then statement true when the condition is false?An if-then claim only forbids one case, the condition holding while the result fails, so when the condition is false there is nothing left for it to forbid and it stays true.

How models work

1 to 25 of 71
  1. Why does the model get stuck repeating the same sentence over and over?The model is not stuck in a memory loop: once it has written a phrase, that phrase is in its context and makes writing it again more likely, so a decoder that always picks the top word cannot leave.
  2. Why does diffusion add Gaussian noise instead of any random corruption?Diffusion adds Gaussian noise because Gaussian errors combine exactly: each step keeps the spread fixed and gives a closed-form noisy picture at any later step, which makes the reverse step learnable.
  3. Why does the same request cost more tokens in some languages than in others?A token is learned from a corpus, so a language that was rare in the training text is split into many more pieces for the same meaning, and the price multiplier follows.
  4. How do you turn the predicted noise into the mean of the previous image?Subtract a small scheduled fraction of the predicted noise from the current image, then rescale to undo the forward step's shrinkage.
  5. Why does the forward noising process have to be a Markov chain?The forward process is made memoryless so that each step is a fixed random rule and any later state can be computed straight from the original picture.
  6. Does it matter which layers I attach adapters to?Yes. For the same number of trainable parameters, adapters spread over more weight matrices, including the feed-forward ones, usually score higher than adapters concentrated on a few.
  7. What do multiple attention heads actually give you that one big head doesn't?Several heads let a model track several relationships at once, each in its own lower-dimensional subspace, instead of squeezing every relationship into one weighted average that destroys information.
  8. Why does a transformer need a cache that grows with sequence length?A transformer stores the key and value for every token it has already seen so later tokens can look back at them, and that store grows with the conversation.
  9. Why do we divide the attention scores by the square root of the key dimension?The square root of the key dimension is the typical size of an unscaled attention score, so dividing by it stops softmax from becoming a hard pick-one.
  10. Why is the REINFORCE gradient so noisy even though it is unbiased?Because unbiased only fixes the average, while one noisy return is multiplied by every action's score, so the estimate's spread grows with episode length and reward scale.
  11. Why do tokenizers work on bytes instead of on characters?Tokenizers use bytes because 256 byte values can represent any text, while Unicode has too many characters to include safely.
  12. What is actually new in a neural network compared with linear regression?Only three things are new: the slopes are computed by working backwards, the score is no longer a bowl, and the model builds its own inputs. The prediction, the loss, the step and the held out check are unchanged.
  13. How many principal components should I keep?There is no single correct number: keep the smallest number whose combined variation you can afford to lose, and report the share you discarded.
  14. Why can a completely made-up answer come with high confidence?Because the confidence score measures how likely the words are, and a fluent invention is built from likely words, not because the fact was checked.
  15. What is the KL penalty against the original policy actually protecting against?It stops the policy from chasing high scores in parts of the answer space where the fitted reward is only guessing, by charging it for moving away from the model it started from.
  16. Why can a linear recurrence be computed as a convolution?Because a linear recurrence adds up past inputs with fixed decaying weights, and that weighted sum is a convolution with the recurrence's impulse response.
  17. What do you give up when you pool or downsample a feature map?You give up exact position: each block of the map becomes one number, so later layers see a coarser grid, gain reach, and lose the ability to say where a response came from.
  18. What does it mean to use stride 2 in a convolution?Using stride 2 keeps the filter the same but moves its window two positions at a time, so it produces about half as many outputs in each direction and about a quarter of the work in two dimensions.
  19. Why do CNNs use the same weights everywhere instead of learning a separate filter for each spot?They share one small filter across every position because a pattern such as an edge means the same thing wherever it appears; the huge cut in parameters is a consequence, not the main reason.
  20. Why does a convolution make the image smaller if you don't pad it?A convolution shrinks the image because the window must fit completely inside it, so the first and last few positions have no full window. Padding adds extra values at the border so those positions can be used.
  21. Why does logistic regression use a sigmoid instead of a straight line?The sigmoid turns an unbounded straight-line score into a probability between zero and one, and it pairs with the log score so fitting keeps the same gradient shape.
  22. What is the difference between the inputs and the weights in a linear model?The input is supplied by each example and changes from row to row; the weights are the model's adjustable numbers, chosen once by fitting and shared by every example.
  23. What does temperature actually do to the next-word probabilities?It divides the model's raw scores by a constant before they are turned into probabilities, so every ratio between two candidate words is pushed further apart or pulled closer together.
  24. When a model says it is 90% sure, what is that 90% a probability of?It is the share of the model's score mass given to that option out of the options it was offered, not the chance that the answer is correct.
  25. Why does attention use softmax instead of just dividing each score by the sum of all the scores?Dividing by the sum can give negative weights and fails when scores cancel; exponentiating first makes weights positive and turns score gaps into ratios.

If you want the whole thing

Behind every answer here is a course that works through its subject from the beginning, with problems under each idea and a reason rather than a tick when one is marked. An answer settles the question you came with; the course is what makes the next one easier.

The libraryAsk for a course