Basis Lab

Answers

One question, answered the whole way through, with the arithmetic actually done rather than named. Every one of them ends at the page of a course that carries it further, because an answer you cannot build on is a fact you will look up again next month.

300 questions so far, page 3 of 6

How models work

26 to 71 of 71
  1. Why does k-means split one long stretched-out group into two?Because k-means keeps only one centre per group, so a long group's ends end up closer to a second centre and the group is cut across its length.
  2. How do I know if I should increase or decrease the slope to reduce the error?The sign of how fast the loss changes with the slope tells you: positive means lower the slope, negative means raise it.
  3. Why do we need a forward pass before we can backpropagate?The gradient is built from local derivatives evaluated at the values the forward pass computed, so the loss alone cannot supply it.
  4. When should I fine-tune a model instead of just prompting it better?Fine-tune only when the model has been told clearly what you want and still will not comply reliably; missing facts need retrieval, and unstated instructions need a better prompt.
  5. How does masking actually make attention causal?The mask is applied to the scores before they become weights, so forbidden positions end up with exactly zero weight and the rest are renormalised automatically.
  6. Why does attention need positional encoding if it already looks at the whole sequence at once?Attention treats tokens as an unordered set, so position has to be added to the vectors; otherwise a sentence and its anagram get the same representations.
  7. Why does linear regression square the errors instead of just adding their absolute values?Squaring makes large errors count much more than small ones, so it picks a different line from absolute value; absolute value is a different choice about costs, not a mistake.
  8. How can a model remember a sequence without keeping every past input?A model can remember a sequence by keeping a fixed-size block of numbers and rewriting it at every step from its old value and the new input.
  9. How does a gate let information pass through unchanged?A gate lets old information take a direct additive route through the steps, with a learned fraction deciding how much survives each step.
  10. Why is 'hello' a different token from ' hello' with a space in front?A space is a byte that gets merged onto the front of the next word, so the tokenizer stores ' hello' and 'hello' as separate symbols with no shared structure.
  11. What does classifier-free guidance actually add together, and why does it make the prompt stronger?It adds the unconditional prediction to several times the gap between the conditional and unconditional predictions, pushing past the conditional answer so the prompt is obeyed more strictly.
  12. Why is the model's noise prediction really the score of the noised image distribution?The network's noise prediction, divided by the noise level and given a minus sign, is the direction that makes a noised image more plausible.
  13. If you ban certain tokens, does that change what the model would have said?Banning tokens leaves the model's raw scores alone but changes the probabilities you sample from, so the text can change.
  14. Why does a bigger batch make the gradient less noisy?Each step's gradient is an average, and an average's error shrinks with the square root of the number of examples, so four times the batch only halves the noise.
  15. Why do tree ensembles still beat neural networks on ordinary spreadsheet data?Because each spreadsheet column means something on its own, which is exactly what a one-column-at-a-time split needs; where inputs only mean something in combination, networks win by tens of points.
  16. How can I tell whether my clusters are real or just something the algorithm made up?Check whether clusters survive refitting on different rows and whether they predict a held-out column; a convincing 2D plot is weak evidence because such plots can invent clusters.
  17. Why does a model cost the same to run no matter what you ask it?In an ordinary model every weight runs on every token, so the arithmetic per token is fixed by the model size, not by how hard the question is.
  18. How does a decision tree pick the exact number to split a column on?A decision tree sorts the column, tries a cutoff between every adjacent pair of distinct values, scores each one by how mixed the two resulting groups are, and keeps the best.
  19. What is Gini impurity actually measuring?Gini impurity is the chance that two rows picked independently from a group have different outcomes. It is zero for a pure group and largest for an even split.
  20. Why does changing a few rows of the training data give a completely different tree?A tree picks the single best split at each step, so a tiny data change can swap a near-tied split and reshape the whole tree; that is normal, not a bug.
  21. Why does the value of a state have to satisfy a recursive equation with the next state's value?A state's value equals the reward from the next step plus the discounted value of wherever that step lands, because every possible future divides that way and averaging over the first step leaves exactly the next state's value.
  22. Why doesn't temperature scaling change which answer the model picks?Temperature scaling changes how confident the model sounds, not which answer it ranks first, because dividing every score by the same positive number preserves their order.
  23. What is actually stored in the state that a model carries forward?It is a fixed-size block of numbers rewritten at each step from its old value and the new input, not a stored copy of earlier inputs.
  24. When should a model refuse to answer instead of guessing?Refuse when the chance of being wrong times the cost of an error is higher than the cost of handing the case to a person.
  25. Why isn't picking the most likely word at each step the best way to write a sentence?Because the most likely word now can lead to a much worse sentence than a slightly less likely word that sets up a better continuation.
  26. What is wrong with just tokenizing text one character at a time?Nothing is ever unknown, but a passage becomes roughly five times longer than it needs to be, and attention cost grows faster than length, so you pay several times over for the same text.
  27. Why can't a language model just treat each whole word as one token?Because a fixed list of words cannot contain a word nobody has written yet, and enlarging the list never catches up, while the model still has to put a number to every input it is given.
  28. Why do models fail at counting letters or handling numbers and code?Models often fail at letters, digits and code because the text is chopped into pieces before the model sees it, hiding individual characters and place values.
  29. Why do tokenizers split words into smaller pieces instead of keeping whole words?Tokenisers split words into pieces because a fixed list of whole words is enormous and still cannot handle names, typos or new words, while common fragments can build any word without an unknown entry.
  30. Why do so many tokens end up routed to the same few experts?Because picking and training reinforce each other: an expert chosen more often gets trained more, which gets it chosen more often again, so the traffic collapses onto a few.
  31. What does it mean if two words are close together in embedding space?It means training placed their rows in similar positions because the pieces appeared in similar company, so one can often stand in for the other in text; it is not a direct synonym score.
  32. Why can't you just add new tokens to a tokenizer after training?Because each token id is tied to weights the model has already trained, so changing the vocabulary makes old numbers name different pieces while the model keeps the old meanings.
  33. Why does the same word get a different vector depending on the sentence it appears in?The lookup gives the same word the same starting row every time; the different vector is built later by mixing in the words around it.
  34. How does DBSCAN decide that some points are noise and just leave them out?A point is left out when too few neighbours lie close to it and no dense point is close enough to attach it to a group.
  35. What happens to the model if most of the training data comes from one source?The model becomes specialised to that source and loses ground on everything else, because a training mixture is a set of shares that must sum to one.
  36. Why can't I just measure accuracy on data that has no labels?There is no accuracy to measure, because accuracy compares predictions with true answers and unlabelled data has none.
  37. What does a convolution actually calculate at a single point in the image?It multiplies each filter weight by the pixel beneath it, adds those products, and writes the single total at the matching position in the output.
  38. Why does PPO clip the probability ratio?PPO clips the probability ratio so the objective stops rewarding a policy for pushing an action further in the direction it already favours, while still penalising pushes the wrong way.
  39. Why train on a loss function instead of just using accuracy?Because accuracy is a pass or fail count that is flat almost everywhere, so it gives the optimiser no slope to follow, while a loss changes smoothly and tells every weight which way to move.
  40. How do human pairwise comparisons turn into a reward function?Human pairwise preferences become a reward by fitting a scoring model so that preferred responses score higher, then using that score with a penalty for drifting from the starting model.
  41. Why can't I fit temperature on the same data I trained the model on?Because training predictions are already too confident, fitting temperature there confirms that confidence instead of correcting it.
  42. Why did my loss suddenly spike and can I recover without restarting?Your loss spike is probably one bad batch, and you can usually recover by waiting or rolling back to a recent checkpoint instead of restarting the whole run.
  43. Why does a tree that gets every training example right do worse on new data?A tree that fits every training row has memorised individual cases rather than learned a pattern, so its accuracy on new rows falls.
  44. Why does training loss keep falling while validation loss starts rising?Training loss keeps falling because the model is memorising the batches it is shown; held-out loss rises when that memorisation stops helping on data it has not seen.
  45. If it's all just English text, why does where it came from matter?Because the same English words are a record of who was in a position to write them; each source filters for speakers, registers and subjects before you touch it, and no quality filter undoes that.
  46. Can a better model make up for a bad training set?No. A better model can reduce error on what the training set contains, but it cannot supply a skill or fact the set never demonstrates.

Making it fast enough

1 to 4 of 23
  1. Why doesn't breadth-first search give the cheapest path once the links have different costs?Breadth-first search counts links rather than costs, so it can return a route with fewer links that is more expensive than a longer route.
  2. Why is O(2n) treated the same as O(n)?Because it describes how cost grows rather than exact cost, doubling every cost does not change the growth class.
  3. How does a hash function turn a word into an array index?A hash function reads the word as bytes, mixes them into one large number, then takes the remainder after dividing by the table size to get an index from zero to size minus one.
  4. How can you check whether something is in a set without storing every item?Record a few hashed marks per item in a bitmap; all marks set means probably present, any mark clear means certainly absent.

If you want the whole thing

Behind every answer here is a course that works through its subject from the beginning, with problems under each idea and a reason rather than a tick when one is marked. An answer settles the question you came with; the course is what makes the next one easier.

The libraryAsk for a course