Basis Lab

Answers

One question, answered the whole way through, with the arithmetic actually done rather than named. Every one of them ends at the page of a course that carries it further, because an answer you cannot build on is a fact you will look up again next month.

300 questions so far, page 5 of 6

Putting a model into service

22 to 39 of 39
  1. How does a document the model retrieved end up giving it instructions?A retrieved document gives instructions because its text is pasted into the same prompt as your instructions, so the model cannot tell data from command.
  2. What's the difference between a model that refuses because of how it was trained and one that just gets blocked by a filter sitting in front of it?A trained refusal is a habit in the model's weights that an unusual framing can slip past; a filter is separate software that reads the text as data, so no wording reaches it.
  3. How do I tell whether a slow request spent its time waiting in line or actually being worked on?Time a request spent waiting is the total measured where it arrived minus the time anything was actually computing for it; whatever is left was queue.
  4. Why does promising that 99% of requests are fast cost so much more capacity than promising the average is fast?A promise about the slowest requests forces you to keep machines idle so queues never build, while an average promise can be met by making ordinary requests slightly faster.
  5. What do I actually lose by storing a model's weights in 8 bits instead of 16?Half the memory, and twice the speed for anything that was waiting on memory rather than arithmetic, at the cost of coarser stored numbers and a few values that no longer fit the format at all.
  6. What do the scale and zero-point actually do in 4-bit quantization?The scale sets the spacing between the sixteen levels, and the zero-point slides the whole grid so it lines up with the weights you actually have.
  7. Does a benchmark score tell you how good a model really is, or just how it did on those questions?No: a benchmark score is an average over chosen questions and a chosen marking rule, so it describes that test more than the model in general.
  8. Should my evaluation cases be real user queries or ones I write myself?Use real user queries, sampled by behaviour rather than at random, plus a held-out part nobody tunes on. Cases you invent measure what you imagined users ask.
  9. How many cases do I need before the evaluation can even see a small improvement?About four hundred cases per set can see only a six-point move; a two-point improvement needs roughly four thousand two hundred per set.
  10. Why does quantization-aware training use fake quantization instead of just quantizing at the end?Fake quantisation lets training see the rounded weights in the forward pass while still updating the exact weights, so the model learns weights that survive rounding instead of being rounded only after training.
  11. Why doesn't pruning half the weights make the model twice as fast?Because the zeros stay in the matrix: they are still stored and still multiplied, so the model is the same size and the same speed as before.
  12. Why can't I A/B test a change where users can affect each other?A split assumes one person's result does not depend on what version others got. When users compete for the same jobs or stock, the control group is changed by the treatment, so the comparison is biased.
  13. Why does detecting a small improvement take so many more users?Because the noise you are trying to see past shrinks only with the square root of the sample, so halving the change you want to spot takes four times the people.
  14. What's the difference between searching and just filtering a database?A filter returns every row that matches a condition you stated; a search returns a short ranked list of things that might answer you, so ordering, not matching, is the job.
  15. Why does the model have to be asked again after the tool hands back its result?Because a tool call is a request, not an answer. The result must be added to the conversation and the model called again to read it and reply.
  16. How do I know if a failed task was the model's fault or the tool's?You cannot tell from the reply alone: grade the final state of the data, log what each tool actually returned, and assign every failure to the model, a tool, or your own loop code.
  17. Can I judge my model's accuracy from the labels that come back first?No. Early labels are the failures, not a random sample, so an early accuracy figure is biased and usually makes the model look worse than it is.
  18. How small can my test set be and still catch a regression before a user does?About sixty auto-checked cases, run in two minutes on every change, blocking only on outright failures such as a case that used to pass now failing, never on a fall in the overall score.

The systems around the model

15 answers
  1. What's the difference between a CDN cache and caching inside my own app?A CDN cache sits near the user and can only hold answers that are identical for everyone, while a cache inside your app can be personal but exists once per process.
  2. What's the difference between LRU and LFU, and which one should I use?LRU drops the item unused for longest; LFU drops the item requested least often. Both guess at future use, and the cheap win is refusing one-off arrivals rather than counting requests.
  3. Why can't a system just stay consistent and available when the network splits?During a split, a cut-off machine can only answer from the data it already holds, so it must either risk being wrong or refuse to answer.
  4. How can I tell if my traffic is bursty or just high volume?Compare the variance of the gaps between arrivals with the square of their average gap: high volume alone does not make traffic bursty.
  5. Why does average latency hide the slow requests users actually feel?Waiting times are lopsided, so the average sits between the common case and the bad case and describes neither, and it is the bad case that users keep meeting.
  6. Should the gradients be summed or averaged across GPUs?Sum the gradients when each device already divides its loss by the global batch size; average them when each device uses its own local mean. Doing both makes the gradient too small by the number of devices.
  7. Why can a higher hit rate still mean the cache is making things slower?Hit rate counts hits equally and ignores what each hit costs, so a cache with many cheap hits can save less than one with few expensive hits, and can even add more work than it removes.
  8. Should the cache go in front of my app or between the app and the database?Put the shared cache between the app and the database by default, and push a copy in front of the app only for the part of a response that every reader gets identically.
  9. Does a server being up mean my service is available?No. A server being up only says a check got an answer; availability is the share of user requests that succeeded, measured where users are.
  10. Why does concurrent code fail only sometimes even when there is no syntax error?Concurrent code can fail only sometimes because a multi-step update can be interrupted between its read and its write, so two workers act on the same stale value and one update is lost without any error.
  11. Why do retries make an overloaded service worse?Retries add extra requests precisely when a service is already too busy, so they turn a small overload into a much larger one and can keep the failure going by themselves.
  12. How can I estimate wait time from the number of requests in the system?Divide the average number of requests inside the system by the rate at which they arrive, and you get the average time each request spends there.
  13. Why does latency explode when utilization gets close to 100%?Waiting time grows as one over the capacity you have left, so halving the spare capacity doubles the wait and the last few percent of load dominate.
  14. What does it actually mean to say a copy will be 'eventually' consistent?It means the copies will agree in the end, and that for some unspecified time a read may return an out of date value.
  15. What's the difference between a nested loop join and a hash join?One looks each outer row up in the inner table, fast only when the outer side is tiny and indexed; the other builds a lookup table in memory and reads each side once, which suits two large tables.

How a program runs

12 answers
  1. Why does 2 + 3 * 4 equal 14 and not 20?The multiplication is done first: three times four makes twelve, and only then is the two added, giving fourteen. Reading left to right would give twenty.
  2. How many samples does a profiler need before a percentage in the report means anything?It depends on how many samples landed on that line: you need a few hundred samples on the line you care about, because the relative error is roughly one over the square root of that count.
  3. If I make one part of my program twice as fast, how much faster does the whole program get?Less than twice as fast: only the share of run time you actually touch can improve, so the untouched remainder caps the gain, often at a few per cent.
  4. Why does reading one byte from memory load 64 bytes?Memory hardware moves data in fixed sixty-four-byte blocks, so a one-byte read fetches the whole block; the extra bytes cost almost nothing because reaching the memory row costs most of the time.
  5. Why can two threads that each work correctly still give the wrong answer?Each thread reads the shared value and writes it back in two separate steps, so the other thread can run in between and one increment silently overwrites the other.
  6. My change made it 4% faster — how do I know that isn't just noise?You cannot tell from one pair of runs. Run the unchanged program ten times first: only then does a four per cent gap mean anything.
  7. Why does the compiler catch a type mismatch but not division by zero?A compiler checks types before running because types are known for every possible run; it cannot know a divisor is zero until the program actually runs.
  8. Why does every program I run see the same memory addresses?A pointer's number is a private label for a block of memory, not a physical chip location, so two programs can use the same label for different places.
  9. Why is a system call so much slower than calling a function?A system call is slow because the processor has to switch into the kernel and back, and that fixed cost is paid in full even when the request itself is trivial.
  10. Why does my program have to ask the operating system just to read a file?The processor refuses to run the instructions that touch a disk or change memory mappings in your program's limited mode, so your code must trap into the kernel and ask it to do the read.
  11. How does the CPU avoid walking the page table on every single memory access?A small cache of recent translations, called the TLB, is checked on every memory access; a hit skips the page-table walk, and a miss walks once then stores the result.
  12. Why does sorting an array make a loop that scans it and counts matches so much faster?Sorting does not make the comparisons cheaper. It makes the test inside the loop predictable, so the processor stops guessing wrong and stops paying for those mistakes.

Where the data lives

1 to 5 of 24
  1. Why is the date off by one day after I save it and read it back?You saved a calendar date as if it were a precise moment, so showing it in another time zone can move it to the day before or after.
  2. Is UTF-8 the same thing as Unicode, or what's the difference?No: Unicode assigns a number to each character, while UTF-8 is a way of writing those numbers as bytes, using one to four bytes per character.
  3. Why does my CSV file fall apart when one of the fields has a comma in it?Because the same comma both separates fields and sits inside the value, so a reader splitting on commas finds one field too many and every later column shifts.
  4. Why do programmers write numbers in hexadecimal instead of just normal numbers?Programmers use hexadecimal because each digit stands for exactly four bits, so one byte is always written as two digits and the columns line up.
  5. How do I make a step safe to run again when I'm not sure whether it already ran?Change the operation so that sending it twice leaves the same stored data as sending it once, instead of relying on remembering not to send it again.

If you want the whole thing

Behind every answer here is a course that works through its subject from the beginning, with problems under each idea and a reason rather than a tick when one is marked. An answer settles the question you came with; the course is what makes the next one easier.

The libraryAsk for a course