Why Every Score Is a Log Probability
Last timeA Chart With a Squashed Axis
Multiplying a thousand probabilities gives nought on any real computer. Adding their logarithms gives the same quantity and survives, which is why nearly every score in the subject is a log.
A model assigns a probability to each word it reads. The probability of a whole
document is the product of those probabilities, one factor per word. For a
document of a thousand words, that is a product of a thousand numbers, each
below one.
This quantity is at the centre of the subject. It is also, computed as written,
exactly nought.
from math import log, exp, log10
p = [0.3] * 680 # 680 words, each given probability 0.3
product = 1.0
for q in p:
product *= q
print(product) # 0.0, and it reached nought around word 640
total = 0.0
for q in p:
total += log(q) # the same quantity, as a sum of logs
print(total) # -818.7, computed exactly
print(total / log(10)) # -355.6, so the probability is 1e-355.6Why it goes to nought
The arithmetic a computer does with fractional numbers keeps a fixed number of
digits and a separate exponent, which is itself a logarithm: the format is
built out of the idea this course is about. That exponent has a limited range.
For the usual double precision, the smallest positive value is about ten to the
minus three hundred and eight.
Multiply probabilities together and the exponents add, in the manner of the
third lesson. Each factor of a tenth subtracts one from the base ten exponent,
so around seven hundred such factors the exponent passes the floor and the
result becomes nought. Not approximately nought: the value nought, which then
poisons everything downstream, since dividing by it fails, taking its log
fails, and comparing two of them finds them equal.
The important point is that nothing warns you. There is no error, no flag in
ordinary use, and the program continues happily with a number that has lost
every digit it ever had. A model comparison that reports both candidates as
having probability nought looks like a tie and is an arithmetic failure.
What the logarithm buys
Apply the product rule from the third lesson and the product of a thousand
probabilities becomes the sum of a thousand logarithms. The quantity is not
changed; only the way it is reported is.
The lesson stops here
5 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents