Reading the Paper That Started It
Reading the Results Tables With the Cost Column In
Last timeHow It Was Actually Trained
The results table in the 2017 transformer paper has a training cost column, and rebuilding it from the paper's own footnote changes which row wins.
The results section of the paper is two tables and about a page of text, and the
two tables are quoted far more often than they are read. The first has five
columns and almost every summary of the paper quotes one of them. The second has
twelve and almost no summary quotes any.
This lesson rebuilds both with the arithmetic in view. The point is not that the
paper is wrong anywhere. It is that the tables support fewer claims than they are
routinely made to support, and the missing claims are findable by hand.
The column most readers skip
The results table lists ten systems with their scores on two translation tasks,
English to German and English to French, and then gives two more columns headed
training cost in floating point operations. The footnote says how those were
produced. The paper multiplies the training time by the number of devices used
by an estimate of the sustained single precision capacity of each device, and it
states the capacities it assumed, which are 2.8, 3.7, 6.0 and 9.5 teraflops for
the K80, K40, M40 and P100 respectively. One teraflop per second is a million
million floating point operations per second.
Below, the whole table, with the costs written in units of ten to the eighteenth
operations so that they can be compared by eye. One unit is a billion billion
multiply-and-add style operations, and the smallest entry in the table is 3.3 of
them. The single models come first.
| system | EN-DE score | EN-FR score | EN-DE cost | EN-FR cost |
|---|---|---|---|---|
| ByteNet | 23.75 | not listed | not listed | not listed |
| Deep-Att with PosUnk | not listed | 39.2 | not listed | 100 |
| GNMT with RL | 24.6 | 39.92 | 23 | 140 |
| ConvS2S | 25.16 | 40.46 | 9.6 | 150 |
| MoE | 26.03 | 40.56 | 20 | 120 |
| Transformer, base | 27.3 | 38.1 | 3.3 | 3.3 |
| Transformer, big | 28.4 | 41.8 | 23 | 23 |
The two transformer rows carry one cost each because the same configuration was
trained for each language pair, so the single figure spans both columns in the
original.
Then the ensembles, which are several trained copies of one system voting
together. They are in the same table in the paper and they are a different kind
of entry: the score is bought by paying for the whole set, which is why their
cost column runs in the hundreds.
The lesson stops here
9 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents