| 1 | Before we get into the architecture, let's see what the paper is actually claiming. On the WMT 2014 English to German translation task, the Transformer big model scored 28.4 BLEU, beating the previous best ensemble by two full points as a single model. On WMT 2014 English to French, it set a new single-model state of the art at 41.8 BLEU. And it did all this by training for three and a half days on eight P100 GPUs — a small fraction of the cost of the strong baselines. The architecture has no recurrence and no convolution; it is built entirely from attention and feed-forward layers. Quality up, cost down, parallelization unlocked. |