| 1 | To close — this 2017 paper turned out to be one of the most consequential architectural ideas in modern machine learning. The Transformer became the substrate for BERT, for GPT, for T5, for the Vision Transformer, for AlphaFold, for CLIP, for almost every large language model in production today, including the one talking to you right now. The core insight — that just attention, with no recurrence and no convolution, is the right inductive bias for sequence modeling — turned out to generalize far beyond translation, to almost every modality we work with. One paper, a generation of models. |