| 1 | This is table one from the paper, and it is the strongest argument for self-attention. A self-attention layer pays O of n squared times d in compute, but in exchange every pair of positions is exactly one operation apart. A recurrent layer pays O of n times d squared, but it takes O of n sequential operations and the maximum path length between two positions is O of n. Convolutional layers fall in between. For typical NLP workloads where the sequence length is shorter than the representation dimension, self-attention is actually faster than recurrence — and the constant path length makes long-range dependencies far easier to learn. |