| 1 | Because there is no recurrence and no convolution, the model has no built-in notion of token order. The authors recover order with a deterministic sinusoidal positional encoding added to each input embedding. Even dimensions use a sine wave, odd dimensions use a cosine wave, and the wavelengths form a geometric progression from two pi up to ten thousand times two pi. The neat property is that for any fixed offset k, the encoding at position pos plus k is a linear function of the encoding at pos — so the model can easily learn to attend by relative position via a simple linear combination. Order is injected, not learned. |