| 1 | The same attention mechanism shows up in three places in the Transformer, with three slightly different roles. In encoder self-attention, queries, keys, and values all come from the previous encoder layer; every source position attends to every other. In decoder masked self-attention, the same setup applies, but illegal future positions are set to minus infinity before the softmax, so position i attends only to positions less than or equal to i — that's how auto-regression is preserved. In encoder-decoder attention, queries come from the decoder while keys and values come from the encoder output, letting every decoder position retrieve relevant information from anywhere in the source sequence. One mechanism, three roles. |