| 1 | The supporting cast inside every layer is just three small ideas. The position-wise feed-forward network is two linear layers with a ReLU between, applied identically at every position, with an inner dimension of 2048. The residual connection adds the sub-layer's input back to its output, which keeps the gradient signal flowing through deep stacks. Layer normalization then standardizes across the feature dimension. Put them together and the output of every sub-layer is LayerNorm of x plus Sublayer of x. Nothing here is novel by itself — but composing them around attention is what makes the architecture trainable at depth. |