| 1 | Let's name the constraints the authors are trying to lift. First, the sequential bottleneck: in any RNN, hidden state at time t depends on hidden state at t minus one, so you cannot parallelize within a single training example. Second, long-range path length: distant tokens have to be related through many intermediate operations, which makes learning long-range dependencies hard. Third, memory ceiling: long sequences eat GPU memory, capping batch size and throughput. The Transformer attacks all three at once by replacing the entire backbone with attention. |