Transformers and the Attention Mechanism
Dot-product attention laid out in PyTorch: multiple heads, positional encoding, quadratic cost, and what attention does not prove.
Until now, every representation has had the same flaw: it didn't know which word to look at. The bag of words ignores order. Static embeddings give each word a fixed vector. Recurrent networks, which dominated the field until 2017, read the sentence word by word while maintaining a memory, but that memory fades, and the computation is sequential by construction: you can't process word 50 before you've processed word 49.
Take the sentence "Le rapport que la commission avait commandé au cabinet parisien il y a deux ans est enfin arrivé" ("The report that the committee had commissioned from the Paris firm two years ago has finally arrived"). To make the participle "arrivé" agree correctly, you have to link it to "rapport," thirteen words back, while ignoring all the intervening nouns. A recurrent network has to carry that information through thirteen successive steps, and it gets diluted along the way. This is the problem of long-range dependencies, and French, with its embedded relative clauses and long-distance agreement, produces it constantly.
The paper Attention Is All You Need, published by Vaswani and his coauthors in 2017, solves both problems at once with a disarmingly simple idea: rather than carrying information from one step to the next, let each word look directly at all the others, and learn how much it should look at each one. Distance disappears ("arrivé" reaches "rapport" in a single operation), and because all words compute their attention simultaneously, everything parallelizes on a GPU.