Word Embeddings: word2vec and Beyond
From word2vec to fastText and then to CamemBERT's contextual vectors: representing the meaning of words, along with their successes, biases, and real limits.
To a TF-IDF model, "voiture" and "automobile" (both meaning "car") are two independent columns. Their similarity is zero, exactly like that of "voiture" and "betterave" (beet). A classifier trained on texts that say "automobile" will be unable to do anything with a text that says "voiture". The bag of words doesn't represent meaning, it represents spelling.
The idea that unlocks this problem is old and can be stated in one sentence, attributed to the linguist John Rupert Firth in 1957: you shall know a word by the company it keeps. If two words regularly appear in the same contexts — "je conduis ma ___" ("I drive my ___"), "le garage répare ma ___" ("the garage is repairing my ___") — they probably have a similar meaning. This so-called distributional hypothesis has a decisive practical consequence: the meaning of a word can be learned from raw text, without human annotation, from co-occurrences alone.
A word embedding is the result of this learning: a vector of a few hundred real numbers per word, positioned so that words used in similar contexts end up close to each other. We go from a sparse space of fifty thousand dimensions to a dense space of three hundred, where distance carries meaning.