Normalization: Case, Accents, Lemmatization

Normalizing means deliberately choosing to lose information. Lowercase everything, and you give up the distinction between “État” (the state, as an institution) and “état” (a condition). Strip the accents, and “tache” (stain) and “tâche” (task) become the same word. Remove the stop words, and “je n'aime pas ce film” (I don't like this movie) and “j'aime ce film” (I like this movie) end up identical.

That last consequence is not a textbook scenario. It is the most frequent and most costly mistake in French-language sentiment analysis projects: off-the-shelf stop word lists contain ne, pas, plus, sans, aucun, which are exactly the words that carry negation. A model trained after this filtering is structurally unable to tell a negative review from a positive one, and its metrics won't tell you so clearly.

So the right question is not “how do I normalize” but “what is my task allowed to forget.” For document search, flattening accents is a service to users who type without them. For entity extraction, it is a disaster: proper nouns are distinguished precisely by their spelling.

This lesson walks through the five classic normalizations, shows what each one destroys in French, and ends with a reusable function in which every step is a parameter, because no single setting suits every task.