Tokenization: Splitting Text into Units

Tokenization is the first operation in any language processing pipeline, and the only one whose errors propagate all the way to the end without ever announcing themselves. A badly tokenized model doesn't crash: it learns a slightly wrong vocabulary and loses a few points of performance that nobody can explain.

French has no satisfactory equivalent for the word "token," and that's telling. A token is not a word: it's the unit your system decides to treat as atomic. Depending on the task, it can be a word, a punctuation mark, a word fragment, or even a single character. The question to ask is therefore never "how do I split a text" but "what is the right unit for what I want to do."

French makes the question particularly thorny. The apostrophe is a grammatical character, not a quotation mark: in « l'État », the apostrophe marks an elision and separates two words; in « aujourd'hui », it is internal to a single, fossilized word. No typographic rule distinguishes the two cases. The hyphen is even worse: it joins compound words (« porte-monnaie »), postposed pronouns (« donne-le-moi »), and interrogative inversions (« viens-tu »), and they shouldn't all be handled the same way.