Introduction to Natural Language Processing

A database can compare two numbers. It cannot decide whether "the movie wasn't bad" is a compliment. All of natural language processing lives in that gap: text is the most abundant form of data humanity produces, and it is the one form that classical computing cannot handle as anything other than a sequence of bytes.

The difficulty is not the volume, it is the ambiguity. Take the French sentence "Les avions volent bas." A French reader immediately understands aircraft flying close to the ground. But avions is also the imperfect tense of the verb avoir ("we had"), and volent can mean to steal. Purely formally, the sentence is ambiguous; it is your knowledge of the world that settles it, not grammar. Multiply these choices by the length of a document and you can see the problem: there is no exhaustive rule, only statistical clues to accumulate.

On top of that, French is not English. Most NLP tools were designed, trained, and evaluated on English, a language with little inflection, no accents, and where the apostrophe is rare. French elides ("l'État"), agrees ("sont allées"), uses accents that change meaning ("tache" is not "tâche"), and glues its pronouns together ("donne-le-moi"). A pipeline copied from an English-language tutorial produces mangled French text without ever raising an error. That is the worst case, because nothing warns you.