Bag of Words and TF-IDF
Turn text into a numeric matrix with scikit-learn: counting, TF-IDF weighting, character n-grams, and the explosion of dimensionality.
A learning algorithm doesn't consume text, it consumes numbers. Between the two, you need a function that turns a document into a vector, and that function determines the quality of the final result more reliably than the choice of algorithm itself.
The oldest solution is also the most widely used in production: the bag of words. You discard word order and keep only presence and frequency. "The cat eats the mouse" and "The mouse eats the cat" become the same vector. That is an obvious loss of information, and yet this representation is enough to filter spam, route support tickets, or classify news wires by section with an accuracy that embarrasses more sophisticated approaches.
There are three reasons to start here, even if your end goal is a transformer. The first is that it's an honest baseline: if your 110-million-parameter model can't beat a TF-IDF trained in two seconds, the problem isn't in the architecture. The second is interpretability: you can read the coefficients and know which words drive the decision, something no transformer offers as directly. The third is cost: a logistic regression on TF-IDF trains on a laptop and responds in a fraction of a millisecond, with no GPU and no inference server.