Fine-Tuning a Pre-Trained Model

Pre-training CamemBERT required a corpus of more than a hundred gigabytes of French text and thousands of GPU hours. Fine-tuning it on your task takes a few thousand annotated examples and about twenty minutes on a consumer GPU. That ratio, spanning several orders of magnitude, is what made modern NLP accessible outside the big labs.

What transfers deserves to be named precisely, because it explains when the method works and when it fails. The pre-trained model has learned the structure of French: morphology, agreement, constituent order, the scope of negation, and a mass of lexical knowledge about co-occurrences. It has learned nothing about your task. Fine-tuning consists of grafting a small decision layer on top of this linguistic competence, and slightly readjusting the whole model to serve it.

The practical consequence is that fine-tuning is very effective when your task relies on understanding French — sentiment, intent, topic — and much less so when it depends on knowledge absent from pre-training, such as internal company jargon or recent industry-specific nomenclature.

This lesson covers the three approaches in the order you would use them in practice: the standard approach with the Hugging Face Trainer, the manual PyTorch loop to understand what the Trainer does, and low-rank fine-tuning, which cuts memory cost. It ends with the settings that really matter, and with what makes a result non-reproducible.