Overfitting and Regularization

A network that reaches a loss of $0.0001$ on its training data can be completely useless. That isn't a paradox: it's the most common outcome when data is scarce. The model has memorized. It knows the 500 images it was shown by heart, associates each one with its label, and has learned nothing transferable. Show it a 501st image and it answers at random.

The word "overfitting" is actually misleading, because it suggests excess zeal. What happens is more precise: the model exploits regularities that exist in the sample but not in the population. A speck of noise on an image, a chance correlation between two pixels, an artifact from the sensor that photographed only one of the classes. These patterns are real in the training data. The model is right to use them to minimize its loss; it is our objective that was poorly posed.

The methodological consequence is the one that structures the whole discipline: the training loss tells you nothing about the quality of a model. It measures memorization as much as learning, and nothing lets you separate the two by looking at it alone. You need a second dataset, never used to update the weights, and it is on that one, and only that one, that you judge.