Gradient Descent and Optimizers

Training a network means searching for the minimum of a function whose shape you will never see. It lives in a space with millions of dimensions, you can't plot it, you can't explore it exhaustively, and there is no guarantee that it has a single minimum. All you have, at a given point, is the direction of steepest slope. You move in the opposite direction, then repeat. That is the entire algorithm.

This apparent simplicity is deceptive. The behavior of this descent depends on one setting, the step length, and a bad value produces two radically different failures that are often confused. Too small, and the model stagnates, so you suspect an architecture problem. Too large, and the loss shoots off to infinity or gets stuck on a plateau, so you suspect a bug. An experienced practitioner can tell which of the two is happening by looking at three loss values.

There is a second, more insidious source of confusion. Adam, the de facto default optimizer, works so well on so many problems that you end up no longer knowing what it does. Yet its implicit settings (the moving average, the normalization by the square root of the second moment, the bias correction) have precise consequences, and there are situations where they hurt.