Activation Functions

Remove the activation functions from a fifty-layer network. You don't get a fifty-layer network that performs a bit worse: you get, exactly and provably, a linear regression. Fifty million parameters computing what a single matrix would compute. This is the most important result of this lesson, and it fits in two lines of algebra.

These functions, which you write in a single gesture — an nn.ReLU() slipped between two nn.Linear layers — are therefore what makes depth exist. And the choice is not cosmetic. The history of deep learning between 1990 and 2010 is largely the history of a failure wrongly blamed on depth, when it actually came from the activation function in use: the sigmoid crushes gradients, and a ten-layer sigmoid network structurally cannot be trained. The switch to ReLU, around 2011, did more for the feasibility of deep networks than any optimization trick.

This lesson also covers the most common mistake in all of PyTorch: adding a Softmax at the output of a model trained with CrossEntropyLoss. The code runs, the loss goes down, the results are mediocre, and nobody sees why. We will quantify precisely what this mistake costs.