Backpropagation Explained Step by Step

loss.backward(). One call, no arguments, and the derivatives of millions of parameters appear. It is the most useful line in PyTorch, and also the one that most people write without knowing what it does. As long as everything works, ignorance is comfortable.

It stops being comfortable at the first serious problem. Why is the gradient of the first layer a million times smaller than that of the last? Why does this nan show up at the third iteration and not the first? Why does this model that shares an encoder between two heads raise "Trying to backward through the graph a second time"? None of these questions can be solved by trial and error. They are solved by knowing what quantity flows, in which direction, and what multiplies it at each step.

This lesson therefore does something unusual: it computes a gradient entirely by hand, with decimal numbers, on a network small enough that no step is hidden — one input, one hidden neuron, one output neuron, four parameters. Then it asks autograd to redo the same computation and compares the numbers. Then it verifies a third time with finite differences, a checking technique that practitioners really use to validate a hand-written layer.