Recurrent Networks and Sequential Data

"The movie wasn't bad" and "the movie was bad, not good" contain almost the same words and say opposite things. A model that counts words without considering their order cannot tell these two sentences apart: the information isn't in the vocabulary, it's in the sequence. The same problem arises with an electrocardiogram, a stock price, a GPS trajectory, a voice command.

Two difficulties then appear, and none of the architectures seen so far handles either. The first is variable length: sentences don't all have the same number of words, while an nn.Linear requires a fixed input size. The second is long-range dependency: in "the keys that I left on the kitchen table last night are missing," the verb agreement depends on a word located ten positions earlier.

Recurrent networks answer both in a single move: one cell, applied at every time step, that carries an internal state from one step to the next. Length no longer matters, since the cell is applied as many times as needed with the same weights. And the state acts as a memory of everything that came before.