Information / AI & Robotics

LSTM memory networks

Hochreiter and Schmidhuber's LSTM (1997) let neural networks remember across long sequences and drove speech and translation until transformers.

Sepp Hochreiter identified the vanishing-gradient problem in 1991, which stops ordinary recurrent networks from learning long-range dependencies. With Jürgen Schmidhuber he published the Long Short-Term Memory cell in 1997, which uses gates to keep information over many steps. LSTMs powered speech recognition, machine translation and handwriting recognition in the 2010s, and the attention mechanism that led to the transformer was first added to them.