Researched
Residual Networks (ResNet)
He, Zhang, Ren and Sun (December 2015) add shortcuts so each layer learns only a correction to its input; a 152-layer network won ImageNet 2015, and the idea is now in nearly every deep model.
Open in the interactive tree →Before 2015, stacking more layers made networks harder to train and even lowered training accuracy, a failure called degradation. A residual block computes f(x) + x, so a layer can fall back to passing its input through unchanged. The 152-layer ResNet, eight times deeper than VGG, reached 3.57% error on the ImageNet test set as an ensemble and won the ILSVRC 2015 classification task; the paper also analysed 100- and 1,000-layer networks on CIFAR-10. Residual connections are used in all transformers, in AlphaGo Zero and in AlphaFold.
Prerequisites
- Convolutional Networks (LeNet)1989ResNet is a convolutional network with shortcut connections added.
- ImageNet Dataset2009The 152-layer network was trained and tested on ImageNet's 1.2 million labeled images.
- Deep Learning2012It was built to make the very deep networks of the deep-learning boom trainable.
Unlocks
- Transformer Architecture2017Every transformer block wraps its layers in residual connections.