ResNet
Why it matters
Before ResNet, making networks deeper often made them worse, not because of overfitting, but because very deep networks were simply hard to train. Signals and gradients faded as they passed through many layers. Residual connections fixed this by giving information a direct path through the network. Depth stopped being a wall.
That single idea outlived the model. Every transformer, including every modern LLM, wraps each attention and feedforward block in a residual connection. The ResNet paper is one of the most cited scientific papers of all time, in any field.
ResNet models themselves, especially ResNet-50, also became the standard backbone for computer vision for years, used for classification, object detection, segmentation and as a reference point in countless papers.
Deep Dive
ResNet was introduced by Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun at Microsoft Research in 2015. It won the ImageNet image classification competition that year with a network of 152 layers, about eight times deeper than the previous year's winners, and a top-5 error rate of about 3.6%, roughly on par with or better than estimates of human performance on that task.
Its key idea was the residual connection (or skip connection): instead of each block of layers transforming its input entirely, the block learns only a change to add to its input. The original input skips past the block and is added back at the end.
The Residual Block
A normal block computes output = F(x). A residual block computes output = F(x) + x. If the best thing a block can do is nothing, it only needs to learn F(x) = 0, which is easy. In a plain network, the same block would have to learn to perfectly copy its input, which turns out to be hard. This makes extra layers "free" to add: at worst, they learn to stay out of the way.
During backpropagation, the + x path also lets gradients flow straight back to early layers, which avoids the vanishing gradient problem.
The Family
ResNets come in standard depths: ResNet-18, -34, -50, -101 and -152. The deeper versions use bottleneck blocks (a 1×1 convolution to shrink channels, a 3×3 convolution, then a 1×1 to expand back) to stay efficient. They also used batch normalization throughout, which was important to make training stable.
Influence and Successors
- Transformers (2017) adopted residual connections around every sub-layer.
- ResNeXt, DenseNet and EfficientNet built on the idea in vision.
- Vision Transformers (2020) eventually challenged convolutional networks for image tasks, but ResNet-style CNNs remain widely used where speed and simplicity matter, especially on edge devices.
- U-Nets in diffusion models are built from residual blocks.
Why It's Still Worth Knowing
If you fine-tune an image classifier for a practical task, a pre-trained ResNet-50 is still a fast, reliable baseline. And when you read about the "residual stream" in LLM interpretability research, you're looking at ResNet's legacy: the running sum that every layer adds to.