Autoencoder
Why it matters
If you've used Stable Diffusion, FLUX or most video generators, you've used a VAE. These models don't generate pixels directly, which would be far too expensive. They generate in the VAE's compressed latent space (often 8× smaller in each direction, so 64× fewer positions), and the VAE decoder turns the result into a full image at the end. The VAE is why a consumer GPU can generate a 1024×1024 image in seconds, and it's also behind some familiar artifacts: garbled small text, mushy faces at a distance, and color shifts often come from the VAE rather than the diffusion model itself.
Autoencoders also show up in anomaly detection (things the model can't rebuild well are probably unusual), denoising, and, more recently, interpretability research through sparse autoencoders.
Deep Dive
An autoencoder has two halves. The encoder takes an input, such as an image, and compresses it into a much smaller set of numbers. The decoder takes that compressed code and tries to rebuild the original. The network is trained on one simple goal: make the output look as much like the input as possible. Because the code in the middle is small, the network can't just copy the input through. It has to learn what actually matters about the data.
A variational autoencoder (VAE) adds a twist: instead of mapping each input to a single point, the encoder outputs a small probability distribution, and training nudges those distributions to form a smooth, well-organized space. That makes the space something you can sample from and move around in, which is what generative models need.
The Bottleneck Is the Point
A plain autoencoder learns a compressed representation without any labels, which makes it a classic example of self-supervised learning. The smaller the bottleneck, the more aggressively it has to throw away noise and keep structure. Too small and reconstructions get blurry; too large and it learns little beyond copying. The sweet spot depends on the job: image generation VAEs aim for near-perfect reconstructions, while a compression model for search might accept more loss.
What "Variational" Adds
A plain autoencoder's latent space can be full of holes: pick a random point and the decoder produces garbage, because nothing during training ever landed there. A VAE fixes this by adding a second term to the loss (a KL divergence) that pulls every encoded distribution toward a simple standard shape. The result is a continuous space where nearby points decode to similar outputs and random samples decode to plausible ones. The trade-off is that classic VAEs on their own tend to produce slightly blurry images, which is why today they're mostly used as the compression stage for diffusion models rather than as generators themselves.
VAEs Inside Image and Video Models
In latent diffusion, the pipeline is: encode training images with the VAE, train the diffusion model to denoise in that latent space, then decode the final latent back to pixels. Upgrading the VAE alone can noticeably sharpen results. Newer models moved from 4 latent channels (Stable Diffusion 1.x and SDXL) to 16 channels (Stable Diffusion 3, FLUX), which preserves much finer detail such as text and hands. Video models use 3D VAEs that also compress across time, so a few seconds of video fits in memory.
Sparse Autoencoders
A related variant, the sparse autoencoder, is trained on a language model's internal activations instead of images. Forcing most of its code to be zero makes each unit tend to represent one recognizable concept, which has become a key tool for peeking inside models.