The original DiT paper (Peebles & Xie, 2023) showed that simply replacing the U-Net with a standard Transformer and scaling it up produced better image quality. The Transformer processes image patches (similar to Vision Transformers) with added conditioning from the diffusion timestep and class labels. The key finding: DiT follows clear scaling laws — larger models and more compute produce predictably better images, just like with LLMs.
U-Nets process images at multiple resolutions, downsampling then upsampling with skip connections. This inductive bias was useful when compute was limited, but it introduces architectural complexity and doesn't scale as cleanly. Transformers, with their uniform architecture, are simpler to scale and benefit more from additional compute and data. The trade-off: Transformers are more memory-hungry due to the quadratic attention over all image patches.
Stable Diffusion 3 and Flux use MM-DiT (Multi-Modal DiT), which processes text and image tokens through separate streams that interact via cross-attention. This is more effective than the simpler text-conditioning used in the original DiT. The text stream uses a frozen text encoder (like T5 or CLIP), and the image stream uses the diffusion process. The two streams exchange information at each Transformer block.