Most video generation models extend the DiT (Diffusion Transformer) architecture to 3D: instead of processing 2D image patches, they process 3D patches that span both spatial dimensions and time. The model learns to denoise entire video volumes, maintaining spatial consistency (objects look the same across frames) and temporal consistency (motion is smooth and physically plausible). Conditioning works similarly to images: text embeddings guide the generation via cross-attention.
Video generation is extraordinarily compute-intensive. A 10-second video at 30fps is 300 frames — 300x the work of a single image, plus the additional challenge of temporal coherence. Training video models requires video datasets (harder to curate than image datasets) and GPU clusters that make LLM training look modest. This compute requirement is why video generation quality lags behind image generation by roughly 2 years.
Today's models struggle with: long durations (most max out at 5–15 seconds), complex multi-object interactions, physics-defying motion (objects sometimes float or deform), consistent character identity across cuts, and fine-grained text control. The technology is impressive for b-roll, establishing shots, and creative exploration, but not yet reliable enough for narrative filmmaking where specific actions, expressions, and timing matter.