Models

ControlNet

An architecture that adds spatial control to image generation models. Instead of just describing what you want in text ("a person standing"), ControlNet lets you specify how — providing an edge map, depth map, pose skeleton, or segmentation map that guides the composition. The generated image follows the spatial structure of your control input while filling in details from the text prompt.

Why it matters

ControlNet made AI image generation usable for professional workflows. Without it, you get random compositions and hope for the best. With it, you specify the exact pose, layout, or structure you need. This is the difference between "generate something vaguely like what I want" and "generate exactly this composition with these details" — critical for design, advertising, and production work.

Deep Dive

ControlNet (Zhang et al., 2023) works by creating a trainable copy of the diffusion model's encoder and connecting it to the original model via zero-initialized convolution layers. The control signal (edge map, pose, depth) is processed by this copy, and the features are added to the main model's corresponding layers. The zero initialization means the control starts with no effect and gradually learns to guide generation during training, preserving the original model's quality.

Control Types

Common control inputs: Canny edges (outline structure), OpenPose (human body pose), depth maps (3D structure), segmentation maps (which region is what), normal maps (surface orientation), and scribbles (rough sketches). Each control type requires a separately trained ControlNet. Multiple controls can be combined: a pose skeleton plus an edge map gives you both body position and structural details.

IP-Adapter and Beyond

Beyond spatial control, techniques like IP-Adapter provide style control: give a reference image and generate new images in the same style. T2I-Adapter is a lighter alternative to ControlNet that achieves similar control with fewer parameters. The trend is toward increasingly precise, composable control — specifying exactly what you want through a combination of text, spatial guides, style references, and iterative refinement.

Related Concepts

← All Terms

← Contrastive Learning Convolution →