Skip to main content
Zubnet AI › Learn › Wiki › Segment Anything
Models

Segment Anything

Also known as: Segment Anything Model
Meta's family of promptable segmentation models that can cut out any object in an image or video from a click, a box or a text prompt, without task-specific training.

Why it matters

SAM made high-quality segmentation a commodity. Tasks that used to need a labeled dataset and a training run now need a click. It's built into photo and video editors for one-click cutouts, into annotation tools where it speeds up labeling many times over, and into pipelines for medical imaging, satellite imagery, robotics and augmented reality.

Meta also released the code and weights openly, the first two versions under the Apache 2.0 license, which made SAM one of the most widely used building blocks in computer vision.

Deep Dive

Segmentation means finding the exact pixels that belong to an object. Traditionally you trained a separate model for each kind of object: one for tumors, one for cars, one for crops. The Segment Anything Model (SAM), released by Meta AI in April 2023, took a different approach. You point at something, with a click or a rough box, and it returns a precise mask of that object, whatever it is.

SAM brought to segmentation what large language models brought to text: one general model, steered by prompts, that works on things it was never specifically trained for.

How SAM Works

SAM has three parts:

  • Image encoder: a large Vision Transformer that turns the image into an embedding. This is the slow step, and it runs once per image.
  • Prompt encoder: turns points, boxes or rough masks into embeddings.
  • Mask decoder: a small, fast network that combines the two and outputs masks in milliseconds.

Because the heavy encoder runs only once, a user can click around an image and get new masks instantly. When a prompt is ambiguous (a click on a shirt could mean the shirt or the person wearing it), SAM returns several candidate masks with confidence scores.

The Data Engine

The model's strength came from its dataset, SA-1B: 11 million images with 1.1 billion masks, by far the largest segmentation dataset at the time. Meta built it with a loop. Annotators used an early SAM to label images, the model was retrained on the results, and eventually the model generated masks fully automatically, with humans only checking quality.

The SAM Family

  • SAM (2023): images, prompted by points and boxes.
  • SAM 2 (2024): extends to video. After a click on a single frame, a memory mechanism tracks the object across frames, even when it's briefly hidden. It's also faster and more accurate on images.
  • SAM 3 (2025): adds concept prompts. Give it a short phrase like "yellow school bus" or an example image, and it finds and segments every matching instance, then tracks them through video. A companion model, SAM 3D, reconstructs objects and people in 3D from a single image.

What It Doesn't Do

The first two versions don't know what they're segmenting. They output masks, not labels. To get "all the dogs", people paired SAM with a detector that takes text, as in the popular Grounded-SAM pipeline, until SAM 3 built that ability in.

SAM can struggle with very fine structures, low-contrast boundaries and specialized domains such as medical scans, where fine-tuned variants like MedSAM do better. The image encoder is heavy for real-time or mobile use, which led to lighter versions such as MobileSAM, FastSAM and EfficientSAM.

Using It

For interactive editing or labeling, use SAM through a tool that already integrates it, as many annotation platforms and editors do. For automated pipelines, a common recipe is to let SAM generate masks cheaply, have humans correct them, and train a small, fast, task-specific model on the result. That gives you SAM's quality at labeling time and a lightweight model at run time.

← All Terms
ESC