The two dominant self-supervised approaches in NLP are causal language modeling (predict the next token, used by GPT/Claude/Llama) and masked language modeling (mask random tokens and predict them, used by BERT). Causal modeling produces generative models — they can write text. Masked modeling produces understanding models — they excel at classification, search, and analysis but can't generate fluently.
Predicting the next token sounds trivial, but to do it well, a model must learn grammar, facts, reasoning, style, and even some common sense. If the text says "The capital of France is," the model needs world knowledge to predict "Paris." If it says "She picked up the ball and threw it to," the model needs to understand pronouns, physics, and social context. The simple objective of next-token prediction creates pressure to learn deeply about language and the world it describes.
In vision and embeddings, self-supervised learning often uses contrastive objectives: learn representations where similar items are close together and dissimilar items are far apart. CLIP (matching images to text descriptions), SimCLR (matching augmented views of the same image), and embedding models all use this approach. The supervision signal comes from the data structure itself — two crops of the same image should have similar representations, while crops of different images should not.