Zubnet AILearnWiki › Masked Language Modeling
Training

Masked Language Modeling

MLM, Masked LM, Cloze Task
A self-supervised training objective where random tokens in the input are replaced with a [MASK] token, and the model must predict the original tokens from context. BERT popularized MLM: mask 15% of tokens, use bidirectional attention to look at both left and right context, and predict the masked words. This creates powerful text understanding models (as opposed to text generation models).

Why it matters

MLM is the training objective that created BERT and the entire family of encoder models that still power most production search, classification, and embedding systems. Understanding MLM vs. causal language modeling (next-token prediction) explains the fundamental split between understanding models (BERT) and generation models (GPT) — and why each excels at different tasks.

Deep Dive

The process: take a text sequence, randomly select 15% of positions. For those positions: 80% are replaced with [MASK], 10% are replaced with a random token, 10% are kept unchanged. The model must predict the original token at each selected position. The 80/10/10 split prevents the model from learning to only pay attention to [MASK] tokens, which don't appear during actual use.

Bidirectional Context

The key advantage of MLM over causal LM: the model sees both left and right context when making predictions. For the sentence "The [MASK] sat on the mat," the model uses both "The" (left context) and "sat on the mat" (right context) to predict "cat." This bidirectional understanding is why BERT-style models produce richer representations than left-to-right models for understanding tasks.

MLM vs. Causal LM

The trade-off: MLM creates excellent understanding (good for classification, search, NER) but can't generate text fluently (predicting masked tokens isn't the same as generating a sequence). Causal LM (predict the next token left-to-right) generates fluently but understands less deeply (only sees left context). This split drove the encoder-vs-decoder divergence in NLP. Modern LLMs are all causal (decoder-only) because generation is more commercially valuable, but MLM-trained models remain the backbone of search and classification.

Related Concepts

← All Terms
← Mamba MCP →