The formula: PPL = exp(−(1/N) ∑ log P(token_i | context_i)), where N is the number of tokens and P is the model's predicted probability for each actual token. If the model assigns high probability to every correct token, the sum of log probabilities is close to zero, and PPL approaches 1 (perfect). If the model is surprised by many tokens, the sum is a large negative number, and PPL is high.
You can only meaningfully compare perplexities between models that use the same tokenizer, or that are evaluated on the same text. A model with a larger vocabulary might have lower perplexity simply because it has more fine-grained tokens to assign probability to. Evaluation datasets matter too — perplexity on Wikipedia (clean, well-structured text) will be much lower than perplexity on Reddit (noisy, informal). Always check what tokenizer and evaluation set were used.
A model can have excellent perplexity but be terrible as an assistant. Pre-trained base models (before RLHF/DPO) typically have lower perplexity than their aligned counterparts, because alignment training optimizes for helpfulness rather than raw prediction accuracy. The aligned model might assign lower probability to the statistically most likely next token if that token would produce an unhelpful or unsafe response. This is a feature, not a bug — but it means perplexity is a measure of text modeling, not utility.