5-fold CV: split data into 5 parts. Train on parts 1-4, evaluate on part 5. Then train on parts 1-3+5, evaluate on part 4. Repeat for all 5 folds. Average the 5 evaluation scores. The result is more reliable than a single 80/20 split because it's robust to the particular split — a "lucky" or "unlucky" test set can't skew the result. The standard deviation across folds indicates reliability.
For classification with imbalanced classes (rare disease: 5% positive, 95% negative), random splitting might put all positives in one fold. Stratified K-fold ensures each fold has the same class distribution as the full dataset. This prevents folds with no positive examples (useless for evaluation) and gives more reliable performance estimates for minority classes. Always use stratified K-fold for classification.
Cross-validation is computationally expensive (K times the training cost) and rarely used for large models. Fine-tuning a 7B model 5 times for 5-fold CV is impractical. For LLMs, a single held-out validation set is standard because: the datasets are large enough for reliable single-split evaluation, training is expensive, and the model's pre-trained representations make it less sensitive to the specific training split. Cross-validation is most valuable for small datasets with classical ML models.