Typical splits: 80% training, 10% validation, 10% test. For large datasets, smaller percentages for validation and test suffice (even 1% of a million examples is 10,000 — plenty for reliable evaluation). For small datasets, cross-validation is preferred (see: Cross-Validation). The key rule: never use the test set for any decision during development. It's only for the final evaluation. If you peek at the test set during development, your performance estimate becomes biased.
When splitting data, ensure each split has a representative distribution of classes, domains, and other important characteristics. If your dataset is 90% English and 10% French, a random split might put all French examples in the training set, leaving you unable to evaluate French performance. Stratified splitting ensures proportional representation in each split. For time-series data, use temporal splits (train on past, validate on future) rather than random splits.
For LLM pre-training, the validation set is a held-out portion of the training corpus, used to compute perplexity during training. For fine-tuning, it's a held-out portion of the fine-tuning dataset. For alignment (RLHF/DPO), validation is more complex: automated metrics (reward model scores) plus human evaluation on held-out prompts. The validation strategy should match how the model will actually be used — if users will ask diverse questions, the validation set should be diverse.