Datasets come in many forms: text corpora for language models, labeled images for classifiers, question-answer pairs for fine-tuning, preference pairs for alignment, and benchmark datasets for evaluation. The distinction between training set (what the model learns from), validation set (what guides hyperparameter tuning), and test set (what measures final performance) is fundamental — evaluating on training data is meaningless because the model has memorized it.
LLM pre-training datasets have grown from millions of tokens (early GPT) to trillions (modern models). Common Crawl, Wikipedia, books, code repositories, scientific papers, and curated web text form the typical mix. But more data isn't always better — the Chinchilla scaling laws showed that data quality and quantity must scale together with model size. Deduplication, filtering toxic or low-quality content, and balancing domains are all critical steps.
Every dataset carries the biases of its sources. A model trained mostly on English web text will perform worse on other languages. A dataset scraped from the internet inherits society's prejudices. This isn't a problem you can fix with architecture — it requires careful data curation, auditing, and post-training mitigation. The most impactful AI ethics work often happens at the dataset level.