Building a vocabulary: the tokenizer algorithm (usually BPE) starts with individual bytes or characters and iteratively merges the most frequent pairs. After 32K–128K merges, you have a vocabulary where common words are single tokens ("the," "and," "function") and rare words are split into subword pieces ("un" + "common," "pre" + "process" + "ing"). Special tokens like <BOS> (beginning of sequence), <EOS> (end), and <PAD> (padding) are added explicitly.
Larger vocabularies compress text better (fewer tokens per sentence = cheaper, fits more in context) but increase the model's embedding table size. A 128K vocabulary with 4096-dimensional embeddings adds ~500M parameters just for the token tables. For a 7B model, that's 7% of total parameters doing nothing but mapping tokens to vectors. For a 1B model, it would be 50%. This is why smaller models tend to use smaller vocabularies.
A vocabulary's language coverage depends on its training corpus. Llama's early tokenizer was trained predominantly on English and represented Chinese characters as 3–4 tokens each, making Chinese inference 3–4x more expensive than English. Llama 3's tokenizer was trained on more balanced multilingual data, dramatically improving non-English efficiency. This is a solvable problem, but it requires deliberate effort — the default is English-dominant.