NVIDIA Research released X-Token, a cross-tokenizer knowledge distillation method that lets a student model learn from teachers with different tokenizers — the constraint that has blocked distilling across model families. Standard KD requires the teacher and student to share a vocabulary so token positions align; X-Token removes that. The headline result on a Llama-3.2-1B student distilled from Qwen3-4B: GSM8k accuracy goes from 2.56 (with the prior method, GOLD) to 15.54, a 6× recovery, with +3.82 average across benchmarks. Multi-teacher distillation across tokenizer families — previously impossible — reaches 20.39 GSM8k with a Phi-mini + Llama-3B pair. Paper is arXiv 2605.21699; the work runs on a single H100 (128 used for iteration speed).

The mechanism is worth understanding because it explains why cross-tokenizer KD was hard. GOLD, the prior method, had two structural failures. First, uncommon-token suppression: Llama tokenizes "201" as one token, Qwen splits it into "2","0","1" — so all 1,100 multi-digit Llama numerals fall into GOLD's unmatched set and receive identity-agnostic noise plus suppressive gradients, collapsing GSM8k to 2.56. Second, over-conservative matching: GOLD uses strict string equality, so student token "Hundreds" mapping to teacher "Hund"+"reds" is discarded, losing real alignment signal. X-Token fixes both with a deterministic projection matrix W built before training: pass one sets exact string matches to 1; pass two re-tokenizes unmatched student tokens under the teacher vocabulary and, if the result is ≤4 tokens, assigns decayed weights (0.9·0.1^i, so a 2-token span gets 0.909/0.091). Each row sums to 1, making the projection probability-preserving. Two losses follow: P-KL projects the student distribution into teacher vocabulary space; H-KL relaxes matching to top-1 mappings under W.

The ecosystem read: this unlocks multi-teacher distillation across model families, which tokenizer mismatch has been quietly blocking. For builders distilling small models, you are no longer limited to teachers that share your student's tokenizer — you can pull from the strongest teacher per capability regardless of its vocabulary, and combine teachers from different families. The finding that "teacher complementarity, not teacher count, drives gains" is the design guidance: a Phi-mini + Llama-3B pair beat overlapping pairs because the teachers covered different weaknesses, not because there were more of them. This is the open-research counterweight to proprietary distillation pipelines — the cross-tokenizer constraint was a real moat for whoever had matched teacher-student vocabularies, and X-Token erodes it.

If you distill small models Monday morning: X-Token (arXiv 2605.21699) removes the same-tokenizer constraint from your teacher selection, so re-architect your distillation to pull from the best available teacher per skill rather than the best teacher that happens to share your tokenizer. The honest caveats: results are on a Llama-3.2-1B student specifically, code availability is not confirmed in the writeup (check the arXiv repo), and these are NVIDIA's own benchmark numbers pending independent reproduction. The projection-matrix idea is simple enough to reimplement from the paper if the code is not released — which is the real portability test.