For most of its history, NLP was an exercise in clever engineering around the fact that computers have no idea what words mean. The earliest systems relied on bag-of-words representations — literally counting how often each word appears in a document and ignoring word order entirely. TF-IDF improved on this by weighting rare words more heavily than common ones, which made search and document retrieval surprisingly effective for how crude the approach was. Then word2vec came along in 2013 and changed everything by learning dense vector representations where words with similar meanings ended up near each other in vector space. For the first time, a model could capture that "king" minus "man" plus "woman" roughly equals "queen." Recurrent neural networks and LSTMs pushed the field further by processing text sequentially, maintaining a hidden state that carried information forward through a sentence. They worked, but they were slow to train, struggled with long-range dependencies, and each NLP task — translation, summarization, question answering — needed its own bespoke architecture.
The 2017 "Attention Is All You Need" paper didn't just introduce a new architecture — it collapsed an entire ecosystem of specialized models into one general-purpose design. The Transformer's self-attention mechanism let the model weigh the relevance of every word against every other word in the input simultaneously, eliminating the sequential bottleneck of RNNs. What nobody fully anticipated was how well this architecture would scale. Pre-train a large Transformer on enough text and it learns to do translation, summarization, sentiment analysis, code generation, and dozens of other tasks without being explicitly trained on any of them. BERT showed this on the understanding side in 2018, GPT-2 showed it on the generation side in 2019, and by 2023 the pattern was clear: one architecture, scaled up with more data and compute, had effectively unified the entire field of NLP.
Despite the dominance of LLMs, classical NLP tasks haven't disappeared — they've just changed context. Named entity recognition (pulling out names, dates, organizations from text), part-of-speech tagging, sentiment analysis, and text classification are still everywhere in production systems. The question is when to use a dedicated model versus just asking an LLM. If you're processing millions of customer reviews per day to extract sentiment, a fine-tuned BERT classifier running on a single GPU will be orders of magnitude cheaper and faster than sending each review to GPT-4. If you're building a one-off analysis pipeline or handling a task that requires nuanced judgment, an LLM call makes more sense. The economics tilt toward specialized models at scale and LLMs for flexibility and low volume.
This brings up the pipeline question. Traditional NLP workflows are explicit pipelines: tokenize the text, apply POS tagging, run dependency parsing, extract entities, classify intent. Tools like spaCy and NLTK were built for this approach, and they're still excellent when you need deterministic, inspectable processing at high throughput. The alternative — throwing raw text at an LLM and asking it to do everything in one shot — is seductively simple but comes with trade-offs. LLMs are nondeterministic, expensive per call, and hard to debug when they get something wrong. In practice, most production NLP systems in 2026 are hybrids: structured pipelines for the parts that need speed and consistency, LLM calls for the parts that need reasoning and flexibility. A customer support system might use spaCy to extract entities and classify intent, then hand off to an LLM only for generating the actual response.
Multilingual NLP has come a long way, but the gap between English and everything else remains stubbornly real. Models like mBERT, XLM-R, and the multilingual variants of GPT and Gemini can handle dozens of languages, and cross-lingual transfer — training on English data and applying the model to French or Hindi — works surprisingly well for high-resource languages. The problem is the long tail. There are roughly 7,000 languages spoken on Earth, and the vast majority have almost no digital text to train on. Tokenizers trained primarily on English chop languages like Thai, Khmer, or Inuktitut into absurdly long token sequences, which degrades both performance and cost. Even for mid-resource languages like Vietnamese or Swahili, model quality drops noticeably compared to English. The root cause is data: NLP models learn from text, and the internet is overwhelmingly English. Fixing this isn't just a technical challenge — it's a question of whose language gets to participate in the AI revolution and whose gets left behind.