GPT
Why it matters
Deep Dive
The GPT story is really three stories stacked on top of each other. The first is a scaling story: starting with the 117-million-parameter GPT-1 in 2018, OpenAI showed that a transformer trained to predict the next token on enough raw text becomes a surprisingly capable general-purpose language system, and each generation — GPT-2, GPT-3, GPT-4 — bought large capability jumps mostly by buying more compute and data. The second is an alignment story: raw next-token predictors are awkward to use, so OpenAI layered instruction tuning and human feedback on top, turning a text-completion engine into the chat assistant that made the family famous. The third is a unification story: recent releases fold multimodality and deliberate step-by-step reasoning back into a single flagship, so "GPT" now names a whole product line rather than one architecture.
The Early Arc: GPT-1 to GPT-3
GPT-1 (2018) introduced the recipe in a paper modestly titled "Improving Language Understanding by Generative Pre-Training": pretrain a decoder-only transformer on a large unlabeled corpus, then fine-tune it per task. At 117 million parameters it was a proof of concept, but the proof worked. GPT-2 (2019, 1.5 billion parameters) scaled the recipe up and showed that a single model could perform tasks it was never explicitly trained on — translation, summarization, question answering — just by reading a well-phrased prompt; OpenAI initially withheld the full weights over misuse concerns, an early preview of every release debate since. GPT-3 (2020, 175 billion parameters) scaled another hundredfold and made the trick reliable enough to productize: give the model a few examples in the prompt and it picks up the pattern, a behavior known as few-shot learning. Just as consequential, GPT-3 shipped as a commercial API rather than downloadable weights, setting the access model that most frontier labs still follow.
ChatGPT and the RLHF Turn
A base GPT model does not answer questions — it continues text, so it is as likely to add more questions to your list as to answer them. InstructGPT (2022) fixed this with RLHF: human labelers ranked model outputs, a reward model learned their preferences, and the language model was fine-tuned against that reward until it followed instructions reliably. ChatGPT, launched in November 2022, wrapped a GPT-3.5-class model tuned this way in a free chat interface, and it reached roughly 100 million users in about two months — at the time the fastest-growing consumer application ever measured. The lesson the industry took was uncomfortable for pure scaling believers: the model had barely changed, but the alignment layer and the interface had, and that was enough to turn a research demo into a mass-market product.
GPT-4, the O-Series, and GPT-5
GPT-4 (2023) was the generation that made the family credible for professional work — a large jump in reasoning quality, instruction following, and reliability, with image input alongside text. GPT-4o (2024) made the model natively multimodal: one network handling text, images, and audio end to end, faster and cheaper than its predecessor, which is what enabled the real-time voice mode. In parallel, OpenAI spun off the o-series (o1, then o3): models trained with reinforcement learning to spend more tokens thinking before answering, trading latency for accuracy on math, code, and science — the test-time compute idea taken seriously. GPT-5 (August 2025) merged the two branches back into one system that routes each request between a fast conversational model and a slower reasoning model, so users no longer pick between "smart but slow" and "quick but shallow."
ChatGPT Is Not GPT
The terms get used interchangeably, but they name different things, and confusing them causes real bugs in reasoning about behavior. GPT is the model family: the weights you call through the API, with a fixed context window, a system prompt you control, and no memory between calls unless you build it. ChatGPT is a product: one of those models plus a system prompt, conversation memory, web browsing, file tools, and moderation layers, all of which shape what you see before the model even runs. When someone reports that "GPT refused" or "GPT remembered," the behavior usually comes from the product wrapper, not the base model — which matters when you move a prompt from the app to the API and the answers change. A second, milder confusion: "GPT" has drifted into generic use for any chatbot, the way "Kleenex" means any tissue, but strictly speaking it refers only to OpenAI's family; Llama, Claude, and Gemini are not GPTs.
Under the Hood
Architecturally, every GPT is a decoder-only transformer: input text is tokenized, embedded, and passed through a stack of self-attention layers, and training adjusts weights to minimize next-token prediction error across a massive text corpus. There is no encoder and no masked-token objective — that is the key contrast with BERT, the other famous transformer line, which reads text bidirectionally and was built for understanding tasks rather than generation. Because everything the model knows is baked into weights during pretraining, it has a hard knowledge cutoff and no built-in notion of truth, which is why it can state false things fluently; giving it fresh, factual context at inference time through RAG is the standard workaround. Generation itself is autoregressive: the model samples one token, appends it, and repeats, so long answers are literally hundreds of sequential predictions.