TPU
Why it matters
Deep Dive
The TPU starts from one observation: deep learning is mostly matrix multiplication, so the fastest chip for it is one that does matrix multiplication and as little else as possible. Where a GPU carries thousands of flexible cores plus caches, schedulers, and instruction decoders, a TPU strips most of that away and dedicates its silicon to enormous multiply-accumulate arrays fed directly by high-bandwidth memory. Google deployed the first TPU internally in 2015 to run inference, and has iterated roughly every two years since: v2 through v4 scaled the design into multi-thousand-chip pods for training, v5p and v6 (Trillium) pushed pod scale further, and v7 (Ironwood, announced 2025) is the first generation built primarily for inference. The result is the longest-running large-scale alternative to NVIDIA hardware for frontier training — every generation of Gemini has been trained on TPUs, and for years nothing else outside NVIDIA trained models at that level, until Amazon's Trainium reached comparable scale: Anthropic reports training and serving Claude on more than a million Trainium2 chips.
How a Systolic Array Works
The heart of a TPU is the Matrix Multiply Unit (MXU), a systolic array — a grid of tens of thousands of simple multiply-accumulate elements wired directly to their neighbors. Weights are loaded into the grid in advance; activations then stream in from one edge and flow across the array in a regular rhythm, with each element multiplying, adding, and passing its result along. Because data moves neighbor-to-neighbor rather than through a cache hierarchy, almost no silicon is spent on caches, branch predictors, or instruction decode, so the chip spends nearly all its area and energy on arithmetic. The tradeoff is rigidity: the array is only efficient when the compiler can tile the workload into large, dense matrix multiplications — which is exactly what transformer training and inference consist of. Surrounding the MXU sit a vector unit for elementwise math, a scalar unit for control flow, and stacks of high-bandwidth memory, and since the second generation the chips have computed in bfloat16, a reduced-precision format Google introduced with the TPU that kicked off the modern mixed-precision era.
From v1 to Ironwood
Each TPU generation has followed the same trajectory: more compute per chip, more memory bandwidth, and a bigger pod. The 2015 v1 was inference-only, an 8-bit accelerator that quietly served production Google workloads — Google later revealed that AlphaGo's 2016 matches ran on TPUs. TPU v2 (2017) added floating-point training and introduced the pod: 256 chips lashed together with a custom interconnect so they behave as one machine, an early blueprint for distributed training at scale. V3 (2018) doubled down with 1,024-chip pods and liquid cooling, and v4 (2021) reached 4,096 chips per pod with reconfigurable optical links between them. The 2023 v5 generation split in two — the cost-optimized v5e and the high-end v5p — followed by v6 (Trillium) in 2024 and v7 (Ironwood) in 2025, which packs 9,216 chips into a single pod rated at 42.5 exaFLOPS and is the first TPU built primarily for inference rather than training.
The Software Stack
Hardware is only half the story; the other half is XLA, Google's compiler, which takes a model graph and maps it onto the systolic arrays. The first-generation stack assumed TensorFlow, and TensorFlow remains first-class, but the ecosystem has since centered on JAX, whose functional style matches XLA well, with PyTorch supported through the PyTorch/XLA bridge. This is where TPUs genuinely differ from GPUs: NVIDIA's CUDA has a decade of libraries, kernels, and institutional knowledge behind it, while everything on a TPU flows through a whole-graph compiler instead. When the compiler handles the model cleanly — standard transformer architectures usually qualify — utilization is excellent, often better than on GPUs. When the model has exotic ops or dynamic shapes, developers can spend days coaxing XLA, which is why choosing TPUs is as much a software decision as a hardware one and belongs in any serious AI infrastructure plan.
You Can't Buy One
A persistent misconception is that the TPU is Google's answer to the GPU in the same sense — a card you can order and rack in your own servers. It is not. Google has never sold TPUs as discrete hardware (the only exception is the small Edge TPU coprocessor for embedded devices), so the single way to use a full TPU is to rent a slice of a pod on Google Cloud. That shapes everything about adoption: there is no on-prem TPU cluster, no used market, and no way to benchmark one against a GPU in a private lab. It also means Google controls the full stack from silicon to data center, which is part of why the economics work. The broader lesson the industry took from the TPU is that custom silicon is viable: Groq was founded by the engineer who started the TPU project, and Amazon, Microsoft, Meta, and OpenAI have all since launched custom AI chip efforts of their own.
Who Actually Uses TPUs
The anchor tenant is Google itself: Google DeepMind trains and serves every Gemini generation on TPU pods, along with the rest of Google's production models, and TPUs have historically underpinned systems like AlphaGo and AlphaFold. The more interesting signal is who else shows up. Anthropic runs Claude on TPUs at very large scale and announced in 2025 a commitment to use up to a million of them; Apple trained the foundation models behind Apple Intelligence on v4 and v5p clusters and said so in its own technical reports. The pitch is straightforward — because an ASIC skips the general-purpose machinery, TPUs typically deliver more compute per dollar and per watt than GPUs when the workload fits, a gap that matters more as AI energy consumption becomes a binding constraint. The catch is the mirror image of the software section: teams whose stack is deeply invested in CUDA rarely move, while teams already on JAX or willing to standardize on a compiler-driven workflow often find TPUs the cheapest way to train something big.