Skip to main content
Zubnet AI › Learn › Wiki › HBM
Infrastructure

HBM

Also known as: High Bandwidth Memory
Stacked memory chips packaged right next to AI processors, providing the capacity and speed that large models need and forming a key bottleneck in the AI chip supply chain.

Why it matters

AI workloads are hungry for memory in two ways. Capacity decides what fits: the model's weights, the KV cache for every active conversation, the batch being trained. Bandwidth, meaning how fast data can move between memory and the processor, largely decides how fast tokens come out, because generating each token means reading through the model's weights.

HBM is also one of the tightest bottlenecks in AI hardware. Three companies make nearly all of it, production is sold out far in advance, and the number of AI accelerators the world can build depends heavily on how much HBM is available.

Deep Dive

High Bandwidth Memory is the type of memory used in data center AI chips. Instead of laying memory chips flat on a circuit board, HBM stacks many thin DRAM dies on top of each other and connects them vertically with thousands of microscopic wires called through-silicon vias (TSVs). The stack sits millimeters from the processor on a shared base, linked by more than a thousand data connections.

The result is much more memory, moving data much faster, in much less space than conventional memory can manage. When people say an NVIDIA H100 has 80 GB of memory or an AMD MI300X has 192 GB, that memory is HBM.

How It's Built

An HBM stack contains 8, 12 or 16 DRAM dies on top of a base die. TSVs carry signals vertically through the stack. The stack is then mounted beside the GPU or accelerator on a silicon interposer, using advanced packaging such as TSMC's CoWoS. That packaging step has been a bottleneck of its own.

Stacking is hard. The dies must be thinned and bonded precisely, heat has to escape from the middle of the stack, and one bad die can ruin all the others. Yields are lower and costs far higher than for ordinary DRAM.

Generations

  • HBM2 and HBM2E (about 2016 to 2020): NVIDIA V100 and A100.
  • HBM3 (2022 to 2023): NVIDIA H100 with 80 GB, AMD MI300X with 192 GB.
  • HBM3E (2024 to 2025): NVIDIA H200 with 141 GB, Blackwell (B200) GPUs with about 180 to 192 GB, Blackwell Ultra GPUs with 288 GB, AMD MI325X with 256 GB.
  • HBM4 (from 2026): NVIDIA's Rubin generation and AMD's MI400 series.

Each generation raises both capacity per stack and bandwidth. A current data center GPU moves several terabytes per second between memory and processor, several times what the GDDR memory on a consumer graphics card manages.

Who Makes It

  • SK hynix (South Korea) has led the market and been NVIDIA's main supplier.
  • Samsung (South Korea) is the largest memory maker overall, but it fell behind on HBM3E and has been catching up.
  • Micron (United States) is the third supplier and has gained share.

HBM takes roughly three times as much wafer capacity per bit as standard DRAM, so the industry's shift toward it squeezes the supply of ordinary memory. In late 2025 this contributed to sharp price increases for the regular DRAM used in PCs, phones and servers.

Geopolitics

Advanced HBM is covered by US export controls. Since December 2024, sales of advanced HBM to China have been restricted, alongside the controls on AI chips themselves. Building a domestic HBM supply has become a priority for China's chip industry.

Why It Matters to Practitioners

You'll never buy HBM directly, but it shapes what you can rent or buy. The memory on an accelerator sets the largest model, longest context and biggest batch it can handle, and memory bandwidth, more than raw compute, often sets inference speed. It's also a large part of why AI accelerators cost what they do, and why supply stays tight even when chip fabrication capacity grows.

← All Terms
ESC