CUDA
Why it matters
CUDA is a big part of why NVIDIA dominates AI hardware. It's not just that NVIDIA's chips are fast. It's that nearly two decades of software, research code and developer skills are built on CUDA. AlexNet, the 2012 network that kicked off the deep learning boom, was trained on two NVIDIA gaming GPUs using CUDA. Every major framework since has treated CUDA as its first and best-supported target.
That's the famous CUDA moat. Competing chips from AMD, Intel and others can match NVIDIA on paper, but they must also convince developers to move their software, which is the harder problem.
For builders, CUDA shows up in daily life: matching PyTorch to the right CUDA version, installing drivers, and the most common error in AI, CUDA out of memory.
Deep Dive
GPUs were built to draw graphics: thousands of small cores working in parallel on pixels. In 2006, NVIDIA introduced CUDA, a way for programmers to use those cores for any kind of math, not just graphics. CUDA includes a programming model (an extension of C and C++), a compiler, drivers and, crucially, a huge set of optimized libraries such as cuBLAS for matrix math and cuDNN for neural networks.
Today, when PyTorch runs your model on an NVIDIA GPU, it's calling CUDA and its libraries under the hood.
How CUDA Code Works
A CUDA program runs kernels: functions executed by thousands of threads in parallel. Threads are grouped into blocks, and blocks into a grid. Each thread handles a small slice of the work, like one part of a matrix multiplication. Writing fast kernels means managing memory carefully: moving data from the GPU's large but slower memory (HBM) into small, very fast on-chip memory, and reusing it as much as possible.
This is exactly what made Flash Attention so important: it reorganized the attention computation so it stays in fast on-chip memory, giving big speedups without changing the math.
Tensor Cores
Since the Volta generation in 2017, NVIDIA GPUs include Tensor Cores, special units for the small matrix multiplications at the heart of neural networks. They're why lower-precision formats such as FP16, BF16, FP8 and FP4 give such large speedups: each new generation adds support for smaller numeric formats that Tensor Cores process faster.
Versions and Compatibility
CUDA has two numbers that matter: the driver version installed on the machine and the toolkit (or runtime) version a library was built against. A newer driver generally runs older toolkits, but not the reverse. Most install headaches come from mismatches between the driver, the CUDA toolkit and the PyTorch build.
The Alternatives
- ROCm and HIP: AMD's equivalent, with HIP designed to make porting CUDA code straightforward.
- Triton: a Python-like language from OpenAI for writing GPU kernels without raw CUDA. It's used inside PyTorch's compiler.
- Metal: Apple's GPU framework, used by MLX and llama.cpp on Macs.
- Vendor-neutral efforts such as SYCL, oneAPI and Vulkan compute.
Most developers never write CUDA directly. But knowing it's there explains why hardware choices are about software as much as chips.