QLoRA
Why it matters
Deep Dive
QLoRA, introduced by Tim Dettmers and colleagues at the University of Washington in 2023, combines two ideas. First, load the base model in 4-bit precision instead of 16-bit, cutting its memory use by about 4×. Second, keep those 4-bit weights frozen and train small LoRA adapters in higher precision on top. Gradients flow through the quantized model into the adapters, which are the only part that learns.
The headline result: a 65-billion-parameter model could be fine-tuned on a single 48 GB GPU, with quality matching 16-bit full fine-tuning on the benchmarks they tested.
The Three Tricks
The QLoRA paper introduced three pieces that together make 4-bit training work:
- NF4 (4-bit NormalFloat): a 4-bit data type designed for the bell-shaped distribution of neural network weights, so it loses less information than plain 4-bit integers.
- Double quantization: the small scaling constants used by quantization are themselves quantized, saving roughly another 0.4 bits per parameter.
- Paged optimizers: optimizer states can spill from GPU memory to CPU memory during memory spikes, avoiding out-of-memory crashes on long sequences.
Rough Memory Budget
For a 7B to 8B model: the 4-bit weights take about 4 to 5 GB, the LoRA adapters and their optimizer states add a few hundred megabytes, and activations depend on batch size and sequence length. In practice, 16 GB of VRAM is comfortable for an 8B model with gradient checkpointing, and around 48 GB handles a 70B model.
Trade-offs
QLoRA is slower than regular LoRA, often by 30% or more, because weights must be de-quantized on the fly for every forward and backward pass. Quality is usually very close to 16-bit LoRA, but not always identical, especially for tasks that are sensitive to small numeric details.
There's also a deployment subtlety. The adapters were trained against the 4-bit version of the model. You can merge them into a full-precision copy of the base model for serving, which usually works well, but it's worth testing the merged result rather than assuming it matches.
When to Use It
Use QLoRA when memory is the constraint: big model, small GPU. If you have plenty of VRAM, plain LoRA in 16-bit is faster and slightly simpler. If you're fine-tuning a small model (1B to 3B), the savings matter less.