Skip to main content
Zubnet AI › Learn › Wiki › PEFT
Training

PEFT

Also known as: Parameter-Efficient Fine-Tuning
A family of fine-tuning methods that train only a small number of extra or selected parameters instead of the whole model.

Why it matters

PEFT is what made fine-tuning accessible. With methods like LoRA and QLoRA, people fine-tune 7B to 70B models on a single GPU. The output is a small adapter file, often tens of megabytes, instead of a multi-gigabyte model. You can keep one base model and swap adapters per customer or task, and serving systems like vLLM can serve many LoRA adapters on one base model at the same time.

Because the original weights stay frozen, PEFT also reduces catastrophic forgetting: the model is less likely to lose its general skills while learning your task.

Deep Dive

Full fine-tuning updates every weight in a model. For a 70-billion-parameter model, that means storing gradients and optimizer states for 70 billion numbers, which takes a cluster of GPUs, and saving a full new copy of the model for every task. Parameter-efficient fine-tuning freezes the original model and trains only a tiny add-on, often well under 1% of the parameter count.

The results are surprisingly close to full fine-tuning for most practical tasks, at a fraction of the cost. PEFT is also the name of Hugging Face's popular library that implements these methods.

The Main Methods

  • LoRA (Low-Rank Adaptation): adds small low-rank matrices beside existing weight matrices and trains only those. By far the most used method.
  • QLoRA: LoRA on top of a base model quantized to 4 bits, so even big models fit in modest VRAM.
  • Adapters: small bottleneck layers inserted inside each transformer block. One of the earliest PEFT methods (2019), but they add a little inference latency, unlike LoRA, which can be merged into the base weights.
  • Prefix tuning and prompt tuning: learn a few "virtual tokens" that are prepended to the input. Very cheap, but generally weaker than LoRA on hard tasks.
  • IA³ and similar: learn per-channel scaling vectors. Extremely small, useful when you need thousands of adapters.
  • DoRA and other LoRA variants: tweak how the low-rank update is applied for slightly better quality.

What PEFT Is Good At, and What It Isn't

PEFT shines at teaching style, format and behavior: a house voice, a JSON schema, a specific classification scheme, tool-calling conventions. It's weaker at injecting large amounts of new knowledge. If you need the model to know your documentation, RAG is usually the better tool, sometimes combined with a light fine-tune for format.

Practical Choices

The main knobs for LoRA-style methods are the rank (how big the adapter is, commonly 8 to 64), which layers to adapt (all linear layers usually works best), the learning rate (higher than for full fine-tuning, often around 1e-4 to 2e-4) and the number of epochs. Start small and evaluate on held-out examples before scaling up.

Where It Fits

PEFT sits between prompt engineering and full fine-tuning. Try prompting and few-shot examples first. If that hits a ceiling, a PEFT run is the next step. Full fine-tuning is mostly reserved for labs and for large shifts like teaching a new language.

← All Terms
ESC