PEFT vs Full Fine-Tuning

PEFT is the default for most LLM fine-tuning, while full fine-tuning fits edge cases needing deeper model change.

Comparing PEFT and full fine-tuning helps teams decide how to adapt large language models without overspending on GPUs or losing too much accuracy.

At a glance

PEFT

PEFT works because most downstream tasks do not require rewriting the model from scratch. LoRA adds a low-rank update on top of frozen weights, so you are training a tiny slice of the network rather than every parameter. On a 13B model, that means roughly 26 million trainable parameters at r=16, which is small enough to fit on a single GPU and large enough to recover most of the performance of full fine-tuning.

The bigger advantage is operational, not just mathematical. QLoRA pushes the base model into 4-bit storage, then dequantizes it on the fly while training bf16 adapters. That makes previously awkward jobs practical on a 24 GB card, which is why PEFT became the default for teams that need to iterate quickly, support many customer variants, or work without a large GPU cluster.

Full fine-tuning

Full fine-tuning still matters when the task is not a small adaptation but a deep behavioral shift. If you are doing initial instruction tuning, extending the model to a new language, or chasing the last 1 to 2 percentage points on a frontier benchmark, updating all weights gives you more room to move. That extra freedom is exactly why it costs so much more in memory and compute.

The catch is that full fine-tuning scales badly in production. A 13B model in fp16 is already around 26 GB before optimizer states, gradients, and activations are counted, which is why a single run can require multiple A100s. It is the right tool when accuracy or model transformation is the priority, but it is a poor default when you need frequent retraining, per-customer customization, or low-friction deployment.

When to pick what

If you are a product team adapting a strong base model to a domain like legal, support, finance, or code, pick PEFT. It gives you most of the quality gain with far less GPU spend, and it makes it realistic to maintain separate adapters for different customers or internal teams.

If you are a research team or frontier lab trying to change the model’s core behavior, pick full fine-tuning. It is also the better choice when you need to alter language coverage, modality, or tokenization behavior in a way that low-rank updates are unlikely to capture well.

If you are cost-constrained, moving fast, or planning to serve many variants, PEFT is the safer default. If you only have one model to ship and the last bit of benchmark gain matters more than infrastructure cost, full fine-tuning can still be justified.

Default to PEFT, especially LoRA or QLoRA, unless you are doing foundational training or need a truly deep model rewrite.