If you've been living under a rock, half the fine-tuning posts on X use LoRA, QLoRA and DoRA interchangeably. They are not the same thing. They're not even solving the same problem.
One is a training trick, one is a memory trick, and one is a quality upgrade stacked on top of the first. This is the comparison I wish someone had handed me before the GPU bill hit — do not @ me.
What Is LoRA (Low-Rank Adaptation)?
LoRA is the technique everything else here builds on. Instead of updating billions of base weights, it freezes the entire model and trains two tiny matrices whose product gets added to the original weight matrix.
The base model never changes. You ship a small adapter file instead of a whole new model, which means cheap training, tiny artefacts, and almost-full-fine-tune quality for most tasks.
How LoRA works
You pick target layers — the attention projections q_proj, k_proj, v_proj and o_proj are the usual suspects — and inject a low-rank matrix pair into each. The rank r controls adapter capacity, and r=16 is the boring, sensible default that keeps working.
Setup is one install and one config object. Behold:
pip install peft transformers bitsandbytesfrom peft import LoraConfig, get_peft_model
# Plain LoRA
lora = LoraConfig(
r=16, lora_alpha=32,
target_modules=["q_proj","k_proj","v_proj","o_proj"],
lora_dropout=0.05, bias="none", task_type="CAUSAL_LM",
)
model = get_peft_model(base_model, lora)That's genuinely the whole ceremony. You train it like a normal model, except only the adapter weights receive gradients — the frozen base just sits there being enormous.
When to use LoRA
Plain LoRA has the fastest step time of the three, so it wins whenever training speed is the bottleneck. If your model already fits in memory and your dataset is huge — think 100k+ examples — vanilla LoRA at a higher rank is usually the right call.
What Is QLoRA?
QLoRA is LoRA with one twist: the frozen base model is quantised to 4-bit so the whole thing fits on consumer cards. The adapter maths is identical — you've just compressed the model underneath it.
How QLoRA works
You load the base in 4-bit via bitsandbytes — nf4 quant type, bfloat16 compute dtype — then attach ordinary LoRA adapters on top. The quantisation only touches the frozen weights; the adapters themselves still train in higher precision.
The BitsAndBytesConfig in the code block further down is the entire trick. Everything after that line is standard LoRA training.
When to use QLoRA
When the model doesn't fit. That's it — that's the use case. QLoRA buys you the smallest VRAM footprint of the three, and you pay for it with a slightly lower quality ceiling from the 4-bit base.
It's also the reason fine-tuning on consumer hardware is a real workflow now rather than a meme — I've written up fine-tuning on an M4 Mac if that's your hardware situation.
What Is DoRA?
DoRA decomposes each weight matrix into a magnitude and a direction, then applies the LoRA update only to the direction while training the magnitude separately. That decomposition is closer to how full fine-tuning actually moves weights, which is why it tends to land better results.
How DoRA works
Plot twist: in PEFT, DoRA is a one-flag upgrade. Same LoraConfig, flip use_dora=True, done.
from peft import LoraConfig
# DoRA — one flag, better quality at the same rank
dora = LoraConfig(
r=16, lora_alpha=32,
target_modules=["q_proj","k_proj","v_proj","o_proj"],
use_dora=True, # the magic
lora_dropout=0.05, bias="none", task_type="CAUSAL_LM",
)
# QLoRA — load base in 4-bit, then attach LoRA on top
from transformers import BitsAndBytesConfig
bnb = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype="bfloat16",
)Same rank, same target modules, same training loop. The extra magnitude path does the heavy lifting behind the scenes, and you barely change a line.
When to use DoRA
Small datasets are where DoRA earns its keep. In the 2-5k example range — a customer support adapter, a tone-of-voice tune — DoRA consistently beats vanilla LoRA at the same rank.
I've measured it on the Rebuild Relief ticket corpus and it's not subtle. If your dataset is small enough that every example matters, DoRA is free quality you're leaving on the table.
LoRA vs QLoRA vs DoRA: Which Should You Use?
Two questions settle it: does the model fit on your hardware, and how big is your dataset?
If the model doesn't fit, QLoRA — there is no second option, quantise the base and move on. If it fits and you're under 50k examples, DoRA at r=16 is the default. If it fits and you're training on a mountain of data where wall time hurts, plain LoRA gives you every minute of step speed back.
Notice that none of these answers is "full fine-tune". For adapter-shaped problems, adapters win on cost so hard that full fine-tuning needs a genuinely special reason to exist in your pipeline.
The Limits Nobody Puts in the Tutorial
DoRA is slower to train — roughly 1.3-1.5x the step time of plain LoRA, thanks to that extra magnitude path. On a tiny dataset you won't notice; at 100k+ examples the wall time adds up and the quality edge stops paying for itself.
QLoRA's 4-bit base costs you a bit of ceiling on quality. Most of the time that's acceptable; occasionally it's the difference between an adapter that ships and one that quietly gets deleted.
And no adapter technique rescues a bad dataset. Cranking the rank because outputs are mediocre is treating the symptom — the fix is almost always better examples, not more trainable parameters.
The Verdict
Default to DoRA at r=16 for any fine-tune under 50k examples. Reach for QLoRA when the model doesn't fit. Stick with plain LoRA when you need every minute of training speed back.
Three letters, three jobs — pick on purpose. And once the adapter is trained, serve it locally: my Ollama local LLM workflow covers that side of the pipeline.