Low-Rank Adaptation

What Is LoRA

LoRA (Low-Rank Adaptation) is the most common parameter-efficient fine-tuning method for LLMs. Instead of updating all of a model’s billions of parameters during fine-tuning, LoRA freezes the base weights and trains two small low-rank matrices (A and B) per target layer. At inference time the LoRA matrices are added back to the base weights, producing the adapted model. Use LoRA when you need a task-specific or domain-specific model without paying the cost of full fine-tuning.

How It Works

• Frozen base model — original weights are not modified
• Rank decomposition — for each target weight matrix W, LoRA trains two small matrices A (down-projection) and B (up-projection) such that ΔW ≈ B × A
• Rank r — the bottleneck dimension between A and B, typically 4–64; controls capacity vs file size
• Merging — at inference, you can either keep LoRA as a separate adapter (swappable) or merge it into the base weights for zero overhead

Why It Matters

LoRA cut fine-tuning costs by ~10x and unlocked the entire “thousands of community-finetuned models” ecosystem on Hugging Face. A LoRA adapter is typically 10–200 MB versus a 70 GB base model — easy to ship, easy to swap. QLoRA combines LoRA with quantization for fine-tuning that fits on a single consumer GPU.

Examples

• PEFT (Hugging Face) — reference Python library, supports LoRA + many variants
• QLoRA — LoRA on top of 4-bit-quantized base weights, the most popular consumer fine-tuning method
• Stable Diffusion LoRAs — the same technique applied to image diffusion models; the basis of Civitai’s adapter ecosystem
• vLLM / TGI multi-LoRA — production serving with multiple LoRA adapters loaded simultaneously