LoRALoRA (Low-Rank Adaptation)
LoRA (Low-Rank Adaptation)
省钱的微调法：冻住原权重，只训练一小块「补丁」。Budget fine-tuning: freeze the original weights, train only a small 'patch'.
LoRA 冻结预训练权重，往每层注入可训练的低秩分解矩阵。
对比 GPT-3 175B 全量微调：可训练参数约少一万倍，显存需求约降三倍。
训练完可把低秩矩阵并入原权重，不增加推理延迟。
社区里大量消费级显卡上跑出来的微调模型用的就是它。
LoRA freezes pre-trained weights and injects trainable low-rank decomposition matrices into each layer.
Versus full fine-tuning of GPT-3 175B: about 10,000x fewer trainable parameters and roughly 3x less GPU memory.
After training the low-rank matrices can be merged into the weights, adding no inference latency.
Many community fine-tunes trained on consumer GPUs use it.
「在自家显卡上微调」从科幻变成日常，LoRA 是分水岭。选微调方案时它是默认起点。LoRA is the watershed that made 'fine-tune on your own GPU' routine. It is the default starting point when choosing a fine-tuning approach.
回到名词地图看第2站的其他 35 个词 →Back to the map: the other 35 terms in station 2 →
- Hu 等《LoRA: Low-Rank Adaptation of Large Language Models》（2021）Hu et al., LoRA: Low-Rank Adaptation of Large Language Models (2021) ↗