跳到主要内容Skip to main content

学习率Learning Rate

Learning Rate

  • 核心概念Core concepts
  • 第四站 · 看懂Station 4 · Under the Hood
  • 引用核验 2026-11-17refs checked 2026-11-17
解决什么问题What problem it solves

控制每次参数更新的步长,是训练中最敏感的超参数之一。Set the step size of each parameter update; one of the most sensitive hyperparameters in training.

这是什么What it is

太大容易震荡或发散,太小收敛慢甚至卡在平台。

常见策略包括 warmup、余弦退火、按验证集表现衰减。

Too large causes oscillation or divergence; too small converges slowly or stalls on a plateau.

Common schedules include warmup, cosine decay and reducing on validation performance.

On the same loss surface: too small a step never reaches the minimum; too large a step oscillates or jumps past it.
同一个损失曲面:步长太小走不到谷底,步长太大来回震荡甚至跨过谷底。On the same loss surface: too small a step never reaches the minimum; too large a step oscillates or jumps past it.
为什么需要知道Why it matters

换批大小、换优化器、做微调都要重新考虑学习率;微调通常比预训练小几个数量级。Changing batch size, optimizer or fine-tuning regime means reconsidering the learning rate. Fine-tuning usually uses rates orders of magnitude smaller than pretraining.

别名Chinese aliases

lr · 步长 · learning rate

出处Sources
  • PyTorch · torch.optim 文档PyTorch torch.optim documentation ↗
  • PyTorch · SGD 文档PyTorch SGD documentation ↗