训练三阶段SFT → Reward Model → RL
SFT → Reward Model → RL
解决什么问题What problem it solves
聊天模型是怎么练成的：先会续写，再学会好好回答。How chat models are trained: first they learn to continue text, then they learn to answer well.
这是什么What it is
第一阶段是预训练：海量文本上学续写。
第二阶段训练奖励模型：人给多个回答排序，模型学会判断好坏。
第三阶段是强化学习：按奖励模型的打分继续优化。
Stage one is pretraining: learning to continue text from massive corpora.
Stage two trains a reward model: humans rank candidate answers, and a model learns to judge quality.
Stage three is reinforcement learning: keep optimizing against the reward model's scores.
为什么需要知道Why it matters
ChatGPT 一类模型的「听话」来自后两阶段，不是预训练。较新的 DPO 把后两阶段合并成一步，不用单独训奖励模型。What makes ChatGPT-style models "follow instructions" comes from stages two and three, not pretraining. The newer DPO merges the last two stages into one, with no separate reward model.