跳到主要内容Skip to main content

DPO(直接偏好优化)DPO (Direct Preference Optimization)

DPO (Direct Preference Optimization)

  • 模型能力Model capabilities
  • 第三站 · 用深Station 3 · Going Deeper
  • 引用核验 2026-11-17refs checked 2026-11-17
解决什么问题What problem it solves

用成对的「更好 / 更差」回答直接微调模型,省掉奖励模型和强化学习。Fine-tune directly on paired better and worse answers, without a reward model or reinforcement learning.

这是什么What it is

DPO 把 RLHF 的目标改写成分类损失:提高优选回答的概率、降低劣选回答的概率。

它不需要在训练中从模型采样,也不需要单独训练奖励模型。

DPO rewrites the RLHF objective as a classification loss: raise the probability of the preferred answer and lower the rejected one.

It needs neither sampling from the model during training nor a separately trained reward model.

DPO skips the reward model: paired preferred and rejected answers are used directly in a classification-style loss.
DPO 跳过奖励模型:直接拿「优选 / 劣选」成对数据,用一个分类损失把两者的概率拉开。DPO skips the reward model: paired preferred and rejected answers are used directly in a classification-style loss.
为什么需要知道Why it matters

想让模型更符合偏好但不想搭 PPO 那一套时,DPO 是更轻的选择;偏好数据的覆盖面仍然决定效果上限。When you want preference alignment without the PPO machinery, DPO is the lighter option. The coverage of preference data still caps the result.

别名Chinese aliases

直接偏好优化 · direct preference optimization

出处Sources
  • Rafailov 等 · DPO(arXiv 2305.18290)Rafailov et al., Direct Preference Optimization ↗
  • Ouyang 等 · InstructGPT(arXiv 2203.02155)Ouyang et al., Training language models to follow instructions with human feedback ↗