跳到主要内容Skip to main content

RLHF(人类反馈强化学习)RLHF

RLHF

  • 核心概念Core concepts
  • 第三站 · 用深Station 3 · Going Deeper
  • 引用核验 2026-11-19refs checked 2026-11-19
解决什么问题What problem it solves

先训练一个奖励模型学人类偏好,再用强化学习把语言模型往高分方向推。Learn a reward model from human preferences, then optimise the language model against that reward.

这是什么What it is

收集人类对多个回答的排序,训练奖励模型。

用 PPO 等算法优化策略模型,让奖励模型给高分。

同时加 KL 惩罚,防止模型为了高分偏离原模型太远。

Collect human rankings over multiple answers and train a reward model.

Optimise the policy with PPO or similar so the reward model scores it higher.

Add a KL penalty so the model does not drift too far from its starting point.

为什么需要知道Why it matters

它是「会说话」到「听话」的关键一步。理解 RLHF,才能理解偏好对齐、奖励黑客和拒答行为。It is the step from “can talk” to “follows instructions”. Understand RLHF and preference alignment, reward hacking and refusals become clearer.

出处Sources
  • InstructGPT 论文(arXiv)Training language models to follow instructions with human feedback (arXiv 2203.02155) ↗