跳到主要内容Skip to main content

奖励模型Reward model

Reward model

  • 核心概念Core concepts
  • 第三站 · 用深Station 3 · Going Deeper
  • 引用核验 2026-11-19refs checked 2026-11-19
解决什么问题What problem it solves

把「哪个回答更好」变成一个可打分的模型,为强化学习提供信号。Turn “which answer is better” into a scalar score that can drive reinforcement learning.

这是什么What it is

输入是一个问题加一个回答,输出是一个标量分数。

训练数据通常是人类或 AI 对多个回答的排序。

奖励模型不是最终产品,它是训练策略模型时的「裁判」。

Input is a question plus an answer; output is one scalar score.

Training data is usually human or AI rankings over multiple answers.

It is not the product model — it is the judge used while training the policy.

为什么需要知道Why it matters

奖励模型打分会直接决定模型学什么。裁判有偏见,模型就会学会讨好偏见。The reward model defines what the model learns to chase. A biased judge teaches the model to please the bias.

出处Sources
  • Learning to summarize from human feedback(arXiv)Learning to summarize from human feedback (arXiv 2009.01325) ↗