跳到主要内容Skip to main content

GRPOGroup Relative Policy Optimization

Group Relative Policy Optimization

  • 核心概念Core concepts
  • 第五站 · 跑起来Station 5 · Running It
  • 引用核验 2026-11-18refs checked 2026-11-18
解决什么问题What problem it solves

用同一道题的一组回答互相比较,代替一个单独训练的价值网络来估优势。Compare a group of answers to the same question instead of training a separate value network to estimate the baseline.

这是什么What it is

对同一个问题采样一组回答,用组内平均分作为基线,算每个回答的相对优势。

它属于强化学习,但省掉了 PPO 里与策略同规模的价值网络(critic)。

DeepSeekMath 提出,DeepSeek-R1 等推理模型把它带进了大规模后训练。

Sample a group of answers per question, use the group mean as the baseline, and compute each answer’s relative advantage.

It is reinforcement learning without the PPO critic, a value network as large as the policy.

Proposed in DeepSeekMath; DeepSeek-R1 brought it into large-scale reasoning post-training.

为什么需要知道Why it matters

大模型后训练最贵的是「判断一个回答比平均好多少」。GRPO 用组内比较把这件事变成可计算的相对值,显存和工程复杂度都降了一档;理解它,才能看懂「推理能力是训出来的」这句话到底在训什么。The expensive part of post-training is judging how much better one answer is than average. GRPO turns that into a within-group relative value, cutting memory and engineering complexity; it is how “reasoning is trained” actually works.

出处Sources
  • DeepSeekMath(论文 §4)DeepSeekMath §4 (paper) ↗
  • DeepSeek-R1(技术报告)DeepSeek-R1 technical report (paper) ↗