GRPOGroup Relative Policy Optimization
Group Relative Policy Optimization
用同一道题的一组回答互相比较,代替一个单独训练的价值网络来估优势。Compare a group of answers to the same question instead of training a separate value network to estimate the baseline.
对同一个问题采样一组回答,用组内平均分作为基线,算每个回答的相对优势。
它属于强化学习,但省掉了 PPO 里与策略同规模的价值网络(critic)。
DeepSeekMath 提出,DeepSeek-R1 等推理模型把它带进了大规模后训练。
Sample a group of answers per question, use the group mean as the baseline, and compute each answer’s relative advantage.
It is reinforcement learning without the PPO critic, a value network as large as the policy.
Proposed in DeepSeekMath; DeepSeek-R1 brought it into large-scale reasoning post-training.
大模型后训练最贵的是「判断一个回答比平均好多少」。GRPO 用组内比较把这件事变成可计算的相对值,显存和工程复杂度都降了一档;理解它,才能看懂「推理能力是训出来的」这句话到底在训什么。The expensive part of post-training is judging how much better one answer is than average. GRPO turns that into a within-group relative value, cutting memory and engineering complexity; it is how “reasoning is trained” actually works.
回到名词地图看第5站的其他 40 个词 →Back to the map: the other 40 terms in station 5 →