论文卡-InstructGPTPaper Card - InstructGPT
论文卡 · 约 8 分钟Paper card · about 8 min · 引用核验 2026-11-03（仅引用链接与日期） · refs checked 2026-11-03 (reference links and dates only)
「对齐」进入主流训练的标志性论文。这张卡列清它要解决什么、关键数字在哪、以及作者自己列的局限在哪一节。The landmark paper that moved 'alignment' into mainstream training. This card pins down the question, the key numbers, and where the authors list their own limitations.
研究问题：摘要开篇原话 "Making language models bigger does not inherently make them better at following a user's intent"——模型变大不天然更听话，会生成不真实、有害或无用的输出。
核心证据：1.3B 参数的 InstructGPT 输出被标注者偏好于 175B GPT-3（摘要原话 "despite having 100x fewer parameters"，参数量少 100 倍）——对齐质量与参数量解耦，这是全文最硬的一条。
作者自陈局限：第 5 节（当日核对 HTML 版含 limitations 5.3 / open questions 5.4 / broader impacts 5.5 小节）；摘要还明说毒性只有小幅改善、偏见没有改善（"small improvements in toxicity over GPT-3, but not bias"）。
对评测读者的关键一句（摘要）："Public NLP datasets are not reflective of how our language models are used"——公开基准反映不了真实用法。这条直接支撑本站「榜单怎么读」的立场。
它没有主张的：RLHF 解决了幻觉或偏见——论文自己说没有；「对齐=更安全」也不能从这篇直接推出，作者列的是开放问题。
Research question: the abstract opens with "Making language models bigger does not inherently make them better at following a user's intent" — bigger models can be untruthful, toxic, or simply unhelpful.
Core evidence: labelers prefer outputs from the 1.3B InstructGPT to the 175B GPT-3 — "despite having 100x fewer parameters" (abstract). Alignment quality decouples from parameter count; this is the paper's hardest result.
Author-stated limitations: Section 5 (the HTML version contains 5.3 limitations / 5.4 open questions / 5.5 broader impacts); the abstract itself says toxicity improved only slightly and bias not at all.
Key line for evaluation readers (abstract): "Public NLP datasets are not reflective of how our language models are used" — this directly supports our 'How to Read a Leaderboard' stance.
What it does not claim: RLHF did not solve hallucination or bias — the paper says so; 'alignment = safer' cannot be inferred from this paper either.
例子与判断Example and interpretation
以下是教学情境，不是本站运行或实测记录。A teaching scenario, not a measured or executed site test.
分别记录监督数据、偏好数据和优化流程的作用。人类偏好评测说明的是指定设置中的选择，不代表每一条回答都真实，也不意味着偏好优化已经解决所有安全问题。Record the roles of supervised data, preference data and optimization separately. Human preference comparisons apply to specified settings; they do not establish universal truthfulness or complete safety.
AI 常见误区总索引AI Misconceptions: Index · 怎么判断 AI 新闻是不是炒作How to Tell if AI News Is Hype · 怎么评测 RAG 的效果How to Evaluate a RAG System · 怎么自建评测口径How to Build Your Own Eval