对齐与拒答训练Alignment and Refusal Training
Alignment and Refusal Training
解释模型的「不肯答」是怎么被训练进去的，以及这条边界为什么会随版本移动。Explains how a model's habit of saying no gets trained in, and why that line moves between versions.
预训练只教模型续写；会不会拒绝、在哪类请求上拒绝，是预训练之后那几步训练加进来的行为：监督微调、人类偏好数据、以及写成文本的规则目标。
OpenAI 的 Model Spec（2026-10-29 版）把回应方式写成可核对的档位：照做、给更安全的替代、说明理由后拒绝、直接拒绝；文中写明「当预期安全收益大于成本时」可以暂停、给替代或拒绝。
拒答住在权重里，不在开关里：Arditi 等人 2024 年的研究报告，移除模型内部某一个方向后模型不再拒绝有害请求，把同一方向加回去时连无害提问也会被拒。
所以边界会漂：换训练数据、换规则版本、换对齐方法，同一句提问的结局可能不同。这是训练配置的变化，不是你的提问写错了。
过度拒绝和漏答（该拒不拒）是同一条边界的两侧，都由训练强度与目标决定。
Pretraining only teaches continuation. Whether the model declines, and on which requests, comes from the steps that follow: supervised fine-tuning, human preference data, and the written rules a lab optimises against.
OpenAI's Model Spec (2026-10-29 revision) turns response modes into something you can check line by line: follow the instruction, offer a safer alternative, explain then refuse, or refuse outright. The text ties the choice to whether the expected safety benefit outweighs the cost.
Refusal lives in the weights, not in a switch. Arditi et al. (2024) report that removing one internal direction stops a model refusing harmful requests, while adding that same direction makes it refuse harmless ones.
So the boundary drifts. New training data, a new rule revision, or a different alignment method can change the outcome of the same prompt. That is a change in training configuration, not a mistake in your wording.
Over-refusal and under-refusal are two sides of one boundary, both set by training intensity and objectives.
把拒答当成训练产物，你才会去查它依据哪份规则文本、哪个版本、训练目标写了什么，而不是把它读成「模型坏了」或者「供应商心情」。看一家公司的安全姿态，公开的对齐规则与版本记录比单个提问的拒绝结果更有信息量。Treating refusal as a trained artefact sends you to the right documents: which rule text the model was aligned to, which version, what the training objectives say. When you want to read a lab's safety posture, its published alignment rules and version history tell you more than the outcome of one test prompt.
拒答是怎么练出来的
回到名词地图看第4站的其他 38 个词 →Back to the map: the other 38 terms in station 4 →