评测集设计Eval Set Design
Eval Set Design
把「感觉变好了」变成一份能重复跑的题集，加上明确的判分规则。Turn a feeling of improvement into a fixed set of tasks with scoring rules anyone can run again.
一份题集至少要写清四件事：题目从哪来、判分依据是什么、输入字段怎么映射、用规则还是用另一个模型当裁判。Foundry 的评估流程把这些字段列成必需与可选两类：query、response 必需，context、ground_truth、tool_calls 可选。
题目要覆盖你真的会遇到的分布。官方文档的建议是别只挑成功样本，把有挑战的样本放进来，同时给数据集做版本管理，记下哪一版数据跑出哪一次结果。
判分要能重复。同一份题集换个人判、换个裁判模型判，结论差多少，这件事要先测一次再谈分数；否则改进和噪声分不开。
公开榜单只说明那份题集上的表现。HELM 的立场是同一个模型要在任务与指标两个维度一起看，单一分数盖住的东西太多。
A usable eval set states four things up front: where the items came from, what counts as the right answer or the basis for scoring, how inputs map to fields, and whether a rule or a judge model does the scoring. Foundry's evaluation flow makes this explicit — query and response required, context, ground_truth and tool_calls optional.
Items should cover the distribution you actually meet. Microsoft's guidance is not to score only the successful conversations but to include the hard ones, and to version the dataset so a result can be tied to the data that produced it.
Scoring has to repeat. Measure how much the verdict moves when a different person, or a different judge model, scores the same set; without that you cannot separate an improvement from noise.
Public leaderboards describe one dataset only. HELM's position is that a model should be read across scenarios and across metrics together, because a single number hides too much.
没有题集，你手上只有印象。有了题集，改提示词、换模型、加工具以后你能说清是变好还是变差，评审里也才敢引用它，而不是让人相信你的感觉。Without an eval set all you have is an impression. With one, a prompt change, a model swap or a new tool becomes a claim you can state either way — and something you can put in front of a reviewer instead of asking them to trust your gut.