论文卡-HELMPaper Card - HELM
论文卡 · 约 8 分钟Paper card · about 8 min · 引用核验 2026-11-03（仅引用链接与日期） · refs checked 2026-11-03 (reference links and dates only)
「别只看一个总分」的评测框架论文。这张卡列清它要解决什么、框架怎么搭、以及它自己声明的取舍。The evaluation-framework paper behind 'don't trust a single score'. This card lays out the question, how the framework is built, and the trade-offs the authors declare.
研究问题：摘要原话——语言模型的能力、局限与风险「未被充分理解」（"their capabilities, limitations, and risks are not well understood"）；HELM 的目标是提高透明度，不是宣布某个模型最强。
框架：先对场景（用例）与指标（期望）做分类，再按覆盖度与可行性选取子集，并明说什么是缺失的（摘要点名：被忽视英语变体的问答、可信度指标）——「承认覆盖面有缺口」本身就是方法的一部分。
核心做法原话：对 16 个核心场景尽可能测 7 个指标（accuracy/calibration/robustness/fairness/bias/toxicity/efficiency），"ensuring metrics beyond accuracy don't fall to the wayside"——保证准确率之外的指标不被丢下。
作者自陈的取舍：选的是「广」不是「全」——摘要明说按 coverage 与 feasibility 选取，未覆盖处会标注；引用 HELM 时要连它的场景清单一起看，脱离场景的 HELM 排名没有意义。
它没有主张的：HELM 不是新基准榜单的替代品——它反对的正是单一口径；「HELM 总分第一」这种说法与论文立场相反。
Research question (abstract): LMs' "capabilities, limitations, and risks are not well understood"; HELM aims at transparency, not at crowning a strongest model.
Framework: taxonomize scenarios (use cases) and metrics (desiderata), select a subset by coverage and feasibility, and state what is missing (the abstract names QA for neglected English dialects and trustworthiness metrics) — admitting coverage gaps is part of the method.
Core approach verbatim: measure 7 metrics (accuracy, calibration, robustness, fairness, bias, toxicity, efficiency) across 16 core scenarios where possible, "ensuring metrics beyond accuracy don't fall to the wayside".
Declared trade-off: broad, not exhaustive — cite HELM together with its scenario list; a HELM ranking detached from scenarios is meaningless.
What it does not claim: HELM is not a replacement leaderboard — it opposes single-metric ranking itself; 'ranked first on HELM' contradicts the paper's stance.
例子与判断Example and interpretation
以下是教学情境，不是本站运行或实测记录。A teaching scenario, not a measured or executed site test.
把两个模型的结果按准确性、校准、稳健性等维度分别看。若你的场景重视某类错误，就追到对应任务和样本，而不是把不同指标压成一个未经解释的总分。Inspect dimensions such as accuracy, calibration and robustness separately. If a failure type matters, trace it to tasks and examples rather than collapsing different metrics into an unexplained total.
基准测试Benchmark 模型评测的局限Evaluation Limits 评测饱和Benchmark Saturation
AI 常见误区总索引AI Misconceptions: Index · 怎么判断 AI 新闻是不是炒作How to Tell if AI News Is Hype · 怎么评测 RAG 的效果How to Evaluate a RAG System · 怎么自建评测口径How to Build Your Own Eval