怎么自建评测口径How to Build Your Own Eval
微型指南 · 约 8 分钟Guide · about 8 min · 引用核验 2026-11-03（仅引用链接与日期） · refs checked 2026-11-03 (reference links and dates only)
公共榜单测不出「它在我这活儿上行不行」。这篇讲怎么搭一套只属于你的评测：题目、判分、跑法。Public leaderboards can't answer "is it good at my job". How to build an eval that is yours alone: questions, grading, and how to run it.
从真实失败案例攒题：20-50 道你业务里真会遇到的输入，比抄公开题集有用。
每题写清「什么算对」：有标准答案的精确比对，开放题给评分标准或让强模型按标准打分。
固定题目版本和输入条件（模型、参数、提示词），否则分数变化说不清原因。
分开看两类错误：格式错（能用规则判）和内容错（要人或模型判），修法完全不同。
跑回归：换模型、改提示词后重跑同一套题，看哪些题从对变错——变错比变对更值得警惕。
警惕题目泄漏：公开题集可能混进训练数据，自建题能规避这一点，也要定期换题。
Collect questions from real failures: 20–50 inputs from your actual work beat copying a public benchmark.
Write down what counts as correct for each question: exact matching where there's a gold answer; a rubric — or a strong model grading against one — for open-ended ones.
Pin the question version and the input conditions (model, parameters, prompt), otherwise score changes have no explanation.
Separate the two error kinds: format errors (rule-checkable) and content errors (need a human or a model judge) — the fixes are completely different.
Run regressions: after swapping a model or editing a prompt, re-run the same set and watch which questions flip from right to wrong — flips to wrong deserve more attention than flips to right.
Watch for contamination: public benchmarks may sit in training data; your own questions dodge that, but rotate them periodically too.
例子与判断Example and interpretation
以下是教学情境，不是本站运行或实测记录。A teaching scenario, not a measured or executed site test.
给订单分类写三条验收：类别在允许列表中、不丢订单号、依据给定规则判断。另备混合评价和缺字段样本。先定义什么算错，再比较模型，避免看完结果才改评分标准。For order classification, require an allowed category, preserved order ID and adherence to supplied rules. Add mixed and incomplete inputs. Define mistakes before comparing models, rather than changing scoring after seeing results.
模型评测的局限Evaluation Limits 基准测试Benchmark 评测饱和Benchmark Saturation 评估Evaluation, Evals
AI 常见误区总索引AI Misconceptions: Index · 怎么判断 AI 新闻是不是炒作How to Tell if AI News Is Hype · 怎么评测 RAG 的效果How to Evaluate a RAG System · 榜单怎么读How to Read a Leaderboard