怎么搭一条最小评测流程How to Set Up a Minimal Evaluation Pipeline
微型指南 · 约 8 分钟Guide · about 8 min · 引用核验 2026-11-06（仅引用链接与日期） · refs checked 2026-11-06 (reference links and dates only)
不必攒几十项基准。一条能用的评测流水线只要五件东西：样本、口径、判法、记录、复跑。You do not need dozens of benchmarks. A usable evaluation pipeline needs five things: samples, criteria, judgment method, records, reruns.
样本从自己的工作里挑 20–50 条，记下来源与场景，比抄一份公开基准更贴近你的用法；留出一部分不参与改提示，专门用来看改动前后。
口径写成人能直接判的问题：不是「这条回答好不好」，而是「它有没有把材料里没写的期限当成依据」。
判法分两层：机器能判的（字段是否齐、引用编号是否真的存在）先跑脚本；判不了的留人工，并记下人工判了多大比例。
记录要能复盘：模型版本、提示版本、样本集版本、日期、每条的对错与理由，缺哪一格，下次就没法跟这一轮比。
换掉任何一环都用同一套样本复跑；同一输入跑两三遍看波动，波动本身也是记录。
举个例子（虚构数据）：输入是 30 条「合同条款问答」样本 → 处理是脚本查引用编号是否存在、人工判断结论是否与条款一致 → 结果判断是这一批 24/30 条通过（虚构计数，不是实测结果）→ 下一步先看错的是引用缺失还是判断偏差：前者改检索与提示，后者改验收口径，别一起动。
样本会老：用户问法变了、政策文本换了，就补新样本并重标一部分旧样本，否则流程还在跑、量的是过去。
Collect 20-50 samples from your own work and note where each came from; this tracks your usage better than copying a public benchmark. Hold some samples back from prompt tuning and use them only to compare before and after.
Phrase criteria as questions a person can answer directly: not "is this answer good", but "does it cite a deadline that the material never states".
Split judgment into two tiers: run scripts over what machines can decide (fields present, citation IDs that really exist); route the rest to humans and record how much needed a human.
Keep records you can revisit: model version, prompt version, sample-set version, date, per-item outcome and reason. Skip one field and the next round is not comparable.
Rerun the same sample set whenever any single piece changes; run one input two or three times to see the spread - the variance is part of the record.
Worked example (fictional data): input is 30 contract-clause Q&A samples -> processing checks citation existence by script and conclusion correctness by hand -> result this round: 24/30 passed (a made-up count, not a measured result) -> next step: first separate missing citations from wrong judgments; fix retrieval or prompt for the former, fix the acceptance rule for the latter, not both at once.
Samples age: when user phrasing or the policy text changes, add new samples and relabel part of the old set, otherwise the pipeline keeps measuring the past.
例子与判断Example and interpretation
以下是教学情境，不是本站运行或实测记录。A teaching scenario, not a measured or executed site test.
客服问答上线前，从真实会话里取 30 条样本，一半用来改提示、一半只用来看改动前后；脚本查引用编号是否真实存在，人判结论是否被材料支撑，并记下人工判了多少比例。Before a support chatbot ships, take 30 samples from real conversations: half to tune prompts, half held back to compare changes. A script checks that cited ids exist; people check whether the answer is supported, and the human-judged share is recorded.
做完例子后怎么检查How to check your practice
先用自己的话解释上面的判断，再检查输入条件，最后回原文验证。说不清原因时回概念；缺日期、字段或权限时补输入；结论无出处时先撤回结论。Explain the example in your own words, inspect input constraints, then verify against the source. Revisit concepts if you cannot explain why; supply missing dates, fields or permissions; withdraw unsupported conclusions.
按这篇主题，先从“结果缺验证”练习查起For this topic, start with unchecked output · 查看全部六个情境与理由See all six scenarios and their reasoning
评估Evaluation, Evals 基准测试Benchmark 模型评测的局限Evaluation Limits 数据污染Data Contamination
- OpenAI Evals 官方指南OpenAI Working with Evals ↗
- Liang 等《Holistic Evaluation of Language Models》（arXiv 2211.09110）Liang et al., Holistic Evaluation of Language Models (arXiv 2211.09110) ↗
- HelloFresh：用持续数据流评测对抗污染与过拟合（arXiv 2406.03428）HelloFresh: continuous data streams against contamination and overfitting ↗
AI 常见误区总索引AI Misconceptions: Index · 怎么判断 AI 新闻是不是炒作How to Tell if AI News Is Hype · 怎么评测 RAG 的效果How to Evaluate a RAG System · 怎么自建评测口径How to Build Your Own Eval