离线评测与在线评测Offline vs Online Evaluation
Offline vs Online Evaluation
分清上线前跑固定题集和上线后看真实流量，各自能证明什么、证明不了什么。Separate rerunning a fixed dataset before launch from scoring real traffic after it, and see what each can and cannot prove.
离线评测是在固定题集上重跑同一批输入，可复现、适合比较版本先后。在线评测是对已经发生的真实交互打分，看的是线上分布有没有变。微软那份文档把这句话写成部署前用它验证行为、部署后用它监控质量。
两者的数据来源不一样。Foundry 支持四种来源：合成数据、已有数据集、已有对话、已有 traces。后两种只有等真实流量跑起来以后才存在，所以在线评测天然是事后。
在线评测更贴近你关心的东西，但它不会把同一批输入重跑一遍。分数掉了，你分不清是提示词改坏了还是当天的输入更难。要控制这点，只能一次只放一个变更，并留一版可比的参照。
离线题集通过不代表线上不出事：真实输入更长也更脏、工具更容易超时、资料更容易过期。离线分数与线上表现之间的通用换算关系，官方没有给出。
Offline evaluation reruns the same inputs against a fixed dataset: reproducible, good for comparing one version with the next. Online evaluation scores interactions that really happened, which is how you notice the production distribution has moved. Microsoft frames it as validating behaviour before deployment and monitoring quality afterwards.
The data sources differ. Foundry offers synthetic data, an existing dataset, existing conversations and existing traces; the last two only exist once real traffic has run, so online evaluation is always about the past.
Online scoring is closer to what you care about, but it does not replay the same inputs. When the number drops you cannot tell whether the prompt got worse or that day's traffic got harder. The only control is shipping one change at a time against a comparable reference.
Passing the offline set does not mean production is safe: real inputs are longer and messier, tools time out more, and material goes stale. No vendor publishes a general conversion from offline scores to online outcomes.
只做离线评测的团队会以为已经安全；只看线上监控的团队说不清为什么变差。两件事各管一半，讲清楚才不会被一句「我们评测过了」带过去。Teams that only run offline evals assume they are covered; teams that only watch online metrics cannot say why quality slipped. The two halves answer different questions, which is what stops one line about we ran the evals from closing the discussion.