评测饱和Benchmark Saturation
Benchmark Saturation
基准被刷到接近满分、失去区分度——榜单为什么需要持续更新。Benchmarks approach perfect scores and lose discriminating power — why leaderboards need constant renewal.
Ott 等人对 3765 个基准的实证研究显示：相当比例的基准迅速接近饱和。
饱和加上静态数据公开可得，导致过拟合评测集。
社区因此主张用持续更新的数据流做评测，替代一考定终身。
看榜单先看基准的发布日期和饱和程度。
Ott et al.'s study of 3,765 benchmarks found a large fraction quickly trended towards near-saturation.
Saturation plus publicly available static data leads to overfitting the eval set.
The community therefore advocates evaluation on continuously updated data streams instead of one-shot exams.
When reading a leaderboard, check the benchmark's release date and how saturated it is.
「在 X 榜第一」的含金量取决于那个榜还剩多少区分度。饱和榜单上的名次差距可能毫无意义。The value of 'number one on X' depends on how much headroom that benchmark still has. Rank gaps on a saturated board can be meaningless.