缩放定律Scaling Laws
Scaling Laws
损失随参数、数据、算力呈幂律下降的经验规律。它是大模型军备竞赛的理论依据。The empirical power-law by which loss falls with parameters, data, and compute — the theory behind the scaling arms race.
Kaplan 等人发现：损失随参数量、数据量、训练算力各自呈幂律变化。这一趋势跨越七个以上数量级。
该文主张固定算力下主要增大模型（并概括为「大模型可能比大数据更重要」）。
Hoffmann 等人发表 Chinchilla 论文。他们基于 400 余个模型的实验提出反驳。结论是算力最优时，参数量与训练 token 数应等比放大。
他们据此指出现有大模型普遍训练不足。
Kaplan et al. found loss follows power laws in model size, dataset size, and training compute, with trends spanning more than seven orders of magnitude.
That paper favored growing model size under a fixed compute budget, summarized as 'big models may be more important than big data'.
Hoffmann et al. (Chinchilla), from over 400 trained models, countered: compute-optimal training scales parameters and training tokens equally.
They concluded that existing large models were significantly undertrained.
「模型多大、喂多少数据」不是拍脑袋，背后有这两篇。各家训练配方转向海量数据，源头就在这。'How big, how much data' is not guesswork — these two papers are behind it. The industry's shift to massive data diets traces to them.
回到名词地图看第4站的其他 35 个词 →Back to the map: the other 35 terms in station 4 →