预训练Pre-training
Pre-training
解决什么问题What problem it solves
在海量文本上做「预测下一个词」，这是模型学到世界知识的阶段。Predicting the next token over vast text — the stage where a model absorbs world knowledge.
这是什么What it is
第一批训练，目标单一：续写文本。成本以亿计。
The first training run, with a single objective: continue the text. It costs hundreds of millions.
为什么需要知道Why it matters
它解释了为什么模型「什么都知道一点」。那是泛读的结果，不是为你的任务专门学的。It explains why a model "knows a bit about everything": that is the result of reading widely, not learning for your task.