训练语料Training Corpus
Training Corpus
模型「读过」的海量文本汇编：它的来源和毛病，决定了模型的一切表现。The massive text compilation a model has 'read': its sources and flaws shape everything the model does.
大模型的预训练语料是对海量公开网络文本的清洗汇编。
Common Crawl 是代表性来源。它是一家非营利组织，2007 年成立，维护免费开放的网页存档。
GPT-3 论文承认大规模网络语料带来方法学问题，语料构成细节见论文 §2.2。
语料里的偏见、错误、过时信息，都会原样带进模型。
A large model's pre-training corpus is a cleaned compilation of vast public web text.
Common Crawl is a representative source: a non-profit founded in 2007 maintaining a free, open repository of web crawl data.
The GPT-3 paper acknowledges methodological issues from training on large web corpora; corpus details are in §2.2.
Biases, errors, and outdated information in the corpus carry straight into the model.
「模型知道什么」上限就是语料。理解语料从哪来、有什么毛病，才理解模型为什么有知识截止和偏见。What a model knows is capped by its corpus. Understanding where it comes from and what's wrong with it explains knowledge cutoffs and bias.