蒸馏与知识来源Distillation and Knowledge Provenance
Distillation and Knowledge Provenance
说明小模型的能力从哪个已有模型学来，以及这条来源线为什么常常说不清。Where a small model's capability came from, and why that lineage is often impossible to confirm.
蒸馏的原始定义：Hinton 等人 2015 年的论文写的是把一个「笨重模型（cumbersome model）」ensemble 的知识压进单个可部署的模型，靠的是匹配软化的输出分布，而不是只匹配标签。
工程上可用的来源有三类：教师模型的输出文本、教师模型的概率分布（需要能取到 logprobs）、公开语料与上一代自研模型。
只调用公开 API 也能近似出软标签：同一提示多次采样统计频率。但能否这么做，先由条款决定，再由技术决定。
多数厂商的服务条款限制把服务用于训练竞品模型。Anthropic 商业服务条款把「访问服务以构建竞争产品或服务，包括训练竞争性 AI 模型」列为不允许用途，同时写明客户保留对 Inputs 的权利并拥有 Outputs。
来源是否公开：模型卡与训练数据摘要通常不写教师是谁；未写出来就是未知，不能靠模型名字里带 -distill 反推。
The original definition is Hinton et al. (2015): compress the knowledge of a cumbersome model ensemble into a single deployable model by matching softened output distributions rather than hard labels alone.
In practice there are three kinds of source: a teacher model's output text, its probability distributions (which requires access to logprobs), and public corpora or the lab's own earlier model.
Soft labels can be approximated through a public API by sampling the same prompt many times and counting frequencies. Whether you may do that is decided by contract first and by feasibility second.
Most providers restrict using the service to train rival models. Anthropic's Commercial Terms list accessing the services to build a competing product or service, including to train competing AI models, as prohibited, while stating that the customer keeps rights in its Inputs and owns its Outputs.
Disclosure is the weak point: model cards and training-data summaries usually do not name a teacher. If it is not written down, the provenance is unknown, and a "-distill" suffix in a model name is not evidence.
「小模型接近大模型」这句话背后可能是一条蒸馏线。你要么查到来源，要么把它记成未知并承担说不清的风险；同时，用别人的输出去喂自己的模型，风险主要在合同条款那一侧，而不是在能不能做到那一侧。A claim that a small model nearly matches a big one may sit on top of a distillation line. Either you find the source or you record it as unknown and live with not being able to explain it. And when someone feeds another provider's outputs into their own training run, the exposure is on the contract side, not on the technical-possibility side.
能力从哪来
回到名词地图看第4站的其他 38 个词 →Back to the map: the other 38 terms in station 4 →