子词分词Subword Tokenization
Subword Tokenization
把词切成更小的碎片，让模型遇到没见过的词也能拼出来。Cuts words into smaller fragments so a model can still assemble words it has never seen.
最常见的方案是 BPE：从字符出发，按出现频率反复合并相邻片段，得到一套子词。
「未登录词」问题由此缓解——整词不在词表里，它的碎片多半在。
The dominant scheme is BPE: start from characters, repeatedly merge the most frequent adjacent pairs into a subword inventory.
It eases the out-of-vocabulary problem — a whole word may miss the vocabulary, but its fragments usually are in it.
你看到模型把生僻词拆成奇怪片段，不是「不认识」，是子词分词在工作。词表大小与合并规则共同决定模型对中文、代码的友好度。When a model splits a rare word into odd fragments, it is not "failing to recognize" it — subword tokenization is at work. Vocabulary size and merge rules together decide how friendly a model is to Chinese and code.
回到名词地图看第4站的其他 35 个词 →Back to the map: the other 35 terms in station 4 →