跳到主要内容Skip to main content

词表Vocabulary

Vocabulary

  • 核心概念Core concepts
  • 第一站 · 进门Station 1 · Getting In
解决什么问题What problem it solves

模型认识的全部字符集合。大小决定了它懂不懂中文、懂不懂代码。The full set of characters a model knows. Its size decides whether it handles Chinese or code well.

这是什么What it is

模型输出时能用的所有字符或片段的集合。

词表越大,模型需要学的概念越多,对冷门语言和代码的支持通常越好。

The set of characters or pieces a model can use when producing output.

The larger the vocabulary, the more concepts the model must learn, and usually the better support for rare languages and code.

为什么需要知道Why it matters

解释了一个常见现象:为什么小模型对中文和代码的表现明显变差。原因是词表里根本没有足够的碎片可用。It explains a common observation: small models get markedly worse at Chinese and code — there simply are not enough pieces in the vocabulary.

出处Sources
  • Hugging Face 分词器课程Hugging Face NLP Course: Tokenizers chapter ↗
  • Cornell CS5740:分词讲义Cornell CS 5740: Tokenization lecture slides ↗