跳到主要内容Skip to main content

Byteification(字节化)Byteification

Byteification

  • 模型能力Model capabilities
  • 第三段 · 用深Stage 3 · Going Deeper
  • 引用核验 2026-11-21refs checked 2026-11-21
解决什么问题What problem it solves

把已经训练好的子词模型改造为直接读写 UTF-8 字节的模型,去掉固定分词器,同时尽量保留原模型的能力。Retrofit an already trained subword model to read and write UTF-8 bytes directly, removing the fixed tokenizer while preserving as much of the original model’s capability as possible.

这是什么What it is

Byteification 是一种特殊的 tokenizer transfer:用两阶段训练把子词模型改成字节级模型,先复制原模型行为,再端到端适配。

它不等于从头训练 byte-level 模型;重点是复用原模型的参数与生态,降低改造成本。

Byteification is a special case of tokenizer transfer: a two-stage procedure first reproduces the source model’s behaviour, then adapts end to end to byte-level operation.

It is not training a byte-level model from scratch; the point is to reuse the source model’s parameters and ecosystem and lower the cost of conversion.

为什么需要知道Why it matters

分词器会把代码、数学、生物序列等细粒度信息切碎,也会让新语言、新符号需要扩表。字节化让模型直接处理原始字节,对字符级任务和多语言扩展更灵活。Tokenizers fragment fine-grained information in code, mathematics and biological sequences, and new languages or symbols often require vocabulary expansion. Byte-level operation gives the model raw bytes and more flexibility for character-level work and multilingual extension.

别名Chinese aliases

字节化 · 字节级语言模型 · tokenizer-free

出处Sources
  • Retrofitting language models to operate over bytes · Nature(2026)Retrofitting language models to operate over bytes, Nature 2026 ↗
  • Byte Language Models · arXiv 2610.05978Byte Language Models: Scaling, Emergent Abstractions, and Information Allocation ↗