MoE（混合专家）Mixture of Experts (MoE)
Mixture of Experts (MoE)
一种模型结构：每层放很多「专家」，每个词只用到其中少数几个。A model architecture: each layer holds many "experts," but each word uses only a few of them.
MoE 层由多个专家子网络加一个可训练的门控（路由）网络组成。
每个 token 只激活少数专家。
参数总量大增，但每 token 的计算量近似不变。
Switch Transformer 把路由简化成每 token 只选一个专家。
An MoE layer is several expert sub-networks plus a trainable gating (routing) network.
Each token activates only a few experts — total parameters grow hugely while per-token compute stays roughly constant.
Switch Transformer simplified routing to one expert per token.
DeepSeek、Mixtral 这类大模型用的就是它。「总参数」和「激活参数」是两个数，看模型别被前者吓到或骗到。DeepSeek and Mixtral use it. "Total parameters" and "active parameters" are two different numbers — do not let the first one intimidate or mislead you.
回到名词地图看第4站的其他 35 个词 →Back to the map: the other 35 terms in station 4 →