MLAMulti-head Latent Attention
Multi-head Latent Attention
把长上下文里最占显存的 KV 缓存压成低秩潜向量,再在推理时展开。Compress the KV cache that dominates long-context memory into a low-rank latent vector, then expand it at inference.
每个头的 Key / Value 先投影到一个低秩潜空间,缓存只存这个潜向量。
推理时再把潜向量投影回各头的 Key / Value,省的是显存,不是计算。
DeepSeek-V2 提出,后续 V3 / V4 系列沿用并继续调整。
Each head’s Key/Value is first projected into a low-rank latent space; only that latent is cached.
At inference it is projected back to per-head Key/Value; the saving is memory, not compute.
Proposed in DeepSeek-V2 and carried forward in V3/V4 with further tuning.
长上下文的第一道墙不是算力,是 KV 缓存。MLA 让「1M 上下文」在单卡或多卡上变得可行,也改变了推理框架的缓存布局;不理解它,就看不懂为什么同样参数量的模型,长上下文成本会差一个数量级。The first wall in long context is not compute but KV cache. MLA makes 1M context feasible on one or a few GPUs and changes the cache layout in inference stacks; without it, the cost gap between similar-size models is hard to explain.
回到名词地图看第5站的其他 40 个词 →Back to the map: the other 40 terms in station 5 →