跳到主要内容Skip to main content

MLAMulti-head Latent Attention

Multi-head Latent Attention

  • 硬件与部署Hardware & deployment
  • 第五站 · 跑起来Station 5 · Running It
  • 引用核验 2026-11-18refs checked 2026-11-18
解决什么问题What problem it solves

把长上下文里最占显存的 KV 缓存压成低秩潜向量,再在推理时展开。Compress the KV cache that dominates long-context memory into a low-rank latent vector, then expand it at inference.

这是什么What it is

每个头的 Key / Value 先投影到一个低秩潜空间,缓存只存这个潜向量。

推理时再把潜向量投影回各头的 Key / Value,省的是显存,不是计算。

DeepSeek-V2 提出,后续 V3 / V4 系列沿用并继续调整。

Each head’s Key/Value is first projected into a low-rank latent space; only that latent is cached.

At inference it is projected back to per-head Key/Value; the saving is memory, not compute.

Proposed in DeepSeek-V2 and carried forward in V3/V4 with further tuning.

为什么需要知道Why it matters

长上下文的第一道墙不是算力,是 KV 缓存。MLA 让「1M 上下文」在单卡或多卡上变得可行,也改变了推理框架的缓存布局;不理解它,就看不懂为什么同样参数量的模型,长上下文成本会差一个数量级。The first wall in long context is not compute but KV cache. MLA makes 1M context feasible on one or a few GPUs and changes the cache layout in inference stacks; without it, the cost gap between similar-size models is hard to explain.

出处Sources
  • DeepSeek-V2(论文 §2)DeepSeek-V2 §2 (paper) ↗
  • DeepSeek-V3(技术报告)DeepSeek-V3 technical report (paper) ↗