上下文并行Context Parallelism
Context Parallelism
解决什么问题What problem it solves
把超长上下文拆到多张卡上分别处理。Split a very long context across several cards for processing.
这是什么What it is
vLLM 官方把它与张量、流水线、数据、专家并行并列为分布式推理手段。
它针对的是注意力开销随序列长度二次增长。
vLLM lists it alongside tensor, pipeline, data and expert parallelism for distributed inference.
It targets the quadratic growth of attention cost with sequence length.
为什么需要知道Why it matters
想要超长上下文靠的是这类技术，不是单卡显存更大。Very long contexts come from techniques like this, not from a single bigger card.
现在能做什么What you can do now
回到名词地图看第5站的其他 29 个词 →Back to the map: the other 29 terms in station 5 →