FlashAttentionFlashAttention

一种更省显存、更快的注意力算法，长上下文的基石。A more memory-efficient, faster attention algorithm; the base of long contexts.

Dao 等人的定义：IO 感知的精确注意力算法。用分块减少显存与片上缓存之间的读写。

原论文指出自注意力的时间与空间复杂度随序列长度二次增长，这是长序列慢的根源。

Dao et al.'s definition: an IO-aware exact attention algorithm using tiling to cut reads and writes between GPU memory and on-chip SRAM.

The paper identifies quadratic cost in sequence length as why long sequences are slow.

「上下文开大就慢」的底层原因在这里；它也是各家推理库的标配内核。This is the underlying reason bigger contexts are slower, and a standard kernel in serving stacks.

去 Models 图谱：看这个模型有哪些量化版、按你的显存该下哪一档Models atlas: which quantisations exist and which tier fits your VRAM ↗

回到名词地图看第5站的其他 29 个词 →Back to the map: the other 29 terms in station 5 →

• Dao 等《FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness》FlashAttention paper ↗