吞吐 · 学 AI | 投机取巧

解决什么问题What problem it solves

每秒能生成多少字。决定长文生成要等多久。How many tokens per second come out. It determines how long you wait for a long generation.

这是什么What it is

每秒输出的 token 数量。吞吐高则长文本生成快。

Tokens produced per second. Higher throughput means faster long-form generation.

为什么需要知道Why it matters

它和首字延迟是两个独立指标，要一起看。只看其中一个容易误判实际体验。It is a separate metric from time-to-first-token, and the two must be read together — looking at only one misleads the actual experience.

别名Chinese aliases

tokens per second · tok/s

接着读什么，为什么What to read next, and why

首字延迟Time To First Token, TTFT
开始等待时间与持续处理能力分开看。Separate initial waiting from sustained processing capacity.
OpenAI 兼容接口OpenAI-compatible API
一起用于《接口兼容就等于行为一致》Used together in “A Compatible API Means Identical Behaviour”
KV 缓存KV Cache
在《运维上线路线》中接着读Continue in “The Operator's Route”

现在能做什么What you can do now

去 Models 图谱：看这个模型有哪些量化版、按你的显存该下哪一档Models atlas: which quantisations exist and which tier fits your VRAM ↗

回到名词地图看第5站的其他 29 个词 →Back to the map: the other 29 terms in station 5 →

出处Sources

NVIDIA AIPerf 指标参考NVIDIA AIPerf server metrics reference ↗
vLLM 性能基准套件vLLM benchmark suites ↗

吞吐Throughput, TPS