vLLMvLLM
vLLM
解决什么问题What problem it solves
在服务器上高并发跑模型的服务引擎。A serving engine for running models at high concurrency on a server.
这是什么What it is
面向生产的高吞吐推理引擎，通过连续批处理等手段提升并发效率。
A production-oriented high-throughput inference engine that improves concurrency via techniques such as continuous batching.
为什么需要知道Why it matters
个人本地部署用不上它。但如果你要给别人提供 API，就需要这一类工具。You will not need it for a personal local setup, but you do need something like it to offer an API to others.