跳到主要内容Skip to main content

视觉语言模型(VLM)Vision-Language Model (VLM)

Vision-Language Model (VLM)

  • 模型能力Model capabilities
  • 第四站 · 看懂Station 4 · Under the Hood
  • 引用核验 2026-11-17refs checked 2026-11-17
解决什么问题What problem it solves

同时理解图像和文字,能看图回答问题或按文字描述处理图片的模型。A model that understands images and text together, answering questions about images or acting on text instructions.

这是什么What it is

VLM 通常由视觉编码器把图片变成向量,再与文本 token 一起送入语言模型。

CLIP 用 4 亿图文对做对比学习,证明了自然语言可以作为视觉概念的接口。

A VLM usually uses a vision encoder to turn an image into vectors, then feeds them with text tokens into a language model.

CLIP trained on 400 million image-text pairs with contrastive learning, showing natural language can serve as an interface to visual concepts.

A two-part VLM: a vision encoder turns the image into tokens, and the language model reads image and text tokens together.
VLM 的两段式结构:视觉编码器把图片变成 token,语言模型把图像 token 与文字 token 一起读。A two-part VLM: a vision encoder turns the image into tokens, and the language model reads image and text tokens together.
为什么需要知道Why it matters

票据、截图、图表、质检都靠它;但「看图说话」不等于精确测量,细节与文字仍需专门模型。Receipts, screenshots, charts and inspection all rely on it. But describing an image is not precise measurement; fine detail and text may still need dedicated models.

别名Chinese aliases

VLM · 多模态模型 · 视觉语言模型

出处Sources
  • Radford 等 · CLIP(arXiv 2103.00020)Radford et al., Learning Transferable Visual Models From Natural Language Supervision ↗
  • Liu 等 · LLaVA(arXiv 2304.08485)Liu et al., Visual Instruction Tuning ↗