跳到主要内容Skip to main content

LLM-as-a-Judge(模型评审)LLM-as-a-Judge

LLM-as-a-Judge

  • 选型与成本Selection & cost
  • 第三站 · 用深Station 3 · Going Deeper
  • 引用核验 2026-11-17refs checked 2026-11-17
解决什么问题What problem it solves

让一个强模型按评分标准给另一个模型的回答打分或比较。Have a strong model score or compare another model’s answers against a rubric.

这是什么What it is

常见做法是给评审模型题目、参考答案和评分维度,让它输出分数或选择。

MT-Bench 与 Chatbot Arena 的论文系统检验了位置偏差、冗长偏差和自我偏好等问题。

A common setup gives the judge the question, a reference answer and scoring dimensions, and asks for a score or choice.

The MT-Bench and Chatbot Arena paper systematically examined position, verbosity and self-enhancement biases.

A judge reads the question, reference answer and rubric, then outputs a score or preference; its biases need human spot checks.
评审模型读题目、参考答案和评分标准,输出分数或偏好;它的偏差需要抽样人工复核。A judge reads the question, reference answer and rubric, then outputs a score or preference; its biases need human spot checks.
为什么需要知道Why it matters

人工评测贵且慢;模型评审能扩大规模,但必须校准偏差、抽样人工复核,不能把分数当 ground truth。Human evaluation is expensive and slow; model judging scales it up. But biases must be calibrated and sampled for human review — scores are not ground truth.

别名Chinese aliases

模型评审 · LLM judge · AI 评审

出处Sources
  • Zheng 等 · Judging LLM-as-a-Judge(arXiv 2306.05685)Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena ↗
  • FastChat · 官方仓库FastChat repository ↗