Agent 评估怎么做How to Evaluate Agents

先定义任务成功条件：不是「回答像样」，而是「目标完成 + 工具调用正确 + 没有越权」。

记录完整轨迹：消息、工具调用、参数、返回值、步数、耗时与 token；没有轨迹就无法归因。

分开评四件事：最终答案、工具选择、参数正确性、过程效率（步数与成本）。

多次运行时分别记录成功率与失败类型；pass@k 表示多次尝试中至少一次成功，不能用它代替每次都可靠。需要衡量重复可靠性时，应明确“全部 k 次成功”的统计口径。

按每次调用分别计算输入、输出和工具费用，再对全部步骤与重试求和；不要将已经汇总的 Token 再乘步数。

建回归集：把线上失败案例收进固定题集，每次换模型/改提示词后重跑。

Define task success first: not "a plausible answer" but "goal achieved + correct tool calls + no overreach".

Record the full trajectory: messages, tool calls, arguments, return values, steps, latency and tokens; without it you cannot attribute failure.

Score four things separately: final answer, tool choice, argument correctness and process efficiency (steps and cost).

Across repeated runs, record success rate and failure types. pass@k concerns at least one success, not success every time. For repeatability, explicitly report an all-k-success metric.

Calculate input, output and tool charges for each call, then sum steps and retries. Do not multiply already aggregated tokens by the step count again.

Build a regression set: put production failures into a fixed question set and rerun after every model or prompt change.