事故响应预案Incident Response Plan for an AI Feature
Incident Response Plan for an AI Feature
出事时谁在多久之内做什么、用什么手段把损害停下来。Say who does what, within how long, and with which lever, when an AI feature goes wrong.
成熟骨架已经有了：NIST SP 800-61 第 3 版（2025 年 4 月）自述目标是帮组织为事件响应做准备、减少发生的事故数量与影响，并改进事件检测、响应与恢复活动的效率。你的预案要接住同样的三段：事前定好判定口径，事中有人能按下暂停，事后有可复盘的记录。
AI 功能要在「检测」这一格补自己的判定条件：拒答率或输出格式突然变化、护栏放行异常、工具被诱导执行了未授权动作、同一提示词的成本与延迟异常。没有判定口径，事件会长期以「最近模型有点怪」的形式存在。
遏制手段要预先写好并指定决策人：切回上一个模型版本、关闭工具调用、收紧提示词或护栏配置、按 key 吊销访问、把流量降级到只读或人工处理。
对外通报有前提与时间口径：Anthropic 的负责任披露政策承诺在三个工作日内确认收到漏洞报告，并在既定时间线内确认修复策略——这是他们的口径，不能当作你的义务长度，也不能反过来推别人的时限。
教学虚构情境（非真实事件）：某内部文档问答机器人在提示词里带出了其他租户的片段摘要。预案要回答的是接下来四步——谁在十分钟内关停该检索通道、谁负责保留日志与样本、谁判定影响范围并决定是否通报、什么条件下才允许重新打开。
复盘产出的是控制项与门禁改动，不是一句「已加强培训」。
A mature skeleton already exists: NIST SP 800-61 Rev. 3 (April 2025) states its aim as helping organisations prepare for incident response, reducing the number and impact of incidents, and improving the efficiency of detection, response and recovery. Your plan has to cover the same three stretches: judgement criteria set in advance, someone able to press stop during the event, and a record you can review afterwards.
AI features need their own detection criteria written down: a sudden shift in refusal rate or output format, guardrails passing what they used to block, a tool being steered into an unauthorised action, abnormal cost or latency for the same prompt. Without criteria, an incident lives on as "the model has seemed odd lately".
Containment levers should be pre-written with a named decider: roll back to the previous model version, disable tool calling, tighten the prompt or guardrail configuration, revoke access by key, degrade traffic to read-only or human handling.
External notification has its own terms and clock. Anthropic's responsible disclosure policy commits to acknowledging a submitted vulnerability within three business days and confirming a remediation strategy within an established timeline. That is their commitment to reporters - it is neither your legal duty's length nor a way to infer anyone else's deadline.
Teaching scenario, invented for illustration and not a real event: an internal document assistant leaks a snippet belonging to another tenant inside a generated answer. The plan must answer the next four moves - who disables that retrieval path within ten minutes, who preserves logs and samples, who assesses blast radius and decides on notification, and what must be true before it is re-enabled.
The review should output control changes and gate changes, not a line saying "the team has been retrained".
没有预案时，团队会在事故中间现场发明流程：决策权不明导致停不下来，日志没保导致影响范围说不清，最后连「我们到底出了什么事」都要靠回忆。预案的价值是把这几件事在它发生之前定好。Without a plan, teams invent process mid-incident: unclear authority means nothing gets stopped, unpreserved logs mean the blast radius cannot be described, and in the end even "what happened" is reconstructed from memory. A plan's value is that those decisions are made before the event exists.
AI 事故预案