Blog · ← All posts
AI Agent Incident Response: What to Do in the First Hour, in Order
By the Ingenious AI team · Published: November 14, 2026 · Reviewed: November 14, 2026 · 7 min read
The short answer: In the first hour of an AI agent incident: (1) stop the bleeding — pause the agent or revoke the specific tool access doing harm; (2) freeze the evidence — export the ledger range and mint an integrity certificate before anything changes; (3) verify the chain so you know the record is trustworthy; (4) scope the damage from the records — which actions, customers, and dollars, enumerated exactly; (5) communicate internally with those facts and externally with what you know, what you are doing, and when the next update lands. Order matters: containment before analysis, evidence before repair.
Replay a recorded agent incident free → Step through a real recorded run in the replay view, or paste ledger records into the public verifier and get a VERIFIED/BROKEN verdict in seconds — no signup.
Agent incidents have a distinctive first-hour pathology: the system that caused the problem is still running while people discuss it, and the records that would explain it are being written over by the response itself — restarts, rollbacks, hurried fixes that change the very behavior under investigation. The teams that handle agents well treat the first hour as a procedure, not a discussion.
This playbook is that procedure, ordered and timed, written for teams running agents with a hash-chained ledger like Agent Flight Recorder. It assumes the worst version: a consequential agent (money, messages, or data) has done something wrong at unknown scale, and the next sixty minutes decide the incident's final size.
What is containment? the first response objective in an agent incident: stopping further harm without destroying evidence. For agents, containment is usually a pause of the agent, a revocation of the specific tool grant doing harm, or a tightening of caps — chosen to be reversible, recorded as a ledger event, and fast enough to precede analysis. Analysis without containment studies a fire while it spreads.
Minutes 0–10: contain, and record the containment
Pause first, diagnose second. If an agent is actively misbehaving, every minute of analysis is paid for in additional bad actions. Pause the agent entirely if the failure mode is unknown; if it is known and narrow (one tool, one action class), revoke that grant or drop its cap to zero and let the rest run. Prefer the pause you can execute in one action from the runbook over the elegant one that requires a deploy.
Record what you did as events in the ledger's world: the pause, who ordered it, when. The incident file starts now, and its first entries are response actions — future readers (auditors, customers, courts) judge the response alongside the failure, and a documented ten-minute containment reads very differently from a reconstructed one.
Minutes 10–20: freeze the evidence before touching the system
Before any rollback, restart-with-new-config, or data repair: export the affected agent's ledger for the window (pad it — start earlier than you think the incident did), run chain verification over the export, and mint a Tamper-Seal certificate over the current state. This trio takes minutes and buys the investigation its foundation: a provably intact record of what happened, sealed before the response could be accused of tidying it.
Resist the repair impulse for twenty minutes. Fixes change prompts, configs, and data — the exact context the replay needs to show. If a fix genuinely cannot wait (an actively leaking credential), make it, but note the minute and the change in the incident file so the timeline can mark where the observed system stopped being the incident system.
Minutes 20–40: scope from the record, not from reports
Scope means enumeration: which actions in the window were wrong, which customers and objects they touched, what the totals are. Work from the ledger and the replay — walk the run from its first anomalous step, use interrogation to enumerate the action class ('every refund above the cap since 02:00'), and reconcile against the downstream system (the processor, the mail queue) to confirm the ledger's picture matches the world's.
The deliverable of scoping is a bounded statement: 'Between 02:12 and 02:47 the agent issued 17 refunds totaling $2,314 to 17 customers; no other action classes were affected; the chain verifies.' Bounded statements end panic, focus repair, and are the only honest basis for the external message that comes next. If scope is not yet boundable, say which question is open and what you are running to close it.
Minutes 40–60: communicate — inside first, then outside
Internally, circulate the bounded statement plus the working theory of the first divergence, labeled as theory. Externally, affected customers get the facts that concern them — what happened to their account, in what window, what has been done about it — and a named time for the next update. Do not wait for the complete root cause to send the first external message; silence during a known impact is itself a message, and it is never the one you would have chosen.
Draft from the timeline, not from adjectives. 'Our agent issued a refund to your account in error at 02:31; we have reversed it and paused the system' survives every later scrutiny. 'A technical issue may have caused some irregularities' survives none of it, and customers compare notes.
After the hour: the investigation proper
With containment holding, evidence frozen, scope bounded, and communications running, the incident moves to the postmortem track: first-divergence analysis, failure-family classification, regression case, corrective actions. Schedule the postmortem inside the first day while the ledger walk is fresh, and keep the response's own records — alerts, decisions, messages — filed with the incident's records, because the audit trail of the response is part of the control story you will tell about this incident forever.
Five actions you must take today
The first hour is rehearsed in peacetime or improvised in crisis. Prepare it now:
- Write the pause into a runbook and test it. One action, known to on-call, executed in a drill this quarter — a pause nobody has performed is a hope, not a control.
- Pre-stage the evidence freeze. The export, verification, and certificate steps scripted or documented to the click, so minute 10–20 takes five.
- Assign the first-hour roles. Incident lead, ledger/scope analyst, communications — named per shift before any incident exists.
- Draft the external message skeleton. Blanks for window, impact, action taken, next update time; fill-in speed beats blank-page speed at minute 45.
- File this playbook with the ledger links. The response starts from the record system's front door: replay, verification, interrogation — bookmarked for the on-call, not discovered mid-incident.
Worked example: sixty minutes at a payments startup
At fictional payments startup Tallybrook, the anomaly alert fired at 03:12: the disbursement agent's action rate hit 340 in ten minutes against a baseline of 30. On-call paused the agent at 03:16 — four minutes, one runbook action. The ledger export for the night, verification (PASS across 88,914 records), and a Tamper-Seal certificate were complete by 03:27.
Scope, from interrogation and replay: a partner API's success-code change had turned confirmations into perceived failures, and the agent had re-disbursed 212 payments totaling $96,400 between 02:58 and the pause — every one enumerated by record, and every one traceable at the partner for reversal because the ledger carried the downstream IDs. The first-divergence step was visible by 03:50: the first 'failed' confirmation that was actually a success, the textbook lost-acknowledgement shape.
Customer and partner messages went at 04:05 — inside the hour, with exact counts and the reversal plan. Final tally: zero unrecovered duplicates, a postmortem whose timeline nobody disputed, and an enterprise prospect who cited the incident report — sent unasked — in their security review as a positive.
FAQ
Should we ever let the agent keep running during an incident?
Only when the failure is provably narrow and the containment matches it — one tool revoked, one action class capped — and the ledger watch continues live. 'Provably' is doing the work in that sentence: if you cannot yet enumerate what the agent is doing wrong, you cannot know the running remainder is safe. The default for an unbounded failure is pause; resumption is cheap and reversible, unbounded harm is not.
Why freeze evidence before fixing? Fixes feel urgent.
Because the fix changes the crime scene. A new prompt, a rolled-back config, repaired data — each erases context the investigation needs to find the first divergence, and each hands a skeptic the argument that the records were curated during the response. The freeze takes minutes: export, verify, certify. Almost no real fix loses anything by waiting ten minutes; many investigations lose everything by not waiting.
What if verification fails during the freeze?
Then the incident has two threads from that moment: the agent failure and a records-integrity failure. Note the break point, preserve the export exactly as taken, and expand the response team — integrity failures implicate access paths and storage, and they change what can be claimed to auditors and customers. Do not repair the chain mid-incident; a re-chained ledger presented later proves nothing and suggests much.
How much should the first external message admit?
Everything that is established about the recipient's own impact, plainly: what happened to them, when, what has been done, what happens next and when they will hear again. Withhold speculation about cause and other customers' details. The ledger's gift is that the established set is large and precise — most first messages fail by vagueness, not by candor, and vagueness is a choice when the records are this good.
Who owns the decision to resume the agent?
A named role — typically the incident lead with the agent's owner concurring — against stated criteria: root cause identified or harm path severed, fix or guard in place, regression case passing, monitoring tightened for the resume window. Record the resume decision and its basis as an event. 'It seemed fine by Friday' is not a resumption criterion anyone can audit.
Start logging your agents free → The free tier gives you the full hash-chained ledger for 1 agent and 1,000 actions a month, with chain verification and replay included.