Blog · ← All posts
AI Agent Anomaly Alerts That Matter: Bursts, First-Seen Tools and After-Hours Spikes
By the Ingenious AI team · Published: November 14, 2026 · Reviewed: November 14, 2026 · 7 min read
The short answer: The three highest-value anomaly alerts for AI agents are: action bursts (an agent's action rate exceeding three standard deviations above its own baseline), first-seen tool use (an agent calling a tool it has never called in production), and after-hours spikes (activity materially above the agent's normal pattern outside operating hours). They work because they detect behavior change rather than known-bad content — most agent incidents are the agent doing a legitimate thing at an illegitimate rate, time, or scope, and these rules fire while the damage is still countable.
Replay a recorded agent incident free → Step through a real recorded run in the replay view, or paste ledger records into the public verifier and get a VERIFIED/BROKEN verdict in seconds — no signup.
Agent monitoring fails in two opposite ways: dashboards nobody watches, and alert storms everybody mutes. Both come from the same mistake — alerting on what is easy to count instead of on what distinguishes an incident from a Tuesday. Content-based detection (flag the bad word, the big refund) catches yesterday's incident. Behavior-based detection catches the shape of incidents in general, including the ones nobody has imagined yet.
Agent Flight Recorder's alert layer is built on three behavioral rules over the action ledger. This guide explains what each rule detects, why it works, how to set its baseline without drowning in false positives, and what a good alert contains when it fires — because an alert that does not hand the responder a starting point in the record is only half a control.
What is a 3-sigma burst? an action rate that exceeds the agent's historical mean by more than three standard deviations for its baseline window. Because the comparison is against the agent's own behavior — not a fleet-wide constant — the rule adapts per agent: a batch processor's normal Tuesday would be a support agent's emergency, and each is judged against its own history recorded in the ledger.
Rule one: action bursts — the loop and the spree detector
Runaway loops, refund sprees, bulk deletions, and messaging floods share one signature: the agent is acting far faster than it ever acts. The burst rule watches the rate of recorded actions per agent over short windows and compares it with the agent's own baseline distribution. Three sigma is the conventional starting threshold — tight enough to catch genuine excursions within minutes, loose enough that ordinary busy periods stay quiet once the baseline is honest.
Two tuning decisions matter. The baseline window must cover the agent's real rhythm — at least a few weeks, including its weekly cycle — or every Monday becomes an incident. And the rule should evaluate consequential action classes separately where volumes differ wildly: a burst of reads is weather; a burst of refunds is climate. The ledger's per-action typing makes that split cheap.
Rule two: first-seen tool — the scope-change tripwire
An agent's tool repertoire is its capability boundary, and a first production use of a new tool is a boundary event by definition. Sometimes it is legitimate — a feature shipped. Sometimes it is the incident: a prompt injection talked the agent into a dormant admin tool, a config error widened its tool list, a planner discovered a destructive endpoint the team forgot existed. Either way, a human should learn about it from an alert, not from the consequences.
The rule needs one companion discipline: an expected-first-use registry fed by deployments. When a release intentionally grants a tool, the release notes the expectation and the first use arrives pre-explained. Unregistered first uses page; registered ones log. That single habit keeps the rule's precision high enough to stay unmuted for years.
Rule three: after-hours spikes — the calendar as a sensor
Most agents have a temporal shape: they work when the business works. Activity that departs from that shape — a support agent suddenly processing hundreds of actions at 3 AM in a business that sleeps — deserves a look even when each individual action is legitimate. The famous agent incidents are disproportionately nocturnal for a mundane reason: nobody is watching, the agent's counterparties are batch jobs and other agents, and a small error compounds for hours before a human sees a dashboard.
Define 'after hours' per agent from its own baseline, not from office folklore: some agents legitimately run nightly batches, and their rule should watch for deviation from the batch's normal size and composition instead. The ledger makes the agent's actual rhythm measurable, so the calendar sensor is calibrated from evidence.
What a good alert contains
An alert is a handoff to a responder who may be asleep. It should carry: the agent, the rule, the observed value against the baseline ('214 actions in 10 minutes; baseline 22 ± 9'), the window's first and last ledger sequence numbers, and a link into the replay for exactly that range. With that, the responder's first question — what has it been doing? — is answered by the alert itself, and the second question — since when? — is one scroll away.
Alerts also belong in the ledger's story: file each alert and its disposition with the incident records, because the audit question 'did anything notice?' deserves a documented answer. An alert that fired and was investigated is evidence of a working control; an alert that fired into an unmonitored channel is a finding wearing a timestamp.
False positives: the tuning ethics
Every muted alert was once a false positive too many. Protect the channel: review alert precision monthly, retire or re-baseline rules that cry wolf, and never solve noise by raising thresholds past the size of a real incident — a burst rule that only fires at a thousand actions is a press release generator, not a detector. The goal is a small set of high-precision rules whose every firing earns investigation, which is why three rules, well kept, outperform thirty rules, decaying.
Five actions you must take today
Stand up the three rules with the discipline that keeps them trusted:
- Baseline each agent from its own ledger history. At least a few weeks including its weekly rhythm, before any threshold goes live.
- Split consequential action classes. Bursts of writes, payments, sends and deletions get their own thresholds, tighter than reads and lookups.
- Open the expected-first-use registry. Deployments declare new tool grants; unregistered first-seen tool calls page a human.
- Make every alert carry its replay link. Agent, rule, observed vs baseline, and the exact ledger range — the responder starts at the record, not at a dashboard.
- Review precision monthly and file dispositions. Every alert gets an outcome note; every noisy rule gets re-baselined or retired in the open.
Worked example: three alerts, one weekend
At fictional travel platform Wayfare Stays, the alert layer earned its keep across one weekend. Friday 23:12, the first-seen-tool rule fired: the itinerary agent had called a bulk-email tool it had never used in production. Investigation took ten minutes — a canary deployment had widened its tool list a day early; the call was legitimate, the deployment process was not, and the registry gained a rule.
Saturday 03:40, the after-hours rule fired on the payments agent: 96 actions in an hour against a baseline of 4. The replay showed a retry storm against a partner API returning errors — each failed booking retried with a fresh hold on customer funds. The holds totaled $18,240 across 61 customers before the agent was paused at 04:05; every hold was enumerated from the ledger and released with a timeline sent to the partner.
No third alert was needed: the burst rule's baseline absorbed Sunday's legitimate holiday traffic without a page, because it had been calibrated on the agent's own history rather than a fleet average. Three rules, two real catches, zero noise — the ratio that keeps responders reading alerts in year two.
FAQ
Why three sigma and not a fixed actions-per-minute cap?
Fixed caps encode one agent's normal as every agent's law: the batch agent trips them weekly, the quiet agent can misbehave for hours beneath them. Sigma thresholds scale with each agent's demonstrated variance, so they express the actual question — is this agent behaving unlike itself? Start at three sigma, then tune per agent from firing history; the right number is the one your precision review defends.
What about slow anomalies — the agent that degrades gradually?
Burst rules catch fast failures. Slow drift needs a different sensor: periodic comparison of action mix and outcome rates against a trailing baseline (refund share of actions creeping from 2% to 9% over a month). Run it as a weekly report rather than a paging alert. The ledger supports both because the underlying records are complete; choose the cadence that matches the failure's speed.
Should anomaly alerts ever stop the agent automatically?
For a narrow class, yes: extreme bursts in money-moving action classes justify an automatic pause pending human review, because the cost of a false pause is small against an unbounded spree. Make the auto-pause threshold distinctly higher than the alert threshold, log the pause as a ledger event, and rehearse the resume path. Everything else alerts a human — automation that halts the business on a statistical wobble teaches teams to disable it.
How do we baseline a brand-new agent with no history?
Start in observe mode: record everything, compute the baseline over the first weeks of supervised operation, and alert only on the coarse rules (first-seen tool is meaningful from day one; after-hours can use the business's stated hours until a rhythm exists). Graduate thresholds as the baseline firms up. The dangerous window is real but short, and human supervision is the compensating control while statistics accumulate.
Do alerts replace reviewing the ledger?
No — they triage it. Alerts decide what a human reads today; scheduled reviews (weekly samples, monthly verification) decide that the un-alerted majority was also sound. An alert-only posture misses the quiet, in-distribution failures: the agent that wrongly refunds one customer a day, every day, at perfect statistical normality. That failure is found by sampling and replay, which is why the audit habits exist alongside the tripwires.
Start logging your agents free → The free tier gives you the full hash-chained ledger for 1 agent and 1,000 actions a month, with chain verification and replay included.