Blog · ← All posts
Lost Acknowledgement Recovery: When an Agent Action Succeeds but the Confirmation Never Arrives
By the Ingenious AI team · Published: November 14, 2026 · Reviewed: November 14, 2026 · 7 min read
The short answer: Lost-acknowledgement recovery means deciding, after a timeout or dropped response, whether an agent's action actually happened — without blindly repeating it. The safe method is: check the recorded ledger and the downstream system for the action's unique record before any retry; if the action is recorded and confirmed downstream, treat it as done and record the recovery decision; retry only when both sources agree the action never landed. The ledger turns a guess that duplicates payments into a lookup.
Replay a recorded agent incident free → Step through a real recorded run in the replay view, or paste ledger records into the public verifier and get a VERIFIED/BROKEN verdict in seconds — no signup.
Every distributed system has this ghost: the request succeeded, the response died on the wire, and the caller stands there not knowing. For a human clicking a button, the cost is a double-click warning. For an autonomous agent with a retry loop, the cost is a second charge, a duplicate order, a double-sent contract — repeated at machine speed until someone notices the pattern in a reconciliation report.
The failure is not the retry; it is retrying blind. This guide explains the lost-acknowledgement pattern precisely, why idempotency keys alone do not save agents, and how an action ledger gives the recovery path something solid to stand on: a provable record of what was attempted, what was confirmed, and what the recovery decision was.
What is a lost acknowledgement? the failure mode in which an action completes on the receiving system but its confirmation never reaches the actor — a timeout, a dropped connection, a crashed gateway — leaving the actor unable to distinguish 'done' from 'never happened'. For AI agents, whose retry logic acts without human hesitation, resolving that ambiguity from records before retrying is the difference between recovery and duplication.
Why agents are hurt by this more than other software
Conventional services handle ambiguity with idempotency keys and human escalation. Agents break both habits. Their tool layers often call third-party APIs — payment processors, email services, CRMs — that were not designed with per-call idempotency, and an agent's planner, told 'the refund may not have gone through', will cheerfully reason its way to 'issue it again to be safe'. The agent has no memory of the ambiguity across sessions unless the system gives it one.
The retry is also faster than the evidence. A duplicate API call lands in seconds; the reconciliation that reveals it lands in tomorrow's report. By then an earnest agent may have 'recovered' the same phantom failure across dozens of customers in one overnight run.
The three states every action actually lives in
Recovery starts by admitting there are three states, not two. Confirmed: the downstream system acknowledged success. Absent: the downstream system has no trace of the action. Ambiguous: the call was dispatched and the outcome is unknown. Most retry logic collapses ambiguous into absent, which is the entire bug.
The ledger's job is to make the third state visible and durable. The agent records the attempt — tool, arguments digest, timestamp — as its own ledger event before or as the call is dispatched, then records the outcome when known. An attempt with no outcome record, found later, is precisely an ambiguous action, and it can be enumerated: not guessed at, listed.
The recovery procedure, in order
When an action is ambiguous, the procedure is lookup before action. First, query the downstream system by the strongest identifier available — the idempotency key if the API supports one, the argument digest and time window if it does not — and establish ground truth. Second, record the finding in the ledger: a recovery event that cites the original attempt, states what the downstream check found, and records the decision (confirmed-done, safely-retried, escalated). Third, act on the decision: skip, retry once with the same identifier, or hand to a human with the full record attached.
Recording the recovery decision matters as much as the decision itself. Weeks later, the duplicate-that-wasn't and the retry-that-was need to be distinguishable in the audit trail, and 'the agent retried at 03:12 after a downstream check at 03:11 returned no record' is a complete answer, while a bare second action record is an accusation waiting for a reader.
Idempotency keys: necessary, not sufficient
Where an API accepts idempotency keys, use them — derive the key from the ledger record so the attempt and the key are born together. But keys expire, some APIs scope them per-endpoint in surprising ways, and many tools agents use (email sends, messages, webhooks into customer systems) have no key concept at all. The ledger is the layer that survives those gaps: even where the downstream system cannot deduplicate, your own records can tell the agent it has already been here.
The planner needs the rule stated plainly, in policy and in tooling: never repeat a consequential action on ambiguity alone. Ambiguity is a lookup task, not an action task. Encoding that rule in the recovery path — rather than hoping the model's judgment holds at 3 AM — is what separates a control from a hope.
Designing the ledger events for recovery
Three record types do the work. The attempt record: tool, argument digest, idempotency key if any, and the downstream identifier expected. The outcome record: confirmed success with the downstream's own ID, confirmed failure with the error, or — written by a sweeper, not the agent — outcome-unknown after a defined window. And the recovery record described above. Agent Flight Recorder's append-only chain holds all three in order, so the recovery story is itself tamper-evident: nobody can later insert a tidy 'we checked first' record into a history that does not contain one.
A periodic sweeper closes the loop: it scans for attempts whose outcome window has expired, runs the downstream lookup, and files the recovery record or raises the case to a human queue. Ambiguity should have a half-life measured in minutes, not in reconciliation cycles.
Five actions you must take today
Lost acknowledgements are inevitable; duplicate actions are optional. These five steps move you from one to the other:
- Record attempts as first-class events. The dispatch of a consequential action goes in the ledger with its argument digest and idempotency key, before the outcome is known.
- Ban blind retries in policy and in code. Ambiguous outcomes route to a lookup procedure; the agent may not repeat a money-moving or message-sending action on a guess.
- Run a sweeper for stale attempts. Attempts older than your outcome window get an automatic downstream check and a recorded recovery decision, or a human ticket.
- Record the recovery decision itself. What was checked, what it showed, what was done — cited against the original attempt's sequence number.
- Rehearse one recovery. Kill a confirmation in staging and watch the path execute; the first real lost ack should not be the procedure's first test.
Worked example: 41 refunds, zero duplicates
A fictional subscription-box company, Fernway Goods, runs a billing agent that issues refunds through a payment API. One night the API's responses began timing out mid-run: 41 refund calls were dispatched into an acknowledgement blackout over 22 minutes. A naive retry loop would have re-issued all 41 — $6,180 in duplicate refunds, discovered days later.
Instead, each attempt already sat in the ledger with its argument digest and the refund's idempotency key. The sweeper picked up the 41 outcome-unknown attempts, queried the processor by key, and found ground truth: 38 had completed, 3 had never landed. It filed 41 recovery records — 38 marked confirmed-done with the processor's refund IDs, 3 marked safely-retried with the same keys — and the re-issued three completed exactly once. Total refunds: 41. Duplicates: zero.
The morning report the team read was the ledger itself: attempt, ambiguity, check, decision, outcome, in order, chain-verified. The processor's timeout became a footnote in an incident file instead of a month of chargeback archaeology.
FAQ
What is the difference between a lost acknowledgement and a failed action?
A failed action is known not to have happened — the system returned an error. A lost acknowledgement means the outcome is unknown: the action may have completed perfectly. Treating the two identically is the root of duplicate-action incidents. They need opposite handling: failures may be retried; unknowns must be resolved first.
Can't we just make every API call idempotent?
Where the API supports it, absolutely — and you should. But coverage is never total: email, messaging, webhooks, and plenty of vendor APIs lack idempotency, keys expire or are scoped unexpectedly, and an agent chains tools whose combined effect no single key covers. The ledger is the control that spans all of them, because it records the agent's side of every attempt regardless of the tool's features.
How long should an attempt stay 'ambiguous' before recovery runs?
Minutes, tuned to the slowest legitimate confirmation path for that tool — long enough that a merely slow ack is not misclassified, short enough that a queue of unknowns cannot accumulate into an overnight incident. The sweeper's window is a policy decision; write it down per tool class, and alert when the ambiguous count itself spikes, because that spike usually means a downstream outage, not 41 independent accidents.
Should the agent itself perform the recovery lookup?
It can, under a procedure that constrains it: lookup tools only, no consequential actions until the lookup resolves, and the decision recorded. Many teams prefer a deterministic sweeper outside the agent for exactly this step — the component deciding whether money already moved should be the least creative software in the building. Either way, the recovery record lands in the same ledger, in sequence.
What does the audit trail need to show after a recovery?
Four linked facts: the original attempt (tool, digest, key, time), the ambiguity (no outcome within the window), the resolution evidence (what the downstream check returned, and when), and the decision (skip, retry, escalate) with its outcome. If any link is missing, a later reviewer sees either an unexplained duplicate or an unexplained gap — both read as control failures, even when the recovery itself was correct.
Start logging your agents free → The free tier gives you the full hash-chained ledger for 1 agent and 1,000 actions a month, with chain verification and replay included.