Blog · ← All posts
The AI Agent Audit Trail Checklist: 25 Questions to Ask Before You Trust an Agent With Real Work
By the Ingenious AI team · Published: November 14, 2026 · Reviewed: November 14, 2026 · 7 min read
The short answer: Before trusting an AI agent with consequential work, ask 25 questions in five groups — recording (does it capture actions, decisions, outcomes, refusals, as they happen?), integrity (can alteration be detected and independently verified?), completeness (is coverage measured against the real world?), privacy (are records masked and proportionate?), and operations (lifecycle, alerts, incident response, retention, exit). Score each answer as mechanism, policy, or promise: mechanisms count, policies count partially, promises count for nothing. Any platform — including Agent Flight Recorder — should be graded exactly this way, by you, before the agent touches money, customers, or data you cannot un-touch.
Replay a recorded agent incident free → Step through a real recorded run in the replay view, or paste ledger records into the public verifier and get a VERIFIED/BROKEN verdict in seconds — no signup.
Checklists work because they move judgment from the worst moment to the best one: the questions are written calmly, in advance, and asked mechanically when enthusiasm or urgency would otherwise skip them. Agent accountability is overdue for that treatment. The market's vocabulary — 'full visibility', 'enterprise-grade logging', 'audit-ready' — is ungraded adjectives all the way down, and the differences that matter (can a record be edited? is coverage measured?) hide comfortably inside them.
This checklist replaces the adjectives with 25 questions whose answers are checkable. It is written for buyers evaluating a vendor, founders evaluating their own stack, and platform teams evaluating a build. Grade any of them — this product included — and let the pattern of answers, not any single one, make the decision.
What is a mechanism-grade answer? an answer that names a verifiable process — 'records are hash-chained; here is a verification you can run' — as opposed to a policy-grade answer ('only admins can edit logs', which describes intention and access) or a promise-grade answer ('we take integrity very seriously', which describes nothing). The checklist's scoring rule is the point: in evidence systems, only mechanisms survive contact with a motivated skeptic, so only mechanisms earn full marks.
Recording: five questions on what is captured
1. Is every tool action recorded as it happens, from the execution path itself — or reconstructed later from other logs? 2. Does each record carry the agent's stated reasoning, or only the action? 3. Are outcomes recorded, including failures and the downstream system's identifiers? 4. Are refusals recorded — guardrail declines and the agent's own — with what happened next? 5. Are lifecycle events (enrollment, mandate changes, suspensions) part of the same record? Score the pattern: a system capturing actions only has built the spine and skipped the nervous system — it will answer 'what' fluently and 'why, under whose authority, and what stopped it' not at all.
Integrity: five questions on whether records can be trusted
6. What mechanically prevents or detects alteration of a historical record? 7. Can verification be run by an outsider, on exported records, without the vendor's stack or permission? 8. Are there signed snapshots (certificates or equivalent) anchoring history at period closes? 9. Who inside the vendor — or inside your own team, for a build — can write to the record store, and is that list short, named, and itself logged? 10. What happens, demonstrably, when verification fails — is there a break-point concept and an investigated example? The disqualifying pattern is integrity by access control alone: 'nobody is allowed to edit' is a policy, and policies are what incidents reveal the exceptions to.
Completeness: five questions on what might be missing
11. How is coverage measured — is the record reconciled against independent downstream systems, and what is the current figure? 12. Can the agent reach any tool through a path that bypasses recording, and is that list written down? 13. Are failed and refused actions included, or only successes? 14. Is there an omission or coverage report you can see on a schedule, not just during incidents? 15. When a gap was last found, what was the disposition — is there an example with a paper trail? Completeness is where confident vendors get quiet, because measuring it requires admitting the residue exists. A stated 99.7% with an investigated residue outranks an unmeasured 100% claim at any serious review.
Privacy and data: five questions on proportion
16. What personal data do the records contain — masked summaries, or raw payloads and transcripts? 17. Where full content is kept, what is its retention — days, or the same years as the audit record? 18. What does the capture policy say, per data type, and can you read the document? 19. Who can read the ledger, under what roles, and are reads themselves logged? 20. How are erasure and access requests handled against records that must also serve as evidence? The pattern to reward is a system that can prove more while holding less — digests standing in for payloads, masking at capture rather than by cleanup. The pattern to fear is a data lake justified as an audit trail: it will fail the privacy review that follows the security review, in the same procurement, a month later.
Operations and exit: five questions on the life around the record
21. What watches the records — anomaly alerts on behavior (bursts, first-seen tools, off-hours), and who receives them? 22. What is the incident path — can the agent be paused in one action, the evidence frozen, the impact enumerated from the record inside an hour? 23. What are the retention periods per record tier, and are deletion and legal hold mechanical rather than manual? 24. What does the audit handoff look like — a scoped export with verification that an assessor can re-run, produced in days or in weeks? 25. Exit: can you take the complete record, in an open format, verifiable after you leave — and have you seen it done? Exit is the question vendors least expect and most reveal themselves on; a record you cannot leave with is not your record, and every other answer should be re-read in that light.
How to score it: the five-step method
The questions are only half the instrument. Score them this way:
- Demand demonstrations, not documents. For at least the integrity and completeness questions, ask to see it run — a verification, a coverage figure produced live, an export checked on the spot.
- Grade each answer: mechanism, policy, or promise. Mechanisms score 2, policies 1, promises 0 — and require a minimum integrity total before any other section can rescue the evaluation.
- Ask for the failure examples. The last verification failure, the last coverage gap, the last incident — a system with no failure stories has no operating history, whatever its feature list says.
- Test the exit before you enter. Perform a full export and an independent verification during evaluation, not during the divorce.
- Re-score annually. Platforms drift like agents do; last year's mechanisms become this year's policies through quiet deprecations, and the checklist is cheap to re-run against a live system.
Worked example: scoring two finalists
A fictional mid-size platform, ClearLedger Clinics, scored its two finalists for a patient-messaging agent. Vendor A demoed beautifully — rich tracing, a polished dashboard, 'immutable logs' in the deck. Under the checklist: integrity graded policy (access controls, no independent verification — question 7 failed outright), completeness unmeasured (question 11 answered with a promise), lifecycle events absent from the record, exit untested. Total: strong recording marks, disqualifying integrity pattern.
Vendor B — the flight-recorder pattern — demoed plainly and scored mechanisms nearly throughout: verification re-run by ClearLedger's own engineer on an export during the evaluation meeting, coverage stated at 99.8% with the residue's ticket attached, refusal records sampled live, exit export performed in the trial week. The evaluation memo's conclusion is the checklist's thesis in one line: 'We are not buying features; we are buying the answers we will be able to give in the worst meeting of the year. Only one finalist's answers survive that meeting.' The contract followed the memo.
FAQ
Can a vendor pass by open-sourcing or self-hosting instead?
Self-hosting changes who operates the system, not whether its answers are mechanisms — a self-hosted editable log is still an editable log, and the checklist grades it identically. Openness helps exactly where it should: it makes mechanisms inspectable, so verification claims can be checked in the code as well as in the demo. Grade the properties, not the deployment model; the checklist is deliberately indifferent to everything except whether the evidence works.
What score should block a purchase?
Treat three answers as gates rather than points: independent verifiability (question 7), measured completeness (question 11), and exportable exit (question 25). A zero on any gate means the system's evidence ultimately rests on trusting the vendor at the exact moments trust is contested — the definition of the problem the purchase exists to solve. Everything else can be traded against price and fit; the gates cannot.
Our internal build scores poorly. Is the checklist unfair to builds?
It is equally unfair to everyone, which is its virtue — vendors face the same 25. A build scoring poorly has received the cheapest possible education: the list of what to build next, in the order external reviewers will ask. The one asymmetry to respect is time: a build's gaps close over quarters while its history accrues at its current standard, so a poor score today argues for adopting a mechanism-grade system for the consequential agents now and letting the build mature behind it, if it matures.
How do we use the checklist on a free trial in one week?
Run the demonstrations: enroll an agent, execute a day of real actions, export, verify independently, reconcile against one downstream system, attempt the exit export. Five questions — 6, 7, 11, 24, 25 — exercised live in the trial answer more than a month of documentation review. Trials that cannot accommodate those five exercises are telling you the answers in the only language that never lies: behavior.
Should we share our scoring with the vendor?
Yes — it improves the market you buy from. Vendors told precisely which mechanisms a buyer grades will build them, and a shared vocabulary ('mechanism, policy, promise') raises the cost of adjective-driven selling for everyone. This checklist is published for exactly that reason, and AFR expects — invites — to be graded by it in public, in evaluations we will sometimes lose on other merits. The gates are not negotiable because they are the product.
Start logging your agents free → The free tier gives you the full hash-chained ledger for 1 agent and 1,000 actions a month, with chain verification and replay included.