What Is an AI Audit? What an Independent Audit of a Production AI System Covers, and Which Kind You Need
Last reviewed: November 2026.
In short. "AI audit" is used for four different things: a review of an organisation's AI governance, an assessment against a specific law or standard, the use of AI tools inside financial auditing, and an independent check of whether a particular AI system works. Each answers a different question. If you already run an AI system, or have a pilot that is not moving, the useful one is usually the last: an evidence-based check, done by someone other than the builder, of whether the system works on your data. It ends in findings and a recommendation, not a certificate.
Someone has told you to "get the AI audited": a board, a customer, an insurer, or your own unease about a pilot nobody can vouch for. The first task is to find out which question is being asked, because the same phrase covers work with very different scope, cost and output.
What is an AI audit?
An AI audit is a structured examination of an AI system, or of the organisation that builds and runs it, against a stated standard, with written findings. The examiner gathers evidence, compares it with the standard, and reports what holds and what does not.
Everything turns on the words "against a stated standard". The standard can be a law, a published framework, or criteria agreed at the start for one system. An audit with no agreed standard is an opinion. An audit with one tells you whether the thing you looked at meets it, and nothing more than that.
What are the different kinds of AI audit?
| Kind | The question it answers | Who usually asks | What it examines |
|---|---|---|---|
| Governance and management review | Does the organisation have the roles, policies and risk processes to run AI responsibly? | Board, risk committee, procurement | Policies, ownership, risk registers, documented processes |
| Legal or standards assessment | Does this system, or this organisation, meet a particular law or standard? | Legal, compliance, a regulator or customer | Documents and controls mapped to the requirement named |
| AI in auditing | Which AI software helps auditors and accountants do their own work? | Finance and audit teams | Software features, not your AI system |
| Independent audit of a running system | Does this system work on our data, and can anyone show that it does? | CIO, CTO, the budget owner | The system itself: data, evaluation, deployment, monitoring, ownership |
The last row is the focus of this page. The first two are real and sometimes mandatory, but they mostly examine documents, controls and process. A system can sit inside a well-documented governance process and still be producing wrong answers on stale data, because reviewing the documents does not run the system.
The third kind is a search-result neighbour, not a relative. If you are looking for audit software for an accounting team, this is the wrong page.
One public reference for the governance side is the NIST AI Risk Management Framework, which NIST describes as released on 26 January 2023 and intended for voluntary use. A framework like this gives an organisation a vocabulary and a way to organise risk work. Which laws and standards apply to your system depends on where you operate and what the system does, and that is a question for your legal advisers, not for this article.
What does an audit of a working AI system examine?
An independent audit of a running system looks at five things, at the level of what is checked rather than how:
- The data. Where each input comes from, how old it is when the system acts on it, and what happens when a source goes quiet. The principle is set out in the data-integrity contract: provenance and a freshness limit for every value that feeds a decision, and an honest "data unavailable" when a limit cannot be met.
- The evidence that it works. Whether there is a repeatable way to measure quality on a fixed, representative set of cases, or only a demo that looked right once. See the evaluation harness for what that instrument is.
- What is actually running. Whether the deployed system is the one that was reviewed and tested, and whether it is doing real work on real inputs. A passing health check does not answer this. Liveness and outcome are different questions.
- Monitoring. Whether anyone would learn about a wrong answer before a customer did, and whether the alarms watch outcomes rather than only uptime.
- Ownership and change control. Who can change the system, who approves it, and who answers when it misbehaves.
The same checks are described for a system you are about to ship in how you know an AI system works in production.
Why should the auditor not be the builder?
Because a builder tends to test the cases they expected. The person who wrote the prompts, picked the data and tuned the thresholds knows where the system is strong, and checks those places. An independent examiner has no stake in the answer and no habit of avoiding the weak spots.
Independence is a property of who does the checking, not of how thorough the checking is. A thorough review by the team that built the system is still a self-review. It is useful, and it is not an audit in the sense this page uses. The same rule runs through our own work: whoever builds something does not sign it off.
What does an AI audit deliver, and what does it not prove?
A useful audit leaves you with a written report and a decision. At the category level that means:
- a statement of what was examined and what was not, so nobody reads more into the result than it supports;
- the evidence behind each finding, so the finding can be checked;
- a short list of risks, each with a plain-language note on how serious it looks;
- a recommendation: leave it as it is, fix named things, or go deeper.
An audit also has limits that its report should state:
- It describes a point in time. The system, its data and its model provider can all change the day after. Anything you want to stay true needs ongoing monitoring, not a repeat audit.
- It covers the scope that was agreed. One system, one workflow, one data source. It says nothing about the rest of your estate.
- It is not a guarantee. "Passed" and "certified" are claims about a standard being met. Unless a recognised body has issued one, nobody should hand you a certificate, and no audit should be read as legal advice or as evidence that you comply with a particular law.
- A clean result is a real result. "This is working and we found nothing serious" is a legitimate outcome. An audit that cannot come back clean is a sales process.
Which kind of audit do you need first?
| Your situation | Start with | Why |
|---|---|---|
| A board, customer or insurer asks for evidence of AI governance | A governance review, plus legal advice on what applies | The request is about process and documents |
| A specific law or standard has been named | An assessment against that text, with your legal advisers | Only the named text defines "pass" |
| A live AI system, and no one can show it works | An independent audit of the running system | The question is whether it works on your data |
| A pilot that has stalled | The same check, then a decision to recover, re-scope or retire | The cause of a stall is often visible in the data and the evaluation |
| Several AI ideas and no clear order | A readiness or opportunity assessment | Nothing is running yet to examine |
If the earlier question is whether to bring in outside help at all, when to bring in an AI implementation partner covers that decision.
How is an AI audit different from an AI opportunity audit or a proof of concept?
They answer different questions at different points.
- An opportunity audit asks which AI use cases are worth funding and whether your data is ready. It looks forward. Our version is AI Compass, from $5,000, which returns a ranked, costed roadmap and a data-readiness rating for each system.
- A proof of concept asks whether one defined use case clears a quality bar at an acceptable cost on your data. POC in 3 Weeks starts from $15,000, and a reasoned "do not build" counts as a successful outcome.
- An audit of a running system asks whether something that already exists is trustworthy. For a pilot that has stalled, AI Rescue starts from $25,000 for the whole engagement, and its first-week triage ends in a recover, re-scope or retire verdict with an off-ramp.
- For one system where you want a short, fixed-scope independent read before deciding anything larger, the AI Verification Snapshot is $2,500 and takes two days. It diagnoses; it does not fix.
If you are unsure which applies, three questions place you on the ladder, and an AI production-readiness checklist is a self-check you can run first.
Frequently asked questions
Is an AI audit the same as an AI compliance assessment?
No. A compliance assessment measures a system or organisation against a named law or standard. An audit of a running system measures whether it works, against criteria agreed at the start: what correct output, fresh data and acceptable risk mean for that system. The two can be done together, but passing one does not imply the other, and neither is a substitute for legal advice.
How long does an audit of an AI system take?
It depends on scope. A narrow, fixed-scope independent check of one system can take days; a broader review across several systems can take weeks. We would rather agree the scope in a short conversation than quote a duration without one.
Can we audit our own AI system?
You can review it, and you should. A self-review is not independent, though, for the reason above. If the result will be shown to a board, a customer or an insurer, it carries more weight when someone outside the team produced it.
What do we need to provide?
Typically a plain description of what the system does and who uses its output, read-only access to the system, its logs and a sample of the data it works on, whatever evaluation or test material exists (or a clear statement that there is none, which is itself a finding), and one person who can say how it runs today rather than how it is documented.
Talk it through
If you are not sure which kind of audit you are being asked for, or whether a live system or stalled pilot would stand up to one, we can work through it in a free 30-minute AI working session.
NewGenApps: production AI, proven. Stay a step ahead, always.