The Production-AI Failure Taxonomy

Q: Failure mode 1 — What is data-integrity drift, and how do you catch it early?

Definition. Input distributions (data drift) or the relationship between inputs and labels (concept drift) shift after deployment, silently degrading a model that passed its pre-launch tests. Both produce measurable model drift over time (IBM, What Is Model Drift?; "Time to Retrain? Detecting Concept Drifts in ML Systems," arXiv 2410.09190, 2024). Why it kills production systems. The model that shipped was validated against a snapshot of the world. Production is not a snapshot. When the input distribution moves — new customer segments, a changed upstream feed, a seasonal shift — accuracy erodes without any error being thrown. RAND identifies weak data foundations as among the most common root causes of AI project failure (RAND, RRA2680-1, 2024). The failure is quiet, which is what makes it dangerous. Pre-deploy detectable signal. The tell is the absence of a data contract at the system boundary and the absence of a measured distribution baseline. Concretely: before launch, compute a divergence measure — Population Stability Index or a Kolmogorov–Smirnov statistic — between the training set and a recent, production-representative holdout. A large divergence at deploy time, or the lack of any schema, freshness, and null-rate contract, predicts post-launch drift failure. The fix is a contract plus continuous distribution monitoring with retraining triggers. See the AI data-integrity contract.

Q: Failure mode 2 — Why do live systems still show no outcome?

Definition. The system is deployed, adopted, and visibly "working" — usage is up, uptime is green, demos land — yet it produces no measured business outcome. Activity is tracked; impact is not. Why it kills production systems. This is the mechanism directly under the 95% figure. NANDA's finding was not that the pilots crashed. It was that they ran and delivered no measurable P&L impact, with the gap in learning, integration, and context rather than liveness (MIT Project NANDA, 2025). S&P Global adds a warning sign: perceived positive impact of GenAI fell year over year across every enterprise objective — revenue 81%→76%, cost 79%→74%, risk 74%→70% — even as adoption rose (S&P Global, 2025). RAND's leading root cause is blunt: stakeholders misunderstand or miscommunicate the problem the system is meant to solve (RAND, RRA2680-1, 2024). An unstated outcome metric is that misunderstanding made concrete. Pre-deploy detectable signal. Look at how success is defined. If it is launch, usage, or uptime rather than a pre-registered business KPI with a baseline and a control or holdout, you have this failure before you ship. The test is simple: ask whether anyone can state the outcome metric and its current baseline before launch. If not, the system will read as live and never demonstrate outcome. We treat this distinction as a first-class delivery gate — see production AI: liveness vs outcome and how do you know an AI system works in production.

Q: Failure mode 3 — When does a passing eval lie to you?

Definition. The evaluation that greenlit the system does not predict production behavior, because it is contaminated by training exposure, saturated, or unrepresentative of the real task. Why it kills production systems. A benchmark score is only useful if it correlates with the behavior you care about. Contamination and saturation break that correlation. Scale AI built GSM1k, a fresh test in the style of the widely used GSM8K, and some models dropped by up to roughly 13 points, with the drop correlated to a measured memorization signal for GSM8K problems ("A Careful Examination of LLM Performance on Grade School Arithmetic," arXiv 2405.00332, 2024). Identical model weights can score materially differently depending only on the harness. A model that looks ready on a public leaderboard can be well short of ready on your task. Pre-deploy detectable signal. The warning sign is a go/no-go decision that rests on a public benchmark, with no held-out, task-representative, contamination-resistant eval built from your own data. Detect it directly: re-score the system on a private holdout it has never seen. A large gap versus the public number is the signal that your eval is not measuring your task. For depth on models like Claude, a strong holdout is also where a careful judge harness earns its keep. See the AI evaluation harness.

Q: Failure mode 4 — Why do multi-agent pipelines break when the model is fine?

Definition. Multi-step, multi-agent, or tool-calling pipelines fail from coordination and control-flow faults — not from a weak underlying model. Why it kills production systems. The most detailed public evidence here is MAST, a multi-agent system failure taxonomy built from over 1,600 annotated traces across 7 frameworks, with inter-annotator agreement of κ=0.88 ("Why Do Multi-Agent LLM Systems Fail?", UC Berkeley et al., arXiv 2503.13657, 2025). MAST catalogs 14 failure modes in 3 categories. System-design issues account for 44.2% of failures — including step repetition (15.7%), being unaware of termination conditions (12.4%), and disobeying task specification (11.8%). The authors conclude that failures stem from poor system design, not model performance. Gartner's projection that over 40% of agentic projects will be canceled by 2027 points partly at the same brittleness (Gartner, 2025-06-25). Pre-deploy detectable signal. The tell is no trace-level evaluation of the full multi-step flow, no explicit termination or timeout tests, and no fault injection on tool and agent calls. If you evaluate only single turns, the orchestration faults — loops, lost history, missed stop conditions — are invisible until production. Evaluate the whole trace, and inject faults deliberately. This also informs the architecture choice itself: see single vs multi-agent.

Q: Failure mode 5 — Is your human-in-the-loop actually in control?

Definition. A human reviewer is nominally in the loop but exercises no real control. Automation bias and throughput pressure turn oversight into rubber-stamping. Why it kills production systems. Oversight fails through automation bias, complacency, and inappropriate reliance: humans often do not catch AI errors even when they could, so a nominal review step can produce the appearance of control without the substance (Springer, AI and Ethics, 2026). The consequences are not hypothetical. In Moffatt v. Air Canada (2024 BCCRT 149, 2024-02-14), the British Columbia Civil Resolution Tribunal held the airline liable for its chatbot's negligent misrepresentation and awarded CAD $650.88, finding the company "did not take reasonable care to ensure its chatbot was accurate." A nominal review process would not have prevented that outcome; a real one might. In regulated, large-enterprise deployments we have supported, a review queue that had never once rejected an output was the earliest sign the loop had already collapsed. Pre-deploy detectable signal. Measure the reviewer override or rejection rate in staging. A rate at or near zero means the human is confirming, not deciding — the loop has already collapsed before launch. Mitigations that double as ongoing signals: sample-audit approvals, rotate reviewers, and train reviewers on known failure modes. We wire this measurement into the release gate itself; see deploy and verify AI.

Last updated: July 2026. A living, open taxonomy of how enterprise AI systems fail in production — with a pre-deploy detectable signal for each mode.

What is the production-AI failure taxonomy, in one paragraph?

Most enterprise AI systems fail after they are built, not because the model is inaccurate but because production surfaces failure modes that pre-launch tests never checked. MIT Project NANDA found ~95% of enterprise GenAI pilots show no measurable P&L impact. This taxonomy maps the recurring modes behind that number, and how to catch each one before you ship.

Why does everyone cite the 95% but nobody maps the why?

The headline statistic is now everywhere. In The GenAI Divide: State of AI in Business 2025, MIT Project NANDA reported that roughly 95% of enterprise GenAI pilots produce no measurable P&L impact, after $30–40B of enterprise GenAI spend, with only about 5% reaching rapid revenue acceleration (MIT Project NANDA, 2025; Fortune coverage, 2025-08-18). NANDA framed the root cause as a "learning gap" — tools that do not retain feedback or adapt to context — not model quality.

Other measurements point the same direction. S&P Global Market Intelligence found the share of companies abandoning most of their AI initiatives rose to 42%, up from 17% the prior year, with the average organization scrapping 46% of proof-of-concept projects before production (S&P Global, Voice of the Enterprise: AI & ML, 2025). Gartner projects that over 40% of agentic AI projects will be canceled by end of 2027, citing cost, unclear business value, and inadequate risk controls (Gartner, 2025-06-25). RAND, studying 65 data scientists and engineers, put the AI project failure rate above 80% — roughly twice the rate of non-AI IT projects — and attributed it to organizational causes, not the technology (RAND, RRA2680-1, 2024). And the AI Incident Database recorded 233 incidents in 2024, a 56.4% jump over 2023 (Stanford HAI, AI Index Report 2025).

The statistics agree that the problem is real and structural. What they do not give you is a map of the specific mechanisms, or a way to detect each one before launch. That is what this taxonomy is for. We define five failure families. For each, we give a crisp definition, why it kills production systems, and one pre-deploy detectable signal — something you can measure or check before the system ships. The families draw on public evidence and on generalizable delivery lessons from our own work, described only in non-identifying terms.

For our shorter argument on why pilots stall, see why AI pilots fail. That piece reads post-launch symptoms; this one names the pre-deploy mechanism behind each. This article is the structured reference behind it.

The taxonomy at a glance

Failure mode	What it is	Pre-deploy detectable signal
1. Data-integrity drift	Input distributions or input→label relationships shift after deployment, silently degrading a model that passed launch tests.	No data contract at the boundary and no measured distribution baseline. Compute PSI/KS divergence between training data and a recent production-representative holdout; large divergence, or the absence of any schema/freshness/null-rate contract, predicts drift failure.
2. Liveness mistaken for outcome	The system is live, adopted, and "working" by usage and uptime, but produces no measured business outcome.	Success is defined as launch, usage, or uptime rather than a pre-registered business KPI with a baseline and a holdout or control. If no one can state the outcome metric and its current baseline before ship, the system will read as live but never demonstrate impact.
3. Evaluation gaps	The eval that greenlit the system does not predict production behavior — because it is contaminated, saturated, or unrepresentative.	The go/no-go rests on a public benchmark, with no held-out, task-representative, contamination-resistant eval built from your own data. Re-score on a private holdout the model has never seen; a large gap versus the public score is the tell.
4. Orchestration brittleness	Multi-step, multi-agent, or tool-calling pipelines fail from coordination and control-flow faults, not from a weak underlying model.	No trace-level evaluation of the full flow, no explicit termination or timeout tests, no fault injection on tool and agent calls. If you only evaluate single turns, orchestration faults stay invisible until production.
5. Human-in-the-loop collapse	A reviewer is nominally in the loop but exercises no real control; automation bias and throughput pressure turn oversight into rubber-stamping.	Measure the reviewer override or rejection rate in staging. A rate at or near zero means the human is confirming, not deciding — the loop has already collapsed before launch.

The sections below expand each mode.

Failure mode 1 — What is data-integrity drift, and how do you catch it early?

Definition. Input distributions (data drift) or the relationship between inputs and labels (concept drift) shift after deployment, silently degrading a model that passed its pre-launch tests. Both produce measurable model drift over time (IBM, What Is Model Drift?; "Time to Retrain? Detecting Concept Drifts in ML Systems," arXiv 2410.09190, 2024).

Why it kills production systems. The model that shipped was validated against a snapshot of the world. Production is not a snapshot. When the input distribution moves — new customer segments, a changed upstream feed, a seasonal shift — accuracy erodes without any error being thrown. RAND identifies weak data foundations as among the most common root causes of AI project failure (RAND, RRA2680-1, 2024). The failure is quiet, which is what makes it dangerous.

Pre-deploy detectable signal. The tell is the absence of a data contract at the system boundary and the absence of a measured distribution baseline. Concretely: before launch, compute a divergence measure — Population Stability Index or a Kolmogorov–Smirnov statistic — between the training set and a recent, production-representative holdout. A large divergence at deploy time, or the lack of any schema, freshness, and null-rate contract, predicts post-launch drift failure. The fix is a contract plus continuous distribution monitoring with retraining triggers. See the AI data-integrity contract.

Failure mode 2 — Why do live systems still show no outcome?

Definition. The system is deployed, adopted, and visibly "working" — usage is up, uptime is green, demos land — yet it produces no measured business outcome. Activity is tracked; impact is not.

Why it kills production systems. This is the mechanism directly under the 95% figure. NANDA's finding was not that the pilots crashed. It was that they ran and delivered no measurable P&L impact, with the gap in learning, integration, and context rather than liveness (MIT Project NANDA, 2025). S&P Global adds a warning sign: perceived positive impact of GenAI fell year over year across every enterprise objective — revenue 81%→76%, cost 79%→74%, risk 74%→70% — even as adoption rose (S&P Global, 2025). RAND's leading root cause is blunt: stakeholders misunderstand or miscommunicate the problem the system is meant to solve (RAND, RRA2680-1, 2024). An unstated outcome metric is that misunderstanding made concrete.

Pre-deploy detectable signal. Look at how success is defined. If it is launch, usage, or uptime rather than a pre-registered business KPI with a baseline and a control or holdout, you have this failure before you ship. The test is simple: ask whether anyone can state the outcome metric and its current baseline before launch. If not, the system will read as live and never demonstrate outcome. We treat this distinction as a first-class delivery gate — see production AI: liveness vs outcome and how do you know an AI system works in production.

Failure mode 3 — When does a passing eval lie to you?

Definition. The evaluation that greenlit the system does not predict production behavior, because it is contaminated by training exposure, saturated, or unrepresentative of the real task.

Why it kills production systems. A benchmark score is only useful if it correlates with the behavior you care about. Contamination and saturation break that correlation. Scale AI built GSM1k, a fresh test in the style of the widely used GSM8K, and some models dropped by up to roughly 13 points, with the drop correlated to a measured memorization signal for GSM8K problems ("A Careful Examination of LLM Performance on Grade School Arithmetic," arXiv 2405.00332, 2024). Identical model weights can score materially differently depending only on the harness. A model that looks ready on a public leaderboard can be well short of ready on your task.

Pre-deploy detectable signal. The warning sign is a go/no-go decision that rests on a public benchmark, with no held-out, task-representative, contamination-resistant eval built from your own data. Detect it directly: re-score the system on a private holdout it has never seen. A large gap versus the public number is the signal that your eval is not measuring your task. For depth on models like Claude, a strong holdout is also where a careful judge harness earns its keep. See the AI evaluation harness.

Failure mode 4 — Why do multi-agent pipelines break when the model is fine?

Definition. Multi-step, multi-agent, or tool-calling pipelines fail from coordination and control-flow faults — not from a weak underlying model.

Why it kills production systems. The most detailed public evidence here is MAST, a multi-agent system failure taxonomy built from over 1,600 annotated traces across 7 frameworks, with inter-annotator agreement of κ=0.88 ("Why Do Multi-Agent LLM Systems Fail?", UC Berkeley et al., arXiv 2503.13657, 2025). MAST catalogs 14 failure modes in 3 categories. System-design issues account for 44.2% of failures — including step repetition (15.7%), being unaware of termination conditions (12.4%), and disobeying task specification (11.8%). The authors conclude that failures stem from poor system design, not model performance. Gartner's projection that over 40% of agentic projects will be canceled by 2027 points partly at the same brittleness (Gartner, 2025-06-25).

Pre-deploy detectable signal. The tell is no trace-level evaluation of the full multi-step flow, no explicit termination or timeout tests, and no fault injection on tool and agent calls. If you evaluate only single turns, the orchestration faults — loops, lost history, missed stop conditions — are invisible until production. Evaluate the whole trace, and inject faults deliberately. This also informs the architecture choice itself: see single vs multi-agent.

Failure mode 5 — Is your human-in-the-loop actually in control?

Definition. A human reviewer is nominally in the loop but exercises no real control. Automation bias and throughput pressure turn oversight into rubber-stamping.

Why it kills production systems. Oversight fails through automation bias, complacency, and inappropriate reliance: humans often do not catch AI errors even when they could, so a nominal review step can produce the appearance of control without the substance (Springer, AI and Ethics, 2026). The consequences are not hypothetical. In Moffatt v. Air Canada (2024 BCCRT 149, 2024-02-14), the British Columbia Civil Resolution Tribunal held the airline liable for its chatbot's negligent misrepresentation and awarded CAD $650.88, finding the company "did not take reasonable care to ensure its chatbot was accurate." A nominal review process would not have prevented that outcome; a real one might. In regulated, large-enterprise deployments we have supported, a review queue that had never once rejected an output was the earliest sign the loop had already collapsed.

Pre-deploy detectable signal. Measure the reviewer override or rejection rate in staging. A rate at or near zero means the human is confirming, not deciding — the loop has already collapsed before launch. Mitigations that double as ongoing signals: sample-audit approvals, rotate reviewers, and train reviewers on known failure modes. We wire this measurement into the release gate itself; see deploy and verify AI.

How to use this taxonomy

Treat the five modes as a pre-deploy checklist, not a post-mortem catalog. Before launch, ask five questions. Is there a data contract and a measured distribution baseline? Is success a business KPI with a baseline, not usage or uptime? Does a private holdout confirm the public score? Is the full multi-step flow evaluated with termination and fault-injection tests? Is the reviewer override rate above zero in staging? A "no" to any of these is a detectable signal that one of the five failures is already present — before it costs you a production incident or a scrapped initiative.

This is a living document. The evidence base is growing, and the taxonomy will be refined as new public post-mortems and studies land. If you are citing the 95%, this is the companion resource: the mechanisms behind the number, and what to check before you ship.

Sources: MIT Project NANDA, "The GenAI Divide: State of AI in Business 2025" (2025; Fortune, 2025-08-18); S&P Global Market Intelligence, "Voice of the Enterprise: AI & ML" (2025); Gartner press release (2025-06-25); RAND, "The Root Causes of Failure for Artificial Intelligence Projects," RRA2680-1 (2024); Stanford HAI, AI Index Report 2025, citing the AI Incident Database; IBM, "What Is Model Drift?"; arXiv 2410.09190 (2024); Scale AI, arXiv 2405.00332 (2024); "Why Do Multi-Agent LLM Systems Fail?", arXiv 2503.13657 (2025); Springer, AI and Ethics (2026); Moffatt v. Air Canada, 2024 BCCRT 149 (2024-02-14). Figures are attributed to their original sources and are not NewGenApps measurements.

Book an AI working session