In short. An AI system works in production when it produces evidence of evaluated reliability: measured performance on a frozen, representative, leak-free held-out set — which bounds, not guarantees, production performance, under the assumption that the held-out set matches live traffic — confirmed by outcome instrumentation on live traffic, and checked by someone other than its author. A confident demo, a green dashboard, and a high offline accuracy score are each correlated with a system that works. Each can also be entirely true while the system is failing. The standard matters because the gap between "looks right in the notebook" and "delivers the intended outcome at scale, consistently" is where most AI deployments quietly fail. OpenAI's evaluation guidance (developers.openai.com, confirmed live 2026-06-25) is direct: a common failure mode is "creating eval datasets that don't faithfully reproduce production traffic patterns" — meaning even teams who run evaluations often answer the wrong question.
What does "works" even mean for an AI system?
"Works" means the system produces the intended outcome reliably on the inputs it actually encounters — not on a curated sample, not on average across a test set that was tuned alongside the model.
Three things get conflated, and the conflation is expensive:
Works in the demo means: on a narrow set of inputs chosen by the person who built the system, the system produced output that looked right. A demo has no denominator. It shows you the system can succeed; it says nothing about how often it does.
Works on paper means: on an offline test set the build team controlled, measured at some point during or after development, the system hit a target score. This is more than a demo. It is still measuring performance in a different universe than the one the system runs in.
Works in production means: on the actual distribution of inputs the deployed system encounters, the system produces the intended outcome — and someone has the measurement to prove it. That third definition is the only one that matters for a system in service.
Most AI projects never get an honest answer to this question. The 95% statistic from MIT Project NANDA's 2025 research — that the vast majority of GenAI pilots show no measurable P&L impact — reflects, at least in part, a failure to distinguish these three. That said, production AI does work at scale for a minority of organisations. McKinsey's "State of AI in 2025" (published November 5, 2025; n=1,993 in 105 countries) found that roughly 6% of organisations attribute more than 5% EBIT impact to AI — a small but real cohort of high performers — and 39% attribute any measurable EBIT impact at all. LangChain's "State of Agent Engineering" (survey conducted November 18 – December 2, 2025; n=1,340) found that 57.3% of surveyed teams now have agents running in production. The question this page addresses is not whether production AI can work, but what distinguishes the teams who can demonstrate it from those who cannot. The answer is the evidence regime — evaluated reliability — not deployment alone. The NIST AI Risk Management Framework (AI RMF 1.0, January 2023) puts the requirement plainly: AI trustworthiness requires systems that are "valid and reliable," with test, evaluation, verification, and validation (TEVV) tasks spanning the entire AI lifecycle, not just the pre-launch gate.
Why can't you tell from a demo, a dashboard, or accuracy alone?
Each of the three common proof sources — a demo, a dashboard, and an offline accuracy score — measures something real. Each fails to measure evaluated reliability on live production inputs.
A demo is a sample of size roughly one, on inputs the presenter chose. It has no denominator. What a demo hides is technically called the input distribution: it shows you the mode of the system's behavior, never the tail — and AI systems fail in the tail. A system that handles the ten prompts in the demo can degrade silently on the ten thousand input variations it encounters in practice.
A green dashboard measures liveness: the service is up, latency is acceptable, error rate is within tolerance, throughput is normal. It does not measure outcome — whether the answers were correct, whether the decisions were good. A system can be fully live and entirely wrong: it can return a confident, well-formed, on-time response every time, and the response can still be incorrect. The full derivation of the liveness-versus-outcome distinction belongs to a separate treatment; the one-liner this page owns is: green means "it ran," not "it was right." See production AI liveness vs outcome for the monitoring half of this question.
High offline accuracy is the most deceptive of the three proxies, and where the precision matters most. A model can score high on its own test set and fail silently on production inputs, for three independent reasons:
Train-serving skew. The data or feature pipeline differs between training and serving, so the model encounters different inputs in production than it was evaluated on offline. Google's engineering canon names this directly: Rule #37 of Martin Zinkevich's Rules of Machine Learning (Google for Developers, last updated 2025-08-25) states that "training-serving skew is a difference between performance during training and performance during serving," and recommends explicit monitoring because it otherwise goes unnoticed.
The offline-to-online gap. Even with no skew, offline metrics are proxies for the online objective, and the two correlate weakly. Castells and Moffat, writing in AI Magazine (Wiley/AAAI, Vol. 43, No. 2, June 2022, DOI: 10.1002/aaai.12051), documented for recommender systems that "offline procedures do not yield results sufficiently correlated with" online controlled experiments — a finding that generalizes across ML systems to the extent that the same model sees different traffic offline and online.
Contamination and leakage. If the held-out set is not truly held out — if near-duplicates or label-revealing features appear in training — the offline number measures memorization, not generalization. A study on inference-time decontamination (Zhu et al., Inference-Time Decontamination: Reusing Leaked Benchmarks for LLM Evaluation, arXiv:2406.13990, EMNLP Findings 2024) showed this can account for removing 19.0% of MMLU score inflation. The benchmark saturation problem compounds this: a 2026 study by Akhtar et al. and the Evaluating Evaluations Coalition (arXiv:2602.16763v1) found that 29 of 60 AI benchmarks examined show "high or very high saturation," meaning frontier models can no longer be statistically distinguished on them — the benchmarks themselves have degraded as measurement instruments.
None of these three proxies is worthless. A demo is a reasonable first signal. A green dashboard is a necessary operations requirement. Offline accuracy is a prerequisite for any serious deployment. They are necessary, not sufficient.
What evidence actually proves an AI system works?
Three types of evidence together constitute proof. No single layer catches every failure mode; the goal is a measurement system where what slips past one check is caught by another.
1. A held-out evaluation result on a frozen, representative set
The held-out (test) set is data the model never saw during training or selection, used to estimate how the system will perform on unseen inputs. Two requirements are non-negotiable:
Frozen and leak-free. The set must be fixed before evaluation begins and must not be modified after. Temporal holdout — testing on data that is strictly later than the training cut — is the practitioner's primary defense against contamination, because static sets that persist in the public domain eventually leak into training corpora. The industry shift toward dynamic, time-refreshed evaluation (see Chen et al., arXiv:2502.17521, February 2025) reflects exactly this problem.
Representative of production traffic. A frozen set that does not match the distribution of inputs the deployed system actually receives gives a precise answer to the wrong question. OpenAI's evaluation guidance identifies "creating eval datasets that don't faithfully reproduce production traffic patterns" as a named anti-pattern; its recommendation is to build and maintain eval sets from production data over time.
The precise claim: a held-out evaluation result bounds expected production performance — under the assumption that the held-out distribution matches live traffic. The moment that assumption breaks (input drift, a new user segment, a changed feature pipeline), the bound is void. The copy for this does not say "guarantees" or "proves" or "ensures." It says: this is the best available estimate, stated honestly, with a denominator and a confidence interval. A metric without an interval and without a denominator is an impression, not a measurement.
2. Outcome instrumentation on live traffic
Not "is it serving?" but "did it produce the right outcome?" These are different instruments. Cleanlab's "AI Agents in Production 2025" survey (August 2025), restricted to the cohort already running agents in production (n=95), found that fewer than one in three express satisfaction with their observability and guardrail solutions — and 62% plan to improve observability in the next year. Gartner, in a May 2026 press release, predicts that 40% of organizations deploying AI will implement dedicated AI observability tools by 2028. The implicit message of both: even as of mid-2026, most production AI is instrumented heavily for liveness and barely for outcome.
The reason outcome monitoring is hard — and therefore skipped — is label latency: ground truth often arrives days or weeks after the prediction. Senior teams build the delayed-label join anyway, because a lagging truth is more valuable than a real-time guess at what "looks fine." See production AI liveness vs outcome for the monitoring half of this question.
3. An independent check by someone other than the author
The person who built the system has structural incentives to find it working. This is not a competence claim; it is an incentive-alignment claim. The build team, by definition, knows the inputs the system was designed for, the edge cases they handled, and the framing that makes the outputs look good. An independent evaluation — whether a separate internal team, an external review, or an adversarial red-team — is the control that catches the gaps the author's own testing structurally misses.
The independent-check requirement is sharpest for LLM-based systems, where "LLM-as-judge" evaluation is increasingly common. Using a model to evaluate a model is legitimate and scalable, but it inherits systematic biases: Shi et al., in a 2025 AACL-IJCNLP study across 15 LLM judges (arXiv:2406.07791v8), found that simply swapping the presentation order of the answers changes which response a judge prefers — they quantify this as judges' position consistency and preference fairness varying substantially across models and tasks. Position bias, self-preference, and verbosity bias are not edge cases — they are consistent properties of unaudited LLM judges. An LLM judge that has not been calibrated against human labels, with order randomized and identity masked, is not an independent check; it is the same model family marking its own homework. Independence is a property of the control structure, not of the tool.
The three types of evidence together — held-out result, outcome heartbeat, independent check — are the definition, not the evaluation harness mechanics. What to measure, which metrics, which thresholds, and how to build the infrastructure for continuous evaluation are the scope of the eval-harness pillar (see that piece when live for the mechanism behind this definition).
"Looks like it works" vs "Proven to work"
| Signal | Looks like it works | Proven to work |
|---|---|---|
| Demo passes | Inputs were chosen by the presenter | Held-out evaluation on a frozen, representative, leak-free production-input set — with a denominator and a confidence interval |
| Dashboard is green | System is serving traffic (liveness) | Outcome heartbeat — a live signal that recent outputs were actually correct, not just served |
| Offline accuracy is high | Measured on the team's own test set, at training time | A number from the running system on current traffic, closing the offline-to-online gap |
| Model gives a confident answer | A confidence score is not calibrated reliability | Output checked against a ground-truth or human-review standard by an independent evaluator |
| Ticket is closed / pilot is "done" | Work is done; whether it works is an open question | Freshness verified on inputs, and an end-to-end production probe confirms the live system still produces the right output today |
Frequently asked questions
Is there a standard definition of "AI evaluation" that I can cite?
Yes. The NIST AI Risk Management Framework (AI RMF 1.0, January 2023) establishes "valid and reliable" as a core AI trustworthiness characteristic and defines TEVV — test, evaluation, verification, and validation — as a practice that spans the full AI lifecycle. OpenAI's evaluation guidance (developers.openai.com, confirmed live 2026-06-25) formalizes held-out evaluation and continuous evaluation (CE) on production-representative data as current practitioner norms. The working definition this page proposes is the most precise version of that accepted concept: evaluated reliability means measured performance on a frozen, representative, leak-free held-out set, bounded not guaranteed, confirmed by outcome instrumentation and an independent check.
Can't the team who built the system also evaluate it?
They can measure it; independent evaluation is needed to check it. The structural issue is incentive alignment, not competence. A build team knows which inputs the system handles well, which prompts were tuned, and which edge cases were already addressed. That knowledge shapes their test set, even unintentionally. An independent reviewer does not carry those biases. This is the same reason financial audits require an external auditor and clinical trials require an independent data safety monitoring board — not because internal teams are dishonest, but because the control structure requires separation.
What if our AI system is too new to have production data to evaluate against?
This is exactly when a frozen held-out set drawn from realistic synthetic or pilot inputs matters most. Pre-production, the goal is to construct a representative set from the best available proxy for real traffic: domain-expert-generated inputs, adversarial probes, and inputs sampled from any pilot or shadow-mode run. An honest evaluation on an imperfect-but-documented set is more useful than no evaluation, because it at least states explicitly what it does and does not cover. The eval-harness mechanics — how to build that set, what to measure, and how to maintain it as production traffic accumulates — are covered in the eval-harness pillar (see that piece when live).
How is this different from traditional software testing?
Traditional software tests deterministic behavior: given input X, the system must produce output Y. Pass or fail. AI evaluation tests probabilistic reliability across a distribution of inputs, which requires statistical thinking, not just pass/fail cases. The right question is not "did it get this one right?" but "what is the expected performance, with what variance, across the range of inputs it will encounter?" That shifts the measurement apparatus from a checklist to a sample, from a binary to a distribution, and from a one-time gate to a continuous instrument. The metric itself is a modeling choice driven by which type of error is expensive in the specific system — there is no universal "accuracy."
How do I know the production system is still the version that passed evaluation?
That is the deploy-verification question, distinct from the evaluation question. Evaluation tells you whether the validated version works. Deploy verification tells you whether the thing now serving traffic is the validated version — whether what was merged is what was shipped, and whether it remains what is running. Those are different questions with different instruments.
The gap this question reveals
If the answer to "how do we know it works?" in your organisation is currently "it passed the demo" or "it's been running for three months without complaints," the gap is not unusual. A system that doesn't throw errors is not the same as a system that is right. Models degrade quietly — they do not throw exceptions when their outputs become less reliable, which means silence is the default failure mode of an unmonitored system, not a signal of health.
The starting point is agreeing on what "works" means for the specific system, then building the smallest evaluation that would give an honest answer. If you want to think through what that looks like for your context, how we work describes the method — a small senior team, independent verification, evaluated reliability; not a demo and a handover. Or book an AI working session to start from your system's current evidence state.
NewGenApps — production AI, proven. Stay a step ahead, always.
Sources cited in this piece: NIST AI Risk Management Framework (AI RMF 1.0), January 26, 2023; OpenAI, "Evaluation best practices," developers.openai.com, confirmed live 2026-06-25; Zinkevich, M., Rules of Machine Learning (Rule #37), Google for Developers, last updated 2025-08-25; Castells, P. & Moffat, A., "Offline Recommender System Evaluation: Challenges and New Directions," AI Magazine (Wiley/AAAI), Vol. 43, No. 2, pp. 225–238, June 2022, DOI: 10.1002/aaai.12051; Akhtar, M. et al. and the Evaluating Evaluations Coalition, arXiv:2602.16763v1, 2026; Cleanlab, "AI Agents in Production 2025," August 2025, production cohort n=95; Gartner, AI observability press release, May 12, 2026; Shi, L. et al., "Judging the Judges," AACL-IJCNLP 2025 (ACL Anthology), arXiv:2406.07791v8; Zhu, Q. et al., "Inference-Time Decontamination: Reusing Leaked Benchmarks for LLM Evaluation," arXiv:2406.13990, EMNLP Findings 2024; Chen, S. et al., arXiv:2502.17521, February 2025; MIT Project NANDA, "The GenAI Divide," 2025 research (see the linked 95% pilot explainer for the full citation); McKinsey, "The State of AI in 2025: Agents, Innovation, and Transformation," published November 5, 2025, survey June 25 – July 29, 2025, n=1,993; LangChain, "State of Agent Engineering," survey November 18 – December 2, 2025, n=1,340.