Book a 30-minute working session →
## Who this is for Artificial intelligence consulting from NewGenApps fits best when: - You are a **CIO, CTO, CDO, Head of Data, or founder** who needs AI in production, not another strategy artifact. - You have **a real use case** — support deflection, document intelligence, an internal agent, knowledge retrieval over your own data — and need a team who can ship it. - You have **tried a POC that stalled before production**, and you want a partner who treats "reaches production" as the actual finish line. - You want **senior practitioners**, not a pyramid of junior consultants learning AI on your budget. If you are still deciding whether AI is worth it at all, the 30-minute working session is built for exactly that — an honest read on where the value is, with no obligation. ## Why a track record of more than fifteen years is a moat, not a footnote The first "E" in Google's E-E-A-T framework — and in any enterprise buyer's checklist — is **Experience.** It cannot be manufactured retroactively, and most firms now marketing AI consulting were founded after the current cycle began. Our judgment is timestamped, and the timestamps are public. Our [first blog post, October 5, 2008](/blog/The-journey-has-begun/) read the mobile-cloud-SaaS shift before it arrived — that "the complexity is moving to the cloud and the convenience is moving to small devices like cell phones," named in the same paragraph as iTunes, Android Marketplace, cloud-run business software, and Salesforce. We were building on the platform we called: our [Android Emulator post of October 9, 2008](/blog/Android-Emulator/) is dated to the launch window of the first Android phone, and we [demonstrated augmented reality on that first Android phone (the G1) in 2009](https://vimeo.com/618305787) — in native code, three years before our first written post on AR. The pattern is the proof: read the signal early, learn the raw stack, ship before the category has a name. Choosing a firm for an AI program is a bet on its judgment about what to build and when. Each claim above links to its dated original. For the wider record, browse our [insights archive](/blog/) and the [AI](/blog/tag/ai/) and [machine learning](/blog/tag/machine-learning/) tag hubs, or read the full account on our [about](/about/) page. ## Frequently asked questions **How long does an artificial intelligence consulting engagement typically take?** Engagement length depends on scope. A feasibility and architecture phase typically runs four to eight weeks. A full build-to-production engagement for a defined system — integration, testing, and handover included — typically runs three to six months. Firms that promise production outcomes in weeks without prior architecture work are compressing the wrong phase. **What industries does NewGenApps work in?** NewGenApps has delivered AI systems for organizations in retail and e-commerce, financial services, and technology-product companies, across multiple geographies. Sector breadth matters less than delivery depth — we do not take engagements where a production outcome is not feasible within the agreed scope. **What is meant by "independent verification" in AI consulting?** Independent verification means the quality and reliability of an AI system is assessed by a party other than the team that built it. In practice: a separate technical review of model outputs, an adversarial test of failure modes, and a performance benchmark against the agreed baseline — not a self-assessment by the delivery team. **How is working with a boutique AI consulting firm different from building an in-house AI team?** An in-house team builds accumulated context about a single organization; a boutique consultancy brings pattern recognition across multiple delivery environments. The right answer often depends on whether the AI capability is core to your competitive differentiation or a capability-for-hire. Build-vs-buy is best understood as a delivery-capacity-and-time decision, not a pure cost comparison — you are buying production-delivery experience and the time it takes to acquire, plus the capability you most want transferred back. See [build or buy AI](/build-vs-buy-ai-team/) for the full trade-off analysis. **What happens after an AI system is deployed?** Deployed AI systems require monitoring for performance drift (models degrade as real-world data distributions shift), a clear process for flagging and investigating failures, and a team capable of making changes. NewGenApps builds operational handover into every engagement — the system delivered is one your team can run, not one that requires the consulting firm to remain on retainer to function. This matters at enterprise scale; see [AI in the enterprise](/ai-in-the-enterprise/). ## The bottleneck is delivery, not the model The wider context is sobering: roughly 95% of enterprise generative-AI pilots showed no measurable P&L impact (MIT NANDA, *The GenAI Divide*, Aug 2025), and 42% of firms abandoned most of their AI initiatives, up from 17% the prior year (S&P Global Market Intelligence / 451 Research, *Voice of the Enterprise: AI & Machine Learning*, reported March 14, 2025) — overwhelmingly for delivery reasons, not model quality. Production AI, proven on live data, is the response to that record. The way to test whether it applies to your case is a single conversation — no deck, no obligation.Book a 30-minute working session →
============================================================================== # AI Strategy & Roadmap That Ends in Shipped Code | NewGenApps URL: https://newgenapps.com/ai-strategy/ ==============================================================================  # AI Strategy That Ends in a Shipped POC, Not a Deck Most enterprise AI strategy is a 60-slide deck that ages out before the ink dries. By the time the committee signs off, the models have moved and nobody has shipped anything you can click. We think that's backwards. **The best AI strategy is the one you can prove** — a working build with an eval set behind it, measured against acceptance thresholds, not a claim in a slide. Strategy follows shipping, not the other way around. That comes from more than fifteen years of calling tech waves early — staying a step ahead, always: the cloud-plus-mobile thesis in 2008, and augmented reality running on the HTC Dream (the T-Mobile G1, the first Android phone, which launched in late 2008) in 2009. The pattern never changes: sound strategy, beautiful deck, nothing built. So we flipped the order: AI strategy that produces a working proof of concept inside the engagement. ## Why deck-only AI strategy fails A strategy document is a hypothesis. Until you build something, you don't actually know: - **Whether your data supports the use case.** A high-value idea often runs into data that isn't ready — Informatica's CDO Insights 2025 survey (n=600) found data quality and readiness tied with technology limitations as the most-cited obstacle to AI, each at 43%. You learn that in week two of a build, not in a slide. - **What it costs to run.** Latency, token spend, and model tier (a small model vs. a frontier model) can materially change unit economics — and you can only size it against your real workload, not a slide. - **Whether people will use it.** Adoption is a product problem — you find the friction by putting a clickable thing in front of a real user. - **Whether it can reach production.** Most pilots never get there: IDC's CIO Playbook 2025 for Lenovo (February 2025) found 88% of AI proof-of-concepts fail to reach widescale deployment — roughly four of every 33 make it. And the build path matters: MIT's NANDA *GenAI Divide* report (August 2025) found internally built tools reached deployment about half as often as partner-built ones (~33% vs. ~67%, roughly twice the success rate when you build with a partner). Deck-only strategy optimizes for consensus. Shipping optimizes for truth. > The best AI strategy is the one you can demonstrate. A deck argues; a working POC proves. ## How we do it: the AI Compass Our [AI Compass opportunity audit](/services/) turns "we should do something with AI" into a ranked, costed roadmap **and** a scoped first build. It runs in weeks, not quarters. How the ranking and the red, amber or green data verdict work is in [how to prioritize AI use cases](/ai-use-case-prioritization/). ### What it covers 1. **Opportunity inventory.** Every candidate use case — support deflection, document intelligence, knowledge assistants, agentic workflows. 2. **Impact x feasibility scoring.** Each idea plotted on business impact and feasibility (data readiness, integration, model fit, change load). The step deck-only strategies hand-wave. 3. **Data and readiness check.** We look at the actual data behind the top candidates before promising anything — pipelines we've built since 2016. 4. **A sequenced roadmap.** Quick wins first, then the bigger bets, each with an honest estimate of effort, cost, and risk. 5. **A scoped POC.** The single best use case, specced as a scoped, ready-to-build POC — a tightly scoped first build, typically a few weeks, contingent on data readiness. It ships with how it will be measured: an offline eval set, explicit acceptance thresholds, and basic observability, so "it works" is something you can check rather than assert. A Monday item, not a someday item. ### Prioritization, honestly Sequencing is the whole game. Do the right thing first and the next three get easier. - **High impact, high feasibility** → do it now. This is your POC. - **High impact, low feasibility** → strategic bets, after a quick win earns room to invest. - **Low impact, high feasibility** → fast credibility wins. - **Low impact, low feasibility** → say no, in writing. The output isn't a maturity model. It's a ranked backlog with a build attached to the top.Book a 30-minute working session →
## Strategy follows shipping Once the first POC ships, strategy becomes a loop. You ship a slice, measure it against the thesis (did deflection actually move?), and the roadmap updates with real numbers instead of slideware. Each shipped thing de-risks the next — dodging both endless strategy and aimless pilots. ## What you leave with - A **prioritized roadmap** scored on impact x feasibility, with effort and cost estimates. - A **data and readiness assessment** of your top use cases. - A **scoped, ready-to-build POC** — a tightly scoped first build, typically a few weeks and contingent on data readiness, specced with the non-functionals it needs to become a production slice: an eval harness, observability, and a security and data-governance review. - An **honest "not now" list** so the roadmap is a decision, not a wish. - A **board-ready narrative** that pairs the plan with the evidence behind it. We don't claim to be an official Anthropic partner — we're building toward those credentials. Our engineering depth is agentic delivery: we build with Claude Code and a library of custom skills, the agent-driven workflow that lets a small senior team ship and verify production AI fast — model-flexible, not locked to one vendor. See how that shows up in the [work we deliver](/work/). ## Start with a working session, not a deck Book a **30-minute AI working session** — no slides, no pitch. We'll pressure-test your top use case live and tell you honestly whether it's a Monday-morning build or a six-month foundation project. Stop talking about AI. Start shipping it. **[Book your AI working session →](/contact/)** ============================================================================== # Agentic AI: Building AI Agents for Business | NewGenApps URL: https://newgenapps.com/agentic-ai/ ==============================================================================  # Agentic AI: agents that do the work, not just talk about it **In short:** Agentic AI is software that uses a large language model to plan and complete multi-step tasks on its own — deciding what to do next, calling your tools and systems, acting toward a goal. NewGenApps builds production AI agents that are verified to work — tool use, MCP, guardrails, and evals — model-flexible, with senior engineers from scope to ship. Most "AI agent" content is a chatbot in a trench coat — it answers in a window and won't move a number in your business. An agent earns the name when it can take a goal, work it across your systems, check itself, and hand back a result you can trust. That's the line we hold — building on the frontier for more than fifteen years. Staying a step ahead, always. ## What is agentic AI? The model gets a goal plus the means to pursue it — tools, data, autonomy to sequence the work — and runs until it's done. The difference is structural: a chatbot responds in one turn; an agent plans, calls a tool, reads the result, picks the next step, and repeats until the goal is met or a guardrail stops it. Three capabilities make it agentic: 1. **Tool use.** It invokes real functions — search a knowledge base, hit an API, write to a system of record — and uses what comes back. 2. **Multi-step planning.** It sequences those calls toward a goal, adapting when a step fails instead of running a fixed script. 3. **Grounded context.** Via standards like MCP, it connects to your data and tools in a structured, auditable way — acting on your reality, not a hallucination. > If a vendor can't draw the line between a chatbot and an agent cleanly, they're selling you a demo. ## What are the real enterprise use cases for agentic AI? Agentic AI earns its budget on tasks that are multi-step, rules-bound, and eating senior people's time: - **Customer support.** Not a deflection bot — reads the ticket, pulls order and history, checks policy, then resolves the issue or hands a human a prepped case. - **Sales and revenue ops.** Enrich leads, draft outreach grounded in CRM context, route opportunities to the right rep. (We built and ran our own AI lead-enrichment product, end to end.) - **Back-office and finance.** Invoice processing, reconciliation, exception handling — escalating only ambiguous cases. - **Knowledge work.** Cited drafts for contract review, due diligence, and competitive briefs. - **Engineering and IT.** Triage, log analysis, runbook execution, and first-pass remediation, with a human in the loop on anything destructive. Build an agent when the task is too varied for a rigid workflow tool but too repetitive to do by hand. Picking *which* to build first is an [AI opportunity audit](/services/) — the wrong first use case is the top reason agent projects stall. ## Single-agent or multi-agent AI — when does each win? A single agent wins when the work is one bounded job; multi-agent AI wins when the work spans roles that should not grade each other — gathering, deciding, acting, and checking. Adding agents is not a maturity badge. For a simple, low-stakes, single-step task, one well-scoped agent is the right tool, and a multi-agent system is over-engineering. The calculus changes with stakes and complexity. As the cost of a wrong answer rises and the work spans multiple steps, sources, and side effects, a single agent's hidden costs — undiagnosable failures, silent staleness, no separation between doing and checking — start to dominate. | Factor | Single agent | Multi-agent AI | |---|---|---| | **Best for** | One bounded task, one source, low stakes | Multi-step work across sources and side effects | | **Failure isolation** | One bad input contaminates the whole chain | Bounded scope — failures are attributable and contained | | **Verification** | Self-grades; the only check is its own opinion | A separate agent or deterministic code verifies the result | | **Cost** | Lowest to build and run | More components, more latency, an orchestration layer to engineer | | **When it's the wrong choice** | When a wrong answer is expensive or the work is genuinely multi-step | When the task is simple enough that the extra parts are pure overhead | Knowing which regime you are in — and not paying for coordination you do not need — is the judgment call. The engineering discipline behind a reliable multi-agent system is the subject of our deep-dive on [multi-agent systems in production](/multi-agent-orchestration-production/). ## How do we build agents that reach production? The gap between a slick demo and a system you'd trust with a customer or a ledger is where most agent projects die. We engineer for it from day one — the heart of our [deploy-and-verify](/deploy-and-verify-ai/) practice. Our engineering depth is agentic delivery — we build with Claude Code and a library of custom skills, the agent-driven workflow that lets a small senior team ship and verify production AI fast. We build and run on AWS — Bedrock, SageMaker, GPU compute, and Kiro — accelerated by NVIDIA — so the model itself stays a choice we make per task, never a lock-in. ### Tool use and MCP We give the agent a tight, documented set of tools and connect your systems through the Model Context Protocol where it fits — turning "the agent has access" into a structured, inspectable contract: defined tools, inputs, and permissions. ### The right model per step We route by difficulty: fast steps go to a smaller, cheaper model, hard reasoning to a larger one. Tiered model routing — plus prompt caching and batching — keeps an agent capable *and* affordable at volume. ### Guardrails Autonomy without guardrails is how agents make expensive mistakes: - **Scoped permissions** — only the tools it needs; write actions gated. - **Human-in-the-loop checkpoints** on anything irreversible — sending money, emailing a customer, deleting data. - **Input and output validation** before a tool fires and before a result returns. - **Fail-closed behavior** — when unsure, it stops and escalates rather than guessing. ### Evals You can't ship what you can't measure. Before launch we build a graded test set, agreed success criteria, and regression tests on every change — turning "it worked in the demo" into "it passes on 200 real cases, and we'll know the moment that slips." ## What a project with us looks like You don't have to bet a year to find out if this works: - **Scope it.** A focused [AI working session](/contact/) to find the one agent worth building first — high value, bounded risk, real data available. - **Prove it.** A fixed-scope POC sprint that turns it into a working agent you can test against real cases — in weeks, not quarters. - **Ship it.** We take it to production — deployed, monitored, evaluated — or embed a senior pod as your ongoing [AI team](/services/). Same senior engineers from first call to launch. No junior bench, no hand-offs. See [the work](/work/). ## A note on honesty We go deep where depth pays and stay model-flexible — we don't claim endorsements or partner credentials we don't hold. And we'll tell you when an agent is the wrong tool: if a deterministic workflow or simple integration solves your problem, that's what we'll recommend. Spotting a wave includes knowing when not to ride it. ============================================================================== # Enterprise RAG (Retrieval-Augmented Generation) | NewGenApps URL: https://newgenapps.com/rag/ ==============================================================================  # RAG: Chat With Your Documents, With Receipts **In short:** Retrieval-augmented generation (RAG) lets a language model answer from *your* documents instead of its training data — retrieving the relevant passages at query time and answering grounded in that evidence. Done right, every answer carries a citation back to the source. Done wrong, you get a confident chatbot that quietly makes things up. NewGenApps builds the first kind and proves it in a three-week POC — we don't just say it, we show it. Most enterprises don't have an AI problem. They have a "the answer is in *one* of these 40,000 PDFs and nobody can find it" problem. RAG turns that pile into something you can ask and trust. ## What RAG actually is The model is handed information *retrieved* from your knowledge source — documents, database, wiki — and answers using only that. Retrieval happens at the moment of the question, not during training. That one choice makes RAG the default for "chat with your documents": the model stays current, you control what it sees, and you can show the exact passage each claim came from. The alternative — asking a model cold and hoping its training data holds the right answer — is a non-starter for anything proprietary, time-sensitive, or auditable. ## RAG vs fine-tuning vs long context These three get confused constantly, and choosing wrong is the most expensive mistake we see. | Approach | What it changes | Best when | Watch out for | |---|---|---|---| | **RAG** | *What the model knows* at query time | Knowledge changes often, must be cited | Retrieval quality is the whole game | | **Fine-tuning** | *How the model behaves* — tone, format | You need consistent style, stable knowledge | Adds no fresh facts | | **Long context** | *How much you paste in* at once | Corpus fits the window, rarely changes | Cost scales with every token | The best systems combine them: RAG for the facts, a tuned prompt for behavior, long context for generous evidence. A large context window often makes a lightweight "stuff the docs in" approach viable for a focused corpus — sometimes that beats a heavyweight RAG stack you didn't need. Knowing when *not* to build RAG is part of the job. See [how we build](/claude-native-ai/) — we make that call with you. ## The architecture of a grounded system A production RAG system is a pipeline, not a single API call: 1. **Ingestion & parsing.** Pull documents from their sources; extract clean text, including scanned PDFs, tables, and slides. 2. **Chunking.** Split into self-contained passages. Chunking quietly determines half your answer quality. 3. **Embedding & indexing.** Convert chunks into vectors for fast semantic search, alongside keyword search. 4. **Retrieval.** Blend semantic and keyword search (hybrid), then **rerank** so the strongest evidence rises to the top. 5. **Generation.** Hand the question plus passages to the generation model, instructed to answer *only* from the evidence and cite each claim. 6. **Citation & guardrails.** Return links to sources, and say "I don't know" when the evidence isn't there. 7. **Evaluation.** Measure retrieval accuracy and answer faithfulness continuously — quality degrades silently as documents change. The strongest systems aren't AI magic; they're data engineering. We've built data pipelines and search since before "RAG" had a name — see our [big-data foundations](/blog/tag/big-data/). > Most "the AI is hallucinating" complaints are retrieval failures wearing a costume. If the right passage never reaches the generation model, no model can answer correctly. ## Common failure modes When a "chat with your documents" project disappoints, it's almost always one of these — and almost never the model: - **Bad retrieval, blamed on the model.** The right passage never reaches the generation model. - **Chunking that splits the answer.** A table severed from its header — indexed, but useless. - **No reranking.** Vector search buries the best evidence below the cutoff. - **No "I don't know."** A system that always answers will confidently answer what your docs don't cover. - **Citations that don't check out.** A citation the user can't click and verify is theater. - **No evaluation harness.** Quality erodes invisibly; you find out from an angry user, not a dashboard. - **Ignored access control.** Surfacing a passage the asker shouldn't see is a data-leak incident. ## How NewGenApps ships RAG We don't do "plug in a vector DB and see." We ship RAG the way you'd ship anything that faces real users and auditors: - **Grounded by default.** Every answer cites its sources and declines when the documents don't cover it. Trust is the product. - **Hybrid retrieval, reranked.** Semantic plus keyword with a rerank pass — the exact account number and the fuzzy concept both matter. - **Built with Claude Code and our skills library.** That's our actual engineering depth — the agent-driven workflow that lets a small senior team ship and verify a grounded system fast. The generation model stays swappable: we build on Amazon Bedrock, so a large-context, instruction-following model plugs in without a rewrite when the corpus calls for it. (To be clear: we're an independent firm building toward formal Anthropic credentials, not an official partner.) - **Evaluated, not vibes-checked.** A harness scores retrieval accuracy and faithfulness from day one. Quality is a number you watch. - **Access-aware.** Retrieval respects your permission model. - **POC to production in weeks.** Our [POC in 3 Weeks](/services/) turns one real document-search problem into a working, cited assistant — then we take it to production. This is the same builder-not-pundit posture that's run through our [work](/work/) for more than fifteen years: pick the wave early, then actually ship it. **Ready to chat with your documents and trust the answers?** [Book a 30-minute AI working session](/contact/) — no deck, no pitch, just where RAG would actually pay off. ============================================================================== # AI Rescue — Get Your Stalled AI Pilot to Production | NewGenApps URL: https://newgenapps.com/ai-rescue/ ============================================================================== # AI Rescue **In short:** AI Rescue recovers a pilot that has stalled before production — or decides, with evidence, that it should not be recovered. NewGenApps runs a structured ~90-day arc: triage the failure, stabilize and independently verify the system, then relaunch under a data-integrity contract — with the senior team that scopes the work doing the build. The first deliverable is a verdict, not a fix: recover, re-scope, or retire. The last is production AI, proven — a system your own team can run, confirmed to work by an independent check rather than asserted by the people who built it. If you have a pilot that performed in the demo and then went quiet — held at "promising" for a quarter or more, or quietly shelved — the constraint is rarely the model. It is the engineering between a controlled demonstration and a system that holds up in production: on real data, under load, against the inputs the demo never showed it. You are not an outlier in this. Only 25% of enterprises have moved 40% or more of their AI experiments into production (Deloitte, *State of AI in the Enterprise 2026*, survey field period August–September 2025, n=3,235). The production gap is the common case. It is also fixable — once you know whether this particular pilot is worth fixing. *Last reviewed: June 2026.* ## What is an AI Rescue engagement — and what does it actually deliver? An AI Rescue engagement is a time-boxed, evidence-led process that moves a stalled pilot to verified production — or produces a documented decision to re-scope or retire it. It is the difference between "we can build it" and "we will tell you, with evidence, in weeks, whether this should be built at all." A rescue is a decision problem before it is an engineering problem, so it returns one of three outcomes: **recover** (the approach is sound, the build broke), **re-scope** (the problem is real, the original target was wrong), or **retire** (the foundation cannot bear the load). The engineering begins only once the verdict says recover or re-scope. Three commitments make the eventual production claim trustworthy: a **data-integrity contract** — no synthetic, mocked, or silently stale data is allowed to inform a decision; **independent verification** — whoever builds a change does not get the final word on whether it works; and **evidence-based rollout** — capability earns its way back to production through measurable gates, not a date on a calendar. A small senior team carries this end to end: the engineer who diagnoses your pilot is the one who stabilizes it and verifies it in production, with no handoff to a junior layer. The same gaps that make a rescue necessary are the ones that stall pilots to begin with — worth reading on [why AI pilots stall](/why-ai-pilots-fail/) before deciding to rescue one. ## How long does it take to rescue a stalled AI pilot? Most stalled pilots reach a clear go/no-go verdict within roughly 90 days, structured across three phases: triage (weeks 1–4), stabilize-and-verify (weeks 5–10), and relaunch or retire (weeks 11–13). Each phase ends with evidence, not a status adjective, and each gate is explicit — we tell you what must be true to advance, and we say so when it is not. The arc is a typical default, not a guarantee; data-pipeline complexity, regulatory sign-off, and integration into brittle systems of record are the honest reasons it extends. ### Phase 1 — Triage (weeks 1–4) We establish, honestly, what you have and why it stalled. We trace the pilot end to end across its failure causes — data, evaluation, integration, ownership — and separate what is salvageable from what is not. If you are still diagnosing whether your pilot has actually stalled, [five signals tell you](/has-your-ai-pilot-stalled-5-signal-diagnosis/) before you commit to a rescue. What the first week asks, and how the recover, re-scope or retire call is made, is in [how to rescue a failing AI project](/how-to-rescue-a-failing-ai-project/). We define the precise question the rescue must answer and agree the success bar: the quality, cost, latency, and reliability thresholds the relaunched system must clear, set against the decision the output actually informs. **The gate to advance** is a shared, evidence-backed diagnosis and a costed recovery plan — including the candid possibility that the right answer is to re-scope or stop. ### Phase 2 — Stabilize and verify (weeks 5–10) We fix the root causes the triage found. The common instinct on a stalled pilot is to swap in a larger or newer model; that usually backfires, because ML systems entangle their signals — changing the model perturbs retrieval, prompt sensitivity, latency, cost, and failure modes at once, and the effect cannot be isolated without the evaluation harness the stalled pilot never had (Sculley et al., *Hidden Technical Debt in Machine Learning Systems*, NeurIPS 2015). So we build the measurement first: the data-integrity contract, the evaluation harness, the integration into your real systems, and defined behavior for edge cases, malformed inputs, and the "I don't know" path. Every meaningful change is scored against a frozen, held-out set drawn from your own data. Often this means re-architecting for production rather than trimming features — [what changes between pilot and production](/pilot-vs-production-ai-what-changes/) is rarely cosmetic. **The gate to advance** is a working system that meets the agreed thresholds on real data, with results confirmed by independent verification rather than by the people who produced them. ### Phase 3 — Relaunch or retire (weeks 11–13) If the verdict is recover, we put the system back into production and prove it at four levels: the intended source code, configured and deployed as designed, demonstrably running on real inputs, and producing the right outcome at the agreed quality, cost, and latency. A passing health check confirms the process is alive, not that the work is correct — so we instrument and alarm on the outcome itself, not on a liveness ping. We relaunch through evidence-based gates (limited exposure before full) and transfer ownership to your team with runbooks, because a rescue that leaves you dependent on us has not finished. **The gate to close** is an independently verified, runtime-proven system in production — and a team that owns it. If the verdict is retire, the deliverable is the documented decision and the investment redirected. ### Rescue vs. re-scope vs. retire — the engagement decision The verdict turns on a single question: at what layer is the failure? The table below is the rubric a senior team uses to assign it. | Decision | When it applies | What changes | Typical signal | |---|---|---|---| | **Recover** | Core approach is sound; the build broke | Engineering remediation under a data-integrity contract; same problem, better build | Pilot worked in testing; production data or load broke it | | **Re-scope** | The problem is real; the original target was wrong | Narrow to the slice that can be evidenced; re-run triage against the reduced scope | Pilot "works" but no one can name the P&L line it moves | | **Retire** | The data foundation or problem definition is wrong at root | Stop, document, redirect the investment | Multiple re-scopes have not converged; the data lacks the signal | In three words: recover = wrong build; re-scope = wrong target; retire = wrong foundation. ## When should you NOT rescue a pilot — when is retire the right call? Retire is the right call when the failure is architectural — when the data foundation, the business process, or the original problem definition is wrong at root, and no amount of engineering will fix it. Three signals point to it: the data simply does not contain the signal the use case requires; the underlying business process the AI was meant to serve is itself broken; or multiple re-scopes have failed to converge on anything that clears a reliability bar. Re-scope, by contrast, is for the salvageable case — the problem is real and the value is nearby, but the original target was wrong. Naming retire early is itself the highest-ROI outcome a rescue can produce. The data says it is common: Gartner predicts over 40% of agentic AI projects will be canceled by the end of 2027, citing escalating costs, unclear business value, and inadequate risk controls (Gartner press release, 25 June 2025). The brochure reads cancellation as pure loss; the practitioner read is that many of those projects will be canceled too late, after the production spend, because the kill decision was deferred. The question is not whether some get canceled — it is whether yours gets canceled in week 3 or quarter 4. There is a structural reason an outside team makes that call better. Decision science has shown for fifty years that decision-makers commit the most additional resources to a failing course of action precisely when they are personally responsible for the prior outcome (Staw, *Knee-Deep in the Big Muddy*, 1976) — and that the related sunk-cost effect is driven by the desire not to appear wasteful (Arkes & Blumer, *The Psychology of Sunk Cost*, 1985). The team that built the stalled pilot is, by the literature, least able to call it dead — not because they are weak, but because they are human and invested. An independent senior team has no prior commitment to protect. The same independence that makes verification credible makes the kill decision credible. An honest "no," delivered with evidence in weeks, is a deliverable — and it is the rarest one, because almost no one is incentivized to give it. ## Why bring in an outside senior team rather than rebuild in-house? Because the variable that predicts whether AI reaches production is whether someone owns the unglamorous production surface — evaluation, data integrity, integration, monitoring — and the original internal team, by definition, did not. MIT NANDA found that external partnerships reached deployment roughly twice as often as internal builds (approximately 67% versus 33%) among organizations achieving measurable AI ROI (MIT NANDA, *The GenAI Divide: State of AI in Business 2025*, August 2025). The naive read is "outsource it." The correct read for a rescue is narrower: bring that ownership in temporarily and senior, then transfer it back with runbooks, so the answer to "who runs this now" is your own team. The differentiation is the transfer; a rescue that leaves you dependent has not finished. AI Rescue inherits the engineering discipline we apply to AI systems where correctness is non-negotiable: absolute data integrity, proof-based deployment, and independent verification of every change. The principle is portable across domains — evidence from the running system, not a merged pull request and not a slide, is the only acceptable definition of done. ## Who this is for AI Rescue is for the CIO, CTO, or CDO — and the founder — with a pilot that has stalled, an investment already made, and a board or budget review asking what came of it. It sits downstream of strategy: keep your strategy partner; bring us in to make the thing actually run, reliably, with senior people, on a fixed scope. It is equally a clean answer when the pragmatic call is to recover what works and retire what does not. For the wider picture of how we engage, see our [AI consulting overview](/ai-consulting/) and [how we staff and deliver](/how-we-work/). We have more than fifteen years of cross-domain delivery behind this — enterprise, mid-market, SME and startup — and the same read-the-signal-early instinct that had us building on AWS while it was still in beta. We do not just describe the method; on your stalled pilot, we show it. --- *If a pilot of yours stalled and you need it in production — or a straight answer on whether it belongs there — that is exactly the gap we close. [Book a 30-minute working session](/book/) — no deck, no pitch — or read more about [how we work](/how-we-work/). Stay a step ahead, always.* ============================================================================== # Why AI Pilots Fail — and How to Get Yours to Production | NewGenApps URL: https://newgenapps.com/why-ai-pilots-fail/ ============================================================================== # Why AI Pilots Fail — and How to Get Yours to Production **In short:** most AI pilots fail in *delivery*, not in the model. They stall in the gap between a convincing demo and a system that runs in production — on real data, integrated into the workflow, with a way to prove it is right and someone accountable for keeping it right. The four causes that recur are data, evaluation, integration, and ownership. None of them is "the model isn't good enough," and all of them are fixable. If your pilot demoed well and then went quiet, you are not an outlier — you are in the majority, and the majority's reasons are now well documented. This page explains why pilots fail, why it is usually not the model's fault, and the concrete steps that move a stalled pilot into verified production. ## Why do most AI pilots fail? Most AI pilots fail because the work that makes a demo trustworthy in production was never done — the data was clean and hand-picked, there was no repeatable way to measure whether the system was right, it was never integrated into the systems and workflow that run the business, and no one owned the outcome after the applause. The model was the easy part; the production engineering around it was the part that was skipped. The scale of the problem is consistent across the firms studying it: - Roughly **95% of GenAI pilots show no measurable P&L impact** (MIT NANDA, *The GenAI Divide*, August 2025). - **42% of organizations abandoned most of their AI initiatives in 2025, up from 17% the year before** (S&P Global, *Voice of the Enterprise*, October 2025). - **88% of AI proofs-of-concept never reach widescale deployment** — for every 33 POCs a company launches, only about four graduate to production (IDC, *CIO Playbook 2025*, in partnership with Lenovo, February 2025); and **at least 30% of generative AI projects are abandoned after the proof-of-concept** by the end of 2025 (Gartner press release, July 2024). The four root causes cluster cleanly, and the order matters — data is first because it is the most-cited obstacle (tied with technology) and the cheapest gap to close: 1. **Data.** A top-cited obstacle to moving GenAI from pilot to production — named by **43% of data leaders, tied with technology** at the same figure (Informatica, *CDO Insights 2025*, n=600) — and only a small fraction of organizations report having AI-ready data. A pilot built on a cherry-picked extract meets a different reality the moment it touches the production feed: stale records, missing fields, formats that drift, and the awkward cases that never made the demo set. 2. **Evaluation.** There is no repeatable way to say whether the system is right, and no instrumentation to notice when it quietly stops being right. In our experience this is the gap that most reliably keeps a working demo out of production: without an evaluation harness and outcome monitoring, "it seemed to work in the demo" is the highest standard of proof the pilot ever met. 3. **Integration.** The pilot lives in a notebook or a sandbox, disconnected from the systems of record, the identity layer, and the workflow where the work actually happens. The last mile — from a working model to a thing a real user invokes inside a real process — was never built. 4. **Ownership.** No one is accountable for the outcome in production. The pilot was a project with an end date, not a system with an owner; when the demo ended, so did the momentum. ## Is it the model's fault? No — the model is rarely the main problem. The instinct, when a pilot underperforms, is to reach for a different or a larger model, but that is usually the wrong lever. As NTT DATA consultant Alex Potapov puts it plainly, "the model is rarely the main problem" (Dataconomy, April 2026). Capable foundation models are abundant and improving; what is scarce is the discipline to get one into production reliably and to prove it works there. This matters because it changes what you should do next. If the failure were the model, the fix would be a procurement decision. Because the failure is in delivery, the fix is method: a data contract, an evaluation harness, the missing integration, and an owner. Swapping models on a pilot that stalled for data reasons just produces a more expensive demo that still cannot ship. It is also why model lock-in is a liability rather than a comfort. Keeping models replaceable is a risk-reduction move, not a model-quality one — the production discipline around the model is where reliability is won or lost. Our engineering depth is agentic delivery — we build with Claude Code and a library of custom skills, the agent-driven workflow that lets a small senior team ship and verify production AI fast — running on infrastructure like Amazon Bedrock that keeps the model layer itself replaceable rather than locked to one vendor. That is a depth, not the answer to why pilots stall. ## What does it take to get an AI pilot into production? It takes closing the four delivery gaps and raising the definition of "done" above a convincing demo: a data-integrity contract over the real data, evaluated reliability instead of "it seems to work," the integration the pilot skipped, deployment that is verified rather than assumed, and a named owner. Production is not a bigger demo — it is a different standard of proof. The difference between a pilot and a production-grade system is concrete, and it is worth being precise about, because most pilots quietly fail every column in the table below: | Dimension | Pilot / demo | Production-grade AI | |---|---|---| | **Data** | Clean, hand-picked extract; the awkward cases removed | Real production data under a data-integrity contract — freshness guaranteed per source, "data unavailable" instead of a confident wrong answer | | **Evaluation** | None, or a one-off eyeballing of a few good outputs | Evaluated reliability on a frozen held-out set — quality, cost, latency, and the confident-but-wrong rate, measured and repeatable | | **Integration** | Runs in a notebook or sandbox, beside the real systems | Wired into the systems of record, identity, and the workflow where the work actually happens | | **Deployment** | "It merged" / "it worked in the demo" is the proof | Deploy-and-verify: source → environment → running process → real outcome, confirmed independently | | **Reliability signal** | A green health check ("the service is up") | Outcome monitoring — did the work complete, did fresh and correct output appear — because liveness is not outcome | | **Ownership** | A champion until the demo ends | A named owner, runbooks, and an evidence-based rollout (pilot → limited → full) | The pattern in that table is the whole story: a pilot is graded on a demo, and a production system is graded on evidence from the running system. Our full method for crossing that line is set out in [how we work](/how-we-work/). ## From stalled pilot to production: the steps Getting a stalled pilot to production is a sequence, not a rescue mission — diagnose the real cause, put the data under contract, prove reliability independently, then roll out on evidence. The four steps below are the spine of the recovery, deliberately front-loaded so the first answer comes in days, not weeks. 1. **Triage and diagnose.** Treat the existing pilot as evidence. Run a forensic over its code, data, evaluations (if any), integration points, and run history, and map every finding to one or more of the four root causes — data, evaluation, integration, ownership. The output is a causal diagnosis and an honest verdict: *recover*, *re-scope*, or *retire*. Not every pilot deserves a relaunch, and learning that fast is a successful outcome, not a failed one. 2. **Establish a data-integrity contract.** Re-base the system on the real data it must face, with a freshness guarantee on every source and an explicit "data unavailable" path in place of silent staleness. Because data is among the most-cited obstacles (tied with technology) and the foundation everything else depends on, it is the first thing put under contract — most stalled pilots needed exactly this and never had it. 3. **Prove reliability with independent verification.** Build an evaluation harness so quality, cost, latency, and the confident-but-wrong rate are measured rather than asserted, then have a separate, read-only check confirm the result on the running system. The engineer who builds the fix does not get the final word on whether it works; self-grading confirms the story the builder just told themselves. 4. **Roll out on evidence.** Complete the missing integration, deploy with deploy-and-verify, and promote pilot → limited → full only when the agreed criteria are met at each stage. Instrument the outcome, not liveness, and hand over with runbooks and a named owner so the system does not stall again the moment the engagement ends. This sequence is exactly the engagement we package as [AI Rescue](/ai-rescue/) — a fixed-scope, roughly ninety-day path from a stalled pilot to verified production, or to an evidence-backed decision not to carry it further. ## How do you know an AI system actually works in production? You know an AI system works in production when its behavior on real data is measured rather than assumed, an independent check confirms the result on the running system, and monitoring alarms on the outcome rather than on a ping. A demo proves a system *can* produce a good answer once; production proof shows it *does*, reliably, on the data and load it actually faces. Three signals separate proof from hope: - **Evaluated reliability, not anecdote.** There is a repeatable harness that scores quality, cost, latency, and the confident-but-wrong rate on a frozen held-out set drawn from real data — so the answer to "does it work?" is a measurement, not an impression. - **Independent verification, not self-grading.** A separate, read-only check confirms the deployed system performs as reported. Repositories describe code; production runs processes, and the two drift — a stale process, an un-applied config, a partial deploy, an expired credential. "It merged" is not "it shipped." - **Outcome monitoring, not liveness.** "The service is up" is an infrastructure signal, not a business one. Many stalled pilots had a green health check and produced nothing useful. The honest measure is whether the work completed and fresh, correct output appeared. These are the deliverables, not just principles — independent verification and evaluated reliability are things you receive and can check, which is why we treat them as buyer-facing outputs rather than internal hygiene. The mechanics are detailed in [how we work](/how-we-work/), and the offer that applies them to a stuck pilot is [AI Rescue](/ai-rescue/). ## Should you build an in-house AI team or partner to get to production? For getting a first system to production, partnering succeeds about twice as often as building in-house — and the credible path is to partner first, transfer the capability, then build. **Internal AI builds succeed roughly 33% of the time versus roughly 67% for expert partnerships — partnerships ship about twice as often** (MIT NANDA, *The GenAI Divide*, August 2025). The gap is not talent; it is that production discipline — data contracts, evaluation, deploy-and-verify — is learned by doing it under real conditions, and a stalled pilot is rarely the place to learn it for the first time under deadline. | Question | Build in-house first | Partner, then transfer | |---|---|---| | First-system success rate | ~33% (MIT NANDA, Aug 2025) | ~67% (MIT NANDA, Aug 2025) | | Time to first production result | Hiring cycle, then a learning curve under deadline | Senior team starts on the existing pilot immediately | | Where the production discipline comes from | Built from scratch, often after the first stall | Inherited from teams who run production systems daily | | Long-term ownership | The goal — but expensive as a starting point | Transferred deliberately, so your team owns what runs | The pragmatic answer for most teams with a stalled pilot is not either/or: bring in senior delivery to cross the gap and prove the system, and have that engagement transfer the capability so your team owns the running system afterward. That is the model behind [AI consulting](/ai-consulting/) and [AI Rescue](/ai-rescue/). ## Where to start If you have a pilot that demoed well and then went quiet, the fastest way to know whether it is recoverable — and what it would take — is a triage that maps the stall to its real causes. Read [how we work](/how-we-work/) for the method that crosses the demo-to-production gap, or go straight to [AI Rescue](/ai-rescue/), the fixed-scope engagement that takes a stalled pilot to verified production or tells you honestly, with evidence, that it should not go there. --- *Most pilots do not fail at the idea — they fail in the gap between a demo and a production system. [Book a 30-minute working session](/book/) to talk through where yours is stuck, or read more on [AI consulting](/ai-consulting/). For the full evidence base — the failure statistics cited to primary source and the root-cause taxonomy — see [the production-AI failure taxonomy](/production-ai-failure-taxonomy/).* ============================================================================== # The 95% AI Pilot Statistic, Explained URL: https://newgenapps.com/the-95-percent-ai-pilot-statistic-explained/ ============================================================================== **In short:** Around 95% of generative AI pilots show no measurable P&L impact, according to MIT NANDA research reported in August 2025 (*The GenAI Divide: State of AI in Business 2025*). The number describes a measurement gap, not a technology failure: most pilots are never designed to reach the operational conditions under which financial impact would appear. The distinction matters because the fixable variable is delivery discipline, not the model. --- ## What is the 95% AI pilot statistic? The statistic comes from research by the MIT Initiative on the Digital Economy (NANDA) — *The GenAI Divide: State of AI in Business 2025* — which found that approximately 95% of generative AI pilots show no measurable profit-and-loss impact. The report's exact wording: "Just 5% of integrated AI pilots are extracting millions in value, while the vast majority remain stuck with no measurable P&L impact." (MIT NANDA, *The GenAI Divide: State of AI in Business 2025*; reported widely from 18 August 2025. Authors: Aditya Challapally, Chris Pease, Ramesh Raskar. Research period: January–June 2025.) That phrasing is precise on purpose. "No measurable P&L impact" is a measurement statement, not a verdict on the technology. A pilot that ships, gets used daily, and makes individual contributors faster can still land in that 95% if no one connected it to a revenue or cost line. The report is explicit on this point: consumer-grade tools such as general-purpose AI assistants are deployed across the majority of organizations and "primarily enhance individual productivity" — and individual productivity gains do not, by themselves, register on the income statement. The scope of the study is the necessary context. MIT NANDA examined deployed, integrated enterprise AI pilots — not proofs of concept, not internal demos, not personal-productivity tools. P&L impact was the measurement criterion. The study drew on 52 structured organizational interviews, surveys of 153 senior leaders collected at four major industry conferences, and a review of more than 300 publicly disclosed AI deployments. A directionally consistent picture emerges from independent data. Gartner predicted in July 2024 that at least 30% of generative AI projects would be abandoned after proof of concept by the end of 2025; by 2026, Gartner reported that the actual rate had exceeded 50% (Gartner, *Why Half of GenAI Projects Fail: Avoid These 5 Common Mistakes*, 2026). These are different measurement windows — abandonment after a proof of concept versus P&L impact from deployed pilots — but they converge on the same structural diagnosis. --- ## Does 95% of AI fail — or 95% of pilots? The two readings are not equivalent: the statistic describes pilots that fail to register P&L impact, not AI systems that malfunction or produce wrong outputs. This is the distinction most often collapsed in derivative coverage. The version circulating across social media and syndicated commentary — "95% of AI doesn't work" or "95% of AI is a failure" — is not what the MIT NANDA report says. The report makes no claim about whether the underlying language models functioned correctly. It measures whether integrated enterprise pilots produced financial impact visible on a P&L statement. A pilot can fail the P&L test in three structurally different ways: 1. **Genuine capability failure.** The system produced wrong outputs, was unreliable at production volume, or could not handle the task it was assigned. 2. **Attribution gap.** The system worked and created diffuse productivity gains, but those gains were never instrumented — no baseline was set, no counterfactual defined — so impact could not be attributed to a P&L line. 3. **Production gap.** The pilot succeeded technically and was never taken past the demo stage into the production environment where the P&L actually moves. The misread folds all three into case one. Teams working on stalled or abandoned pilots find that case two and case three are far more common than case one — which is also why "the model is the problem" is so often the wrong diagnosis. Correcting the misread is not a defensive move. It is the most useful thing a careful reading of the report produces: much of the 95% is recoverable. If the gap is attribution or production readiness rather than model capability, the path forward looks very different from "scrap the project." --- ## What the 95% figure says — and does not say The following is the precision this number requires. | The number says | The number does not say | |---|---| | ~95% of GenAI pilots showed no measurable P&L impact in the MIT NANDA sample (reported August 2025) | That AI systems failed technically or produced wrong outputs | | P&L impact was not measurable — which may mean the pilot was never connected to a financial metric | That the underlying models are unreliable or ineffective | | Most pilots did not create the operational conditions needed for financial impact to appear | That organizations should avoid AI investment | | The failure mode is primarily a delivery and measurement problem | That the technology itself is the variable that needs fixing | *Source: MIT NANDA, The GenAI Divide: State of AI in Business 2025, reported August 2025.* The table is the argument. Everything else on this page is commentary on these four rows. --- ## Why do 95% of GenAI pilots show no P&L impact? The primary cause is not model quality — it is the absence of the operational, integration, and measurement infrastructure that converts a working prototype into a production system with a P&L line attached. The MIT NANDA report identifies a structural chasm between piloting and production. For custom enterprise AI tools, approximately 60% of organizations evaluated them, around 20% reached a formal pilot, and only about 5% reached production (MIT NANDA, 2025). Each gate narrows the field, and the skills that get a team through the first gate — building a convincing demo — are nearly disjoint from the skills the second gate requires: reliability at volume, cost-per-task economics, error handling, governance, and deep process integration. Teams optimized for demonstration keep clearing the pilot gate and stalling at the production gate. The report's central diagnosis frames this as a learning gap: "The core barrier to scaling is not infrastructure, regulation, or talent. It is learning. Most GenAI systems do not retain feedback, adapt to context, or improve over time." (MIT NANDA, *The GenAI Divide: State of AI in Business 2025*.) In engineering terms, the gap is the absence of the system around the model — memory and state management, retrieval grounded in the organization's own data, feedback capture, evaluation harnesses, and the workflow integration that lets the system accumulate context rather than start cold on every call. This is consistent with practitioner experience at scale. As NTT DATA consultant Alex Potapov put it in April 2026: "The reasons most enterprise AI projects fail to reach production are not technical but organizational: unready data, unclear ownership, and architectures that were never designed to survive past the presentation." (Dataconomy, 6 April 2026.) A useful leading indicator: if a team cannot name the specific metric the pilot was meant to move — and what that metric read before the pilot began — the measurement conditions for P&L impact were never established. A missing pre-defined success metric is upstream evidence of a mis-scoped pilot, not just a reporting gap. It means the work was framed as a technology experiment rather than a business intervention, and technology experiments do not produce P&L attribution. For the full diagnosis of why pilots stall — the root causes, the organizational patterns, and the recovery conditions — see [why AI pilots fail](/why-ai-pilots-fail/). This page's job is to explain the number accurately; that page maps the causes in detail. --- ## Is the 95% figure reliable? The MIT NANDA figure is credible as a directional indicator, but it is a single study, and its primary value is as a measurement frame rather than a universal rate. The honest accounting: the study rests on 52 structured organizational interviews and surveys of 153 senior leaders collected at four industry conferences. That is a meaningful dataset for qualitative research, but it is not a random sample of the enterprise market. Conference attendance introduces self-selection — organizations engaged enough to send senior leaders to major industry events may differ systematically from those that do not. "No measurable P&L impact" also depends on what each organization defined as P&L and how mature its measurement infrastructure was to begin with. The methodological limits have been noted publicly. Analyst Kevin Werbach, writing for Futuriom in August 2025, argued that "looking through this report with any kind of methodical analysis, you'd be hard-pressed to come up with the conclusion of the 95% failure rate" and called on MIT NANDA to release the full supporting data. That critique is worth taking seriously. The directional claim holds up despite these limits because of convergence. The MIT NANDA finding does not stand alone: Gartner's post-POC abandonment data (30% predicted, July 2024; tracked at over 50% actual by 2026), an S&P Global / 451 Research survey reporting that 42% of businesses scrapped most AI initiatives in 2025, up from 17% the prior year (published October 2025, n=1,006), and IDC/Lenovo data finding that 88% of AI proofs of concept fail to reach production (IDC / Lenovo *CIO Playbook 2026*, n=3,120) — these studies use different methodologies, sample frames, and definitions of "failure," and they point at the same structural phenomenon. When independent studies built on distinct methods agree on direction, the direction is more robust than any single number. The more productive frame is not whether the true rate is 90% or 95% — it is which conditions are fixable. The MIT NANDA figure is useful not as a settled rate but as a diagnostic prompt. The question it raises is: does the organization know which of the three failure modes its stalled pilot represents? --- ## Frequently asked questions **What is the 95% AI pilot statistic?** MIT NANDA's *The GenAI Divide* (reported August 2025) found that approximately 95% of generative AI pilots showed no measurable profit-and-loss impact. The figure measures P&L visibility from pilots, not whether the AI systems worked technically. **Does 95% of AI fail?** No. The figure describes pilots that did not produce measurable P&L impact — not AI systems that malfunctioned. The variable is delivery and measurement design, not model quality. **Why do most GenAI pilots fail to show ROI?** Because pilots are typically designed to prove technical feasibility, not financial return. The operational integration, process change, and measurement infrastructure needed for P&L impact are usually absent at the pilot stage. Gartner's data-quality research (February 2025) adds that 63% of organizations either do not have, or are unsure they have, the right data management practices for AI — a foundational gap the model alone cannot close. **Is the MIT 95% AI statistic accurate?** It is credible as a directional figure and consistent with independent data from Gartner, S&P Global, and IDC. It is a single preliminary study, and "no measurable P&L impact" depends on how each organization defined and measured P&L. Its value is as a delivery-diagnosis prompt, not a settled rate. **What should you do if your AI pilot is in that 95%?** Determine first whether the gap is technical, operational, or structural. Most pilots in the 95% are technically recoverable; the missing ingredient is usually delivery discipline — integration depth, measurement design, and verified deployment. See [how we work](/how-we-work/) or [AI Rescue](/ai-rescue/) for the recovery path. --- The practical reading of the 95% is the optimistic one: most of it is a delivery gap, and delivery gaps are fixable. The standard that closes them is production AI, proven — a system put under a data-integrity contract, measured on real data, and confirmed to work by an independent check rather than by a demo. See [how we work](/how-we-work/) or [AI Rescue](/ai-rescue/) for the recovery path. --- *NewGenApps helps organizations move from pilot to production. Stay a step ahead, always.* ============================================================================== # Has Your AI Pilot Stalled? A 5-Signal Diagnosis URL: https://newgenapps.com/has-your-ai-pilot-stalled-5-signal-diagnosis/ ============================================================================== **In short:** Most AI pilots stall silently. The five observable signals are: (1) no evaluation harness beyond the demo, (2) the dataset is curated rather than production-representative, (3) no named internal owner with delivery authority, (4) "it works in the demo" is the highest bar it has ever met, and (5) the pilot has sat in "promising" status for a quarter or longer. If three or more are true for your project, the pilot is statistically unlikely to reach production without an intervention. S&P Global Market Intelligence's *Voice of the Enterprise: AI & Machine Learning, Use Cases 2025* (survey fielded Oct–Nov 2024, published October 2025, n=1,006 IT and line-of-business professionals across North America and Europe) found that 42% of organisations abandoned most of their AI initiatives in 2025, up from 17% the year before. Stalling is the norm. Diagnosing it early is the differentiator. --- ## What does a stalled AI pilot look like? A stalled AI pilot is one that has demonstrated value in a controlled setting but has not moved measurably closer to production in 60 or more days. The distinction between "stalled" and "failed" matters operationally. A failed pilot has been assessed against production criteria and found unable to meet them — a decision was made. A stalled pilot has never been assessed against production criteria at all. It persists in a pre-gate state: showing well, generating stakeholder confidence, clearing no new bars. The team keeps proving itself to the same internal audience. The evaluation is the demo. The demo is the output. Nothing accumulates toward production. This is more common than most teams admit. The S&P Global data cited above is instructive not just for its scale but for its direction: the abandonment rate nearly tripled in a single year, from 17% to 42%. The companion figure from the same study — that organisations scrapped an average of 46% of proof-of-concept projects before broad adoption — places the discontinued pilot in the majority rather than the margin. The tone to adopt here is diagnostic, not catastrophic. If your pilot is stalled, you are in the majority. The question is whether you name it early enough to act. *(For the root causes that drive pilots into this position in the first place, see [why AI pilots fail to reach production](/why-ai-pilots-fail/) — that piece covers the upstream design and scoping errors; this one is the mirror you hold up to a project already underway.)* --- ## How long is too long for a pilot to stay "promising"? In practice, a pilot that remains in "promising" status beyond 90 days without clearing a single production-readiness criterion has entered stall territory. The 90-day mark is a diagnostic threshold, not a rule. Calendar time is a proxy for the more meaningful question: has any production criterion — security review, latency benchmark, data governance sign-off, integration test against live systems — been cleared in the last 30 days? If the answer is no, the pilot is not progressing. It is being maintained. No verified primary source gives a clean "average pilot dwell time" that meets the evidence standards used throughout this piece, so the time signal is kept qualitative. What the data does establish clearly is the conversion gap. IDC research conducted with Lenovo (*CIO Playbook 2025: It's Time for AI-nomics*, February 2025, surveying 900 IT and business decision-makers across Asia-Pacific mid-to-large organisations) found that for every 33 AI proofs-of-concept a company launched, roughly four reached production. The accumulated remainder is the set of pilots this piece is about: alive on the books, going nowhere in fact. Deloitte's *State of AI in the Enterprise 2026* (survey fielded August–September 2025, n=3,235 business and IT leaders across 24 countries) found that only 25% of organisations have moved 40% or more of their AI experiments into production. Gartner forecast in July 2024 that at least 30% of generative AI projects would be abandoned after proof of concept by end of 2025, citing poor data quality, inadequate risk controls, escalating costs, and unclear business value — supporting the conversion gap without serving as the primary evidence here. The organisational dynamic is worth naming. The longer a pilot stays in "promising," the harder it becomes to cancel or restructure — sunk-time reasoning calcifies around it. Teams describe it in retrospect as having been "almost ready for months." The stall does not announce itself. It accumulates. --- ## The 5 signals your AI pilot won't reach production These five signals are observable before a pilot runs out of budget or goodwill — and three or more in combination is a reliable indicator that production is not the current trajectory. A note on structure: these signals are not independent. They form a causal chain. No exit criterion (Signal 5) is definable because there is no evaluation harness (Signal 1). No harness means quality is judged by the demo (Signal 4), which runs on curated inputs (Signal 2), because no one owns the question of whether it holds up on real traffic (Signal 3). The root is a single missing artifact: a measurable definition of "working" that runs on representative data and is owned by someone. Each signal is that absence viewed from a different angle. --- ### Signal 1 — No evaluation harness beyond the demo **Definition:** The only measurement of the system's performance is whether stakeholders liked a presentation. **Underlying cause:** Success criteria were defined for the demonstration, not for the production environment. Quality lives in someone's head, not in a scoring function. **Why it matters technically:** Without an evaluation harness — an offline test set with known-correct answers, a scoring function, and a regression gate — you cannot detect deterioration. The moment you change a prompt, a model version, a retrieval source, or a preprocessing step, you have no way to know whether you made the system worse. Production systems are not static; model deprecations alone force changes. A system that cannot measure whether a change helped is one that can only drift. Google's *ML Test Score* rubric (Breck, Cai, Nielsen, Salib, and Sculley, *The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction*, IEEE Big Data, 2017) defines production readiness through the presence of 28 specific tests and monitoring checks across data, model, infrastructure, and monitoring domains. "It looks good" scores zero on that rubric. The eval harness is not a QA nicety added at the end of development. It is the instrument that makes iteration possible. Teams without one do not iterate slowly — they cannot iterate at all. Which is precisely why they sit in "promising" for quarters. **Observable check:** Ask the team: "What is your current score on your evaluation set, and what was it a month ago?" If there is no number, or the number is not written down anywhere, you have your answer — independent of how impressive the demo is. --- ### Signal 2 — The dataset is curated, not production-representative **Definition:** The data the pilot runs on was selected or cleaned for the purpose of the demonstration; the same examples appear in every run. **Underlying cause:** The system was tuned, implicitly or explicitly, on the very inputs it is shown succeeding on. This is leakage. The evaluation no longer measures generalisation — it measures the curator's taste. **Why it matters technically:** Real production traffic is a different distribution from any curated sample — longer inputs, messier formatting, adversarial edge cases, the entries the curator quietly removed. A model can perform excellently on a chosen set and fail systematically on live traffic. Paleyes, Urma, and Lawrence's survey of machine learning deployment case studies (*Challenges in Deploying Machine Learning: A Survey of Case Studies*, ACM Computing Surveys, 2022) identifies distribution mismatch between development data and production data as one of the recurring, named failure modes across real deployments. The practical implication: performance on a hand-picked input set is not weak evidence of production-readiness. On a curated set it is no evidence at all, because curation is exactly what hides the failure modes. Curated golden sets are correct and necessary as one evaluation layer — the signal here is when the curated set is the *only* evaluation and doubles as the demo input. **Observable check:** Ask to run the system, live, on an input the questioner supplies on the spot — something not previously shown to the team. Watch whether the team flinches before running it. The flinch is the diagnosis. --- ### Signal 3 — No named internal owner with delivery authority **Definition:** There is enthusiasm and sponsorship, but no individual with budget authority, a mandate to ship, and accountability for a go-live date. **Underlying cause:** The pilot was built by technical champions who lack the organisational authority to clear cross-functional blockers — legal sign-off, security review, procurement, integration with downstream systems. Sponsorship and ownership are different things, and the pilot has one but not the other. **Why it matters technically:** Machine learning systems decay without intervention even when no code changes, because the world around them changes. The research literature on deployed systems formalises this as concept drift — when the statistical relationship between inputs and the target shifts over time, deployed models silently lose accuracy as their environment diverges from training conditions (Gama, Žliobaitė, Bifet, Pechenizkiy, and Bouchachia, *A Survey on Concept Drift Adaptation*, ACM Computing Surveys, 2014). A system with no owner has no one to notice the decay, no one to trigger retraining, no one to be alerted when it breaks. It is structurally abandoned on arrival. Sculley et al. (*Hidden Technical Debt in Machine Learning Systems*, NeurIPS, 2015) found that only a small fraction of a real-world ML system is composed of the ML code itself — the surrounding infrastructure of data collection, verification, configuration, monitoring, and serving dwarfs it, and systems acquire undeclared consumers and unmonitored dependencies precisely because no one owns the boundaries. Ownership is not an org-chart concern layered on top of technical work. It is a reliability condition built into the technical work. **Observable check:** Ask: "Who gets paged when the system returns a wrong answer in production, and how would they know it had?" If the answer is silence, or "we would hear about it from a user," there is no owner. --- ### Signal 4 — "It works in the demo" is the highest bar the pilot has ever cleared **Definition:** Every conversation about the pilot is a re-run of the same demonstration. No harder question has been asked, and no means to answer one has been built. **Underlying cause:** Signals 1 and 2 have compounded into a culture. The team has never been asked — and never built the instrumentation — to answer any harder question than "can it produce one good output while watched." The demo bar and the production bar are separated by everything that does not appear in a demo: integration with live systems, latency and cost under load, error handling for edge cases, failure-mode behaviour, monitoring, rollback capability, and verification that outputs are correct rather than merely plausible. Plausible-but-wrong outputs are the single most dangerous failure class in generative systems. A demo specifically selects against ever surfacing one. **What practitioners recognise but brochures miss:** the polish of a demo is negatively correlated with production-readiness in stalled pilots, because polish is where the remaining discretionary effort went. A compelling demo with no harness, no live-data test, and no owner is not "almost there." It is at the beginning of the production-readiness work, having spent its credibility appearing to be at the end. **Observable check:** List the production requirements the system must clear before deployment — latency under SLA, security review, integration tests, error-handling coverage, rollback plan. Score the pilot against each. The number of cleared items is the honest progress measure. If the list has never been written, that is Signal 4 confirmed. --- ### Signal 5 — The pilot has been "promising" for a quarter or longer **Definition:** More than 90 days have passed without a single production-readiness criterion being cleared. **Underlying cause:** "Promising" with no improving metric is the natural resting state of a pilot with no evaluation harness. There is nothing to push it forward, and nothing to declare it done, so it stays. The IDC / Lenovo 2025 research cited above quantifies the structural version of this: for every 33 AI proofs-of-concept launched, roughly four reach production. The remainder do not fail on a specific criterion — they simply never define one. Time-in-pilot is not a neutral cost. It is evidence of the absence of an exit criterion. A healthy pilot has a written, measurable "done" — a target score on the evaluation set, a latency and cost budget, a go/no-go gate. The reason a pilot can remain "promising" for a quarter is that no such number was ever defined. Duration is the symptom; the missing exit criterion is the disease. **Observable check:** What was the last hard decision made about this pilot, and when? "We decided to continue" is not a hard decision. "We set a 30-day window to clear the security review or restructure scope" is. If no decision date surfaces, the pilot has no clock running. --- ## Signal → cause → next step: a quick-reference table Each signal points to a specific underlying cause; knowing the cause determines whether the pilot is recoverable and what intervention it needs. | Signal | Root cause | Recoverable? | Fastest first step | |--------|-----------|-------------|-------------------| | No evaluation harness | Success criteria defined for demo, not production | Yes, if addressed within current sprint | Define three automated, production-representative test cases this week | | Curated dataset only | Production data complexity was not scoped in | Yes, with a data audit | Pull 200 unfiltered records from the live system and run the model against them | | No named owner | Sponsorship was treated as equivalent to ownership | Yes, with an explicit mandate | Assign a named owner with a go-live date in writing before the next sprint | | "Works in demo" only | Production requirements were never formalised | Yes, with a requirements gate | List all production criteria; score the pilot against each one publicly | | 90+ days "promising" | No escalation trigger or exit criterion exists | Conditionally | Set a 30-day decision checkpoint: proceed with the current scope, restructure, or close | Recoverability is conditional on the fundamentals being sound. A pilot stalled on Signal 3 — no named owner — is often unblocked in days with a single organisational decision. A pilot stalled on Signal 2 with severely unrepresentative training data may require a scope redesign that is functionally a restart. The triage purpose is to distinguish between those two cases before additional cycles are spent. --- ## My pilot is stalled — what do I do next? The first move is a structured triage: not a new vendor evaluation, not a rebuild, but a focused assessment of whether the pilot's fundamentals are sound enough to be worth recovering. A triage has three possible outcomes. First, the pilot is recoverable with the current team and scope, with a clear remediation plan and a production timeline that can now be committed to. Second, the pilot is recoverable but requires an external senior review to unblock a specific gap — an evaluation harness that has not been built, a data audit that requires access the internal team does not have, or an ownership decision that requires authority above the project level. Third, the pilot is not recoverable in its current form — the original business problem has materially changed, or the scoping assumptions were wrong at the foundation — and faster progress comes from a redesigned brief than from repairing what exists. What a triage is not: a sales conversation dressed as a diagnosis. Its function is to give the leadership team a clear verdict, stated plainly, before they spend more budget and more goodwill on a project that may not be salvageable. The verdict should be the same whether or not the diagnosing party has a commercial interest in the outcome. That independence is the point: the same discipline that separates a working production system from a convincing demo — measure it, run it on real inputs, verify the result rather than assert it — is what separates an honest triage from a pitch. The counter-evidence is worth stating here: stalling is not permanent, and the majority of organisations expect to improve. Deloitte's 2026 survey found that while only 25% of organisations had moved 40% or more of their AI experiments to production at the time of surveying, 54% expected to reach that level within three to six months. BCG's *The Widening AI Value Gap* (September 2025) found that AI leaders — the top 20% of organisations — achieve five times the revenue increases and three times the cost reductions of laggards. The structural gap between stalled and producing is real, but it is not a one-way door. The organisations that cross it are the ones that stop managing the stall and start measuring against a definition of "in production, and proven to work." If three or more of the five signals are true for your project, a triage will tell you whether it is recoverable. [AI Rescue](/ai-rescue/) starts with exactly that diagnosis. Or [book a 30-minute session](/book/) to discuss where your pilot currently stands. If you are also weighing whether to bring in an external team to do that assessment, [how to choose an AI partner](/how-to-choose-an-ai-partner/) covers what to look for in a credible triage process and how to vet the firm conducting it. --- ## Common questions **What percentage of AI pilots fail to reach production?** S&P Global Market Intelligence's *Voice of the Enterprise: AI & Machine Learning, Use Cases 2025* (published October 2025, n=1,006 IT and line-of-business professionals across North America and Europe) reported that 42% of organisations abandoned most of their AI initiatives in 2025, up from 17% the year prior. The same study found organisations scrapped an average of 46% of proof-of-concept projects before broad adoption. While "stalled" and "abandoned" are not identical, the data suggests that the majority of pilots that do not reach production are discontinued rather than formally assessed and recovered. IDC's 2025 enterprise research with Lenovo (*CIO Playbook 2025*, February 2025) puts the production conversion rate in structural terms: for every 33 AI proofs-of-concept launched, roughly four reach production. **Is a stalled AI pilot always worth trying to recover?** Not always. The triage question is whether the fundamental scoping and data assumptions are sound enough to build on. A pilot stalled on Signal 3 — no named owner — is often recoverable in days with a single organisational decision that was never made. A pilot stalled on Signal 2 — a dataset that does not represent production — may require a scope redesign that is functionally a restart. The signals are not uniformly fixable; the honest answer after a triage is sometimes "redesign with a clearer brief" rather than "continue with remediation." **How is a stalled AI pilot different from one that simply failed?** A failed pilot has been assessed against production criteria and found unable to meet them — a decision was made, even if the outcome was disappointing. A stalled pilot has never been assessed against production criteria at all. It persists in a pre-gate state indefinitely, still described as "promising," generating cycles without generating evidence. Stalling is more common than failure in the technical sense and more recoverable — but it requires an active intervention to break the loop, because nothing in the current structure will do so automatically. **Should I bring in an external team to diagnose a stalled pilot?** External review is most useful when the block is structural rather than purely technical — no named owner, no production requirements ever formalised, no clear budget path to deployment. An internal team close to the work often lacks the authority or the detachment to make the "stop, restructure, or continue" call. The people who built the demo have an understandable interest in it continuing. An independent triage preserves the authority to make that call cleanly, and produces a verdict the leadership team can act on without relitigating it. **At what point should I cancel an AI pilot rather than triage it?** If all five signals are present and the original business problem has materially changed since the pilot was scoped, cancellation with a redesigned brief is usually faster than attempting recovery. The 90-day "promising" signal (Signal 5) is the most common trigger for this conversation — not because duration is the deciding factor, but because duration is the visible symptom of the missing exit criterion, and a pilot that has never defined "done" has no natural point at which the question of cancellation becomes unavoidable. Setting one is the intervention. --- *NewGenApps — Stay a step ahead, always.* ============================================================================== # Pilot vs. Production AI: What Actually Changes URL: https://newgenapps.com/pilot-vs-production-ai-what-changes/ ============================================================================== # Pilot vs. production AI: what actually changes > **In short.** An AI pilot is graded on a demo; a production AI system is graded on evidence from the running system. The gap between them is not a matter of scale — it is a different standard of proof: curated data vs. real data, scripted scenarios vs. edge cases, approval from a room vs. accountability to an SLA. According to IDC, in partnership with Lenovo, 88% of AI proofs-of-concept do not reach widescale deployment — for every 33 POCs launched, roughly four graduate to production (*IDC / Lenovo, CIO Playbook 2025*, Feb 2025; reported Mar 2025). The table below maps what changes across eight dimensions. --- ## What is the difference between an AI pilot and a production AI system? A pilot demonstrates that a capability is possible under controlled conditions; a production system proves that it works reliably under real conditions, with real data, for real users, over time. **An AI pilot** is a bounded exercise designed to test feasibility. It runs on selected inputs, answers to a project team, and ends with a decision gate — typically a demo. Its standard of evidence is the demo itself: if the outputs look good to the people in the room, the pilot has done its job. **A production AI system** is an operating artifact. It must handle inputs nobody chose, run without a supervisor watching each output, survive updates and model deprecations, connect to live data sources and downstream processes, and surface its own health through metrics — because if it cannot, no one will notice when it starts to decay. The confusion between the two is understandable: a well-run demo looks convincing. But a demo is an *existence proof* — evidence that there exists at least one input on which the system works. Production demands something close to a *universal claim* — that for inputs the system does not get to choose, it works often enough, and that claim can be supported with evidence from the running system. An existence proof tells you almost nothing about the universal claim. That is not a flaw in the demo; it is what demos are designed to produce. The category error is treating one kind of evidence as though it were the other. The eight-dimension comparison table below makes this concrete. --- ## Why does an AI demo that works in the lab fail in production? A demo is built to pass, not to run — the data is curated, the scenarios are scripted, and the definition of "done" is approval in the room, not a service-level agreement that holds under load. Four structural mechanisms drive the gap, each absent from the demonstration by design. **1. Curated data vs. live data.** In a pilot, someone — consciously or not — selects inputs that play to the model's strengths. Messy, ambiguous, and adversarial cases are quietly removed. In production, data arrives dirty, out-of-distribution, and longer than the training distribution expected. The formal name for this is *dataset shift*: the joint distribution of inputs and outputs differs between development and deployment (Quiñonero-Candela, Sugiyama, Schwaighofer & Lawrence, *Dataset Shift in Machine Learning*, MIT Press, 2009). Treating strong demo performance on a curated set as evidence of production-readiness is the common error. It is not weak evidence — it is no evidence, because the curation removed exactly the inputs that would falsify the claim. **2. Scripted scenarios vs. edge cases.** A demo covers the happy path — the sequence of events that makes the system look like the intended use case. Production is mostly edge cases: partial inputs, malformed requests, upstream timeouts, and the long tail of things nobody thought to test. Paleyes, Urma & Lawrence, surveying real ML deployments (*Challenges in Deploying Machine Learning: A Survey of Case Studies*, ACM Computing Surveys, 2022), document distribution mismatch between development and production data as a recurring, named failure mode — not bad luck. **3. No evaluation harness.** A pilot's quality check is human approval of the demo output. A production system needs an automated regression harness — a set of tests that run on every deployment and catch quality drift before users encounter it. Without one, a change to a prompt, a model version, or a retrieval source can make the system worse and nobody will know until the complaints arrive. This is especially sharp for generative systems, where the obvious substitute (using a language model to grade the outputs, known as LLM-as-judge) is feasible but not free: strong judges reach above 80% agreement with human preferences, comparable to inter-human agreement — but they exhibit documented biases. Position bias (favoring the first or second answer by order), verbosity bias (preferring longer outputs regardless of correctness), and self-enhancement bias (a model favoring its own family's outputs) are systematically reproduced in the research (Zheng et al., *Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena*, NeurIPS 2023; position bias systematized in Shi et al., 2024). An uncalibrated judge is not a harness — it is a mirror. **4. No integration, no accountability.** Demos run in isolation. Production systems connect to upstream data sources and downstream processes — auth systems, rate limits, data contracts, dependent services — and each connection is a new failure mode and a new latency budget. Sculley et al. (*Hidden Technical Debt in Machine Learning Systems*, NeurIPS 2015) named the relevant anti-patterns: undeclared consumers, unstable data dependencies, and boundary erosion. A standalone demo cannot surface any of them: there are no boundaries to erode when the system talks to nothing. For a structured breakdown of why pilots stall organizationally, see [Why AI pilots fail](/why-ai-pilots-fail/). --- ## What does "production-ready AI" actually require? Production-ready AI requires verified performance on real data, an automated evaluation harness, integration into live systems, defined reliability signals, a clear ownership model, and a cost basis built around ongoing operation — not one-time build. ### Pilot vs. production AI: what changes across eight dimensions | Dimension | AI pilot (graded on a demo) | Production AI system (graded on evidence) | What the demo conceals (the mechanism) | |---|---|---|---| | **Data** | Curated sample: hand-picked, clean, adversarial cases removed | Live, dirty, out-of-distribution traffic — inputs nobody chose | The curated set is the tuning set. Success on it measures the curator's taste, not the system's generalization. | | **Definition of "done"** | "Looks good in the demo" — no written, measurable target | A pre-committed exit criterion: a score on a held-out eval set, a latency/cost budget, a go/no-go gate | A demo has no failing grade. Without a number, a pilot can be "promising" indefinitely — motion without direction. | | **Evaluation** | Eyeballing outputs; quality lives in someone's head | An eval harness: held-out set, scoring function, regression gate on every deployment | A demo shows one good output; it cannot show the distribution of outputs or whether last week's change regressed quality. | | **Integration** | Standalone and sandboxed; talks to nothing; happy-path only | Wired into live systems — auth, rate limits, upstream/downstream data contracts; degrades gracefully when a dependency fails | Demos run in sealed sandboxes. Integration failures, schema drift, and timeouts are precisely what the sandbox excludes. | | **Reliability signal** | "It worked the times we ran it" (small, chosen n) | Measured success rate on unchosen inputs; tail behavior characterized — p95/p99, not just the median | Small chosen n hides variance. The dangerous output is the plausible-but-wrong one; a demo selects against ever showing it. | | **Deployment and change** | Frozen at the moment it demoed well | Versioned, monitored, rollback-capable; survives model deprecations, prompt changes, retrieval changes | A demo is a photograph of one instant. Every model/prompt/data change is a chance to silently regress — invisible without measurement. | | **Ownership** | "The data-science team built it" — owner of the build, not the correctness on Tuesday | A named owner of ongoing correctness; someone paged when it returns a wrong answer, with a way to know | A demo needs a presenter, not an owner. The human half of the monitoring loop is invisible until the system is live and decaying. | | **Cost basis** | Marginal cost near zero; tokens/compute unmetered for one run | Unit economics under load: cost per successful task at volume, including retries, eval calls, and human review | A demo is one call. Cost, rate limits, and the price of retries and guardrails only appear at volume, off the demonstration path. | Each row above represents a ratchet. Once a system operates at the production standard in a given dimension, reverting to the pilot standard constitutes a regression, not an iteration. The most common failure pattern is treating the pilot's "done" as sufficient and moving to deployment without rebuilding the standard of proof in each dimension. What makes this table more than a checklist is that every row is the same shift seen from a different angle: the unit of evidence changes from a *watched single run* to an *unwatched measured rate*. Data, evaluation, reliability, deployment, ownership, cost — each is a question about what it takes to produce measured evidence on that axis, rather than a watched anecdote. Sculley et al. framed the underlying principle in 2015: "only a small fraction of real-world ML systems is composed of the ML code" — data collection, verification, serving infrastructure, configuration, and monitoring dwarf it (*Hidden Technical Debt in Machine Learning Systems*, NeurIPS 2015). A pilot is, almost by definition, the small fraction built and the large fraction skipped. The comparison table is a tour of the large fraction. The deployment dimension of this table is covered in depth at [Deploy and verify AI](/deploy-and-verify-ai/). --- ## How long does it take to move from an AI pilot to a production system? The honest answer depends more on what was built during the pilot than on calendar time: a pilot that produced a working evaluation harness and a real-data baseline can reach production in weeks; one that produced only a demo typically requires rebuilding the foundation before deployment begins. The range is real, and the variables that drive it are specific: does an evaluation harness exist? How complex is the integration surface? How much data quality work is required before the system can be reliably tested on live traffic? Has an operational owner been named? Each absent answer adds to the timeline not because AI is slow to build but because the large fraction — the right-hand column of the table above — takes time to build correctly. This converts "how long?" into a better question: *what gates remain?* A pilot that demoed last week and a pilot that demoed last year are equidistant from production if neither has started the large fraction. Anthropic's *Building Effective Agents* (December 2024) draws the line practitioners feel directly: a single, well-scoped agent is a weeks-class effort; multi-agent systems take months to get right, with explicit guidance to start with the simplest thing that works and add complexity only when it demonstrably improves outcomes. The architecture choice is a timeline variable, not a fixed constant. Three conditions accelerate the transition, in rough order of impact: an evaluation harness built during the pilot (not after); real-data testing started before the demo, not after; and a named production owner assigned before build, not once the pilot has been approved. These are not process preferences — they are what determine whether the pilot produces a working system or a well-received presentation. --- ## What is the 88% pilot failure rate and where does it come from? Per IDC, in partnership with Lenovo, 88% of AI proofs-of-concept do not reach widescale deployment — for every 33 POCs launched, roughly four graduate to production (*CIO Playbook 2025*, IDC / Lenovo, February 2025; reported by CIO.com, 25 March 2025). This figure is this page's starting point because it quantifies the size of the pilot-to-production chasm, not just its existence. What the figure measures matters. It tracks the transition from proof-of-concept to widescale deployment — a specific, defined threshold, not a subjective assessment of whether the AI "worked." The respondents are IT and business decision-makers across global cohorts; the top reasons cited for stalling include unclear ROI, insufficient AI-ready data, lack of in-house expertise, and low organizational readiness. The same research house found that by 2026, approximately 46% of POCs had progressed to production (IDC / Lenovo, *CIO Playbook 2026*, January 2026) — a substantial improvement from 12% in 2025, which illustrates both how fast enterprise AI maturity is advancing and that a majority still stall. What this stat does and does not mean: it measures the transition rate, not the quality of the technology. AI proofs-of-concept do not fail at 88% because the models are inadequate. They stall because the demo's standard of proof was never upgraded to production's. The gap is a delivery and standards problem. A polished demo with no eval harness, no live-data test, and no named operational owner is not "almost done" — it is at the start of the work, with the budget already spent on the part that shows. Gartner's 2024 prediction that at least 30% of generative AI projects would be abandoned after POC proved conservative — the S&P Global / 451 Research *Voice of the Enterprise* survey (October 2025, 1,006 North American and European IT professionals) found 42% of companies had abandoned most of their AI initiatives in 2025, up from 17% in 2024. The trajectory of abandonment has moved faster than the trajectory of institutional prediction. This is not evidence that AI doesn't work; it is evidence that the pilot-to-production gap is a structural problem, not a temporary one, and that the conditions for crossing it need to be built in — not assumed. --- ## Frequently asked questions **Is an AI pilot the same as an MVP?** No. An MVP is designed to be deployed and operated; a pilot is designed to test feasibility. An MVP has a user, an operator, and a feedback loop. A pilot has an audience and a deadline. The confusion between the two is one of the most common reasons pilots do not reach production: teams scope the pilot to answer "can it?" and then discover that "does it, for real users, reliably?" requires rebuilding most of what the pilot produced. **What is the most common reason an AI pilot fails to reach production?** The most common reason is that the pilot was scoped to produce a demo, not to produce the artifacts a production system requires — an evaluation harness, a real-data baseline, and an integration plan. When those are absent, reaching production means rebuilding from the pilot's output rather than extending it. The eval harness is usually the first missing piece that becomes visible: without it, a team has no way to know whether changes improve or regress the system, so iteration stalls. **Does production-ready AI require a specific technology stack?** No. Production-readiness is a property of the system's operating standards — evaluation, reliability signals, ownership, and integration — not of the underlying technology. The same model that runs in a demo can run in production if the surrounding system is built to production standards. Stack choices affect performance and cost at the margin; the presence or absence of the eight dimensions in the table above determines whether the system qualifies as production at all. **What is an "evaluation harness" and why does it matter for production AI?** An evaluation harness is an automated set of tests that measure model output quality against a defined baseline, run on every deployment. It matters because model behavior can drift as data, prompts, or underlying models change — without a harness, quality regression is invisible until it reaches users. Google's ML Test Score rubric (Breck, Cai, Nielsen, Salib & Sculley, IEEE Big Data 2017) operationalizes production-readiness as the presence of 28 specific tests across data, model, infrastructure, and monitoring. A demo passes none of them. **How is the pilot-to-production gap different for agentic AI systems?** The gap is wider for agentic systems because agents act, not just respond — a misfire in an agent pipeline can propagate downstream before a human sees it. Agents also compound the reliability problem: whereas a single inference endpoint either returns a correct result or doesn't, an agent chain multiplies the failure surface across each step. This is consistent with IDC / Lenovo's finding that 88% of AI POCs (including agent pilots) do not reach widescale deployment, and with Gartner's June 2025 prediction that over 40% of agentic AI projects will be canceled by end of 2027 due to escalating costs, unclear business value, or inadequate risk controls. The reliability bar for agentic systems is higher than for a simple inference endpoint, and the evaluation problem — measuring whether an agent that behaved correctly in a demo will behave correctly on inputs nobody chose — is structurally harder. --- ## What this means in practice Production is not a bigger demo — it is a different standard of proof. The eight dimensions in the table above are not a list to address after launch; they are the minimum that separates "it worked in the demo" from "it works on its own, on inputs nobody picked, and we can show it." That last clause is the whole standard: production AI, proven — performance measured on real data and confirmed by an independent check, not asserted from a presentation. Crossing this gap is what [how we work](/how-we-work/) is built for. If your pilot is already stuck on the right-hand column of the table above, [AI Rescue](/ai-rescue/) is the fixed-scope path across it. --- *NewGenApps. Stay a step ahead, always.* ============================================================================== # AI in the Enterprise: What Actually Reaches Production URL: https://newgenapps.com/ai-in-the-enterprise/ ============================================================================== # AI in the Enterprise: What Actually Reaches Production --- **In short.** AI in the enterprise is the deployment of artificial intelligence systems inside an organisation's live operating environment—integrated with real data, real workflows, and real accountability for performance. The defining problem in 2025–2026 is not that enterprise AI lacks capable models. It is that most of it never reaches meaningful production. IDC's analysis found that for every 33 AI proofs-of-concept an organisation launches, only about four reach widescale production—roughly 88% stall before they deliver operating value (Ashish Nadkarni, IDC, *CIO Playbook 2025*; reported CIO, March 2025). The gap is overwhelmingly a delivery problem: data readiness, evaluation and reliability, integration into existing systems, and clear ownership of the model once it is live. Not model capability. What follows is the evidence for that claim, the four dimensions that separate enterprise AI that ships from pilots that stall, and a practical adoption sequence. --- AI in the enterprise is the use of artificial intelligence—increasingly large-language-model and agentic systems—inside the core operations of a large organisation, where it has to run reliably against real data, real users, and real regulatory and integration constraints. The distinguishing feature is not the model; consumer and enterprise tools often use the same ones. It is the operating environment. Enterprise AI must be integrated into existing systems, governed, monitored, and trusted by the people whose work depends on its output. That is why so much enterprise AI works in a demo and stalls before production: the hard part is delivery into that environment, not access to the model. --- ## What is AI in the enterprise? AI in the enterprise is the deployment of machine learning and large-language-model systems inside an organisation to automate decisions, augment knowledge work, or generate business outputs at scale—distinct from a pilot or proof-of-concept in that it runs in a live operating environment, integrates with existing data and workflows, and is owned by a team accountable for its performance. Three distinctions are worth making clearly. First, enterprise AI is not consumer AI adapted for the office; it operates under procurement, compliance, data-governance, and integration constraints that consumer tools do not face. Second, enterprise AI is not enterprise software that embeds AI as a feature—it is the deployment and operation of the AI layer itself, with all the evaluation and reliability work that entails. Third, and most importantly for this discussion: a pilot proves capability; enterprise AI proves delivery. A pilot can succeed completely—returning accurate outputs on curated data in a controlled environment—and still represent zero delivered value if it never crosses into production. As of 2025–2026, enterprise AI budgets are concentrated in a relatively small number of use case categories: document processing and extraction, internal search and knowledge retrieval, code assistance, and customer-facing conversational interfaces. Gartner forecasts worldwide AI spending of approximately $2.5 trillion in 2026 across the full stack—hardware, software, services, platforms, and application development (Gartner, 15 January 2026), and has identified 2026 as an inflection year for enterprise AI spend specifically. That scale of investment makes the production-gap question urgent: enterprise budgets are scaling into a delivery problem that is not yet solved. --- ## Why does so little enterprise AI reach meaningful production? Because production, not the model, is the bottleneck. IDC's analysis found that for every 33 AI proofs-of-concept an organisation launches, only about four reach production—roughly 88% do not reach widescale deployment. The cause cited by IDC is "the low level of organizational readiness in terms of data, processes and IT infrastructure" (Ashish Nadkarni, Group Vice President, IDC, *CIO Playbook 2025*, Lenovo-commissioned; reported CIO, 25 March 2025). Not model quality. A separate line of evidence points in the same direction at the skill level. A study of approximately 1.4 million real workplace AI interactions from roughly 2,597 employees over eight months—conducted by KPMG LLP and the University of Texas at Austin McCombs School of Business and published in *Harvard Business Review* (19 March 2026)—found that only around 5% of employees consistently demonstrated the sophisticated, iterative patterns of AI use that convert AI access into real output: treating the model as a reasoning partner, decomposing tasks, specifying structure and reasoning requirements. Access to AI is now near-universal. The delivery skill that converts access into outcomes is concentrated in a small fraction of the workforce. These two findings are about different levels of the stack—organisational readiness and individual skill—but they point at the same scarcity: not capability, but the discipline to deploy it. The delivery gap decomposes into four consistent causes: **Data quality and readiness.** A pilot runs on a curated slice prepared for the demo. Production runs on the organisation's real data—incomplete, contradictory, access-controlled, and changing. Informatica's 2025 survey of 600 chief data officers found that 43% cite data quality and readiness as the primary obstacle preventing GenAI pilots from reaching production, with 97% reporting difficulty demonstrating GenAI business value (Informatica *CDO Insights 2025*, January 2025).\* The model works; the input data does not. Gartner's July 2024 prediction that at least 30% of GenAI projects would be abandoned after proof of concept named poor data quality as the primary cause (Gartner, 29 July 2024). **Evaluation and reliability gaps.** Enterprise AI is probabilistic: the same input can produce different outputs, "correct" is a distribution, and quality drifts as upstream models and data change. You cannot acceptance-test that with a one-time demo. A dated, versioned evaluation suite—one that tells you whether a change helped or hurt, and that distinguishes *liveness* (the system is running) from *outcome* (the system is producing correct, current results)—is not a post-launch task; it is the product. Gartner's June 2025 forecast that over 40% of agentic AI projects will be canceled by end of 2027 names inadequate risk controls as a leading cause, alongside unclear business value and escalating costs (Gartner, 25 June 2025). **Integration into existing systems.** Identity, data access, latency and cost budgets, observability, audit trails, compliance review—none of this appears in a pilot. Production must pass through all of it. Enterprises that reach production scope these constraints before the build. Those that stall discover them at month four, after the deck has already declared success. **Ownership and operating model.** A pilot has a sponsor; production needs an owner—a named team accountable for the system after launch, with runbooks, on-call responsibilities, a re-training schedule, and a path to change the system safely. Enterprise AI that reaches meaningful production has answered "who owns this in eighteen months?" before it ships. Most stalled efforts never did. A Gartner survey of 782 infrastructure and operations leaders (surveyed November–December 2025, published April 2026) found only 28% of AI use cases fully met ROI expectations, and 57% of those reporting a failure said their organisations had expected too much, too fast (Gartner, April 2026). MIT NANDA's *The GenAI Divide: State of AI in Business 2025* (August 2025), reviewing 300 publicly disclosed AI initiatives alongside 52 structured interviews and a survey of 153 senior leaders, found roughly 95% showed no measurable P&L impact—attributing the gap primarily to a learning gap (systems that do not retain feedback, adapt to context, or improve over time) and to flawed enterprise integration, rather than to model limitations. The pattern is consistent across methods and geographies. --- ## What separates enterprise AI that reaches production from a pilot that stalls? Enterprise AI that reaches production is distinguished not by a better model but by four organisational conditions: data that is production-ready before the build starts, an evaluation framework tied to business outcomes rather than benchmark scores, integration designed for the existing system landscape, and an operating model with a named team and a clear escalation path. The table below maps the gap at the enterprise-adoption altitude—the organisational readiness dimensions that decide whether a project crosses into production. (For the technical architectural dimensions of the pilot-to-production transition, see [what changes between an AI pilot and production](/pilot-vs-production-ai-what-changes/).) | Dimension | A pilot that stalls | Enterprise AI that reaches production | |---|---|---| | **Data readiness** | Curated demo slice; quality assumed | Real operational data; quality and freshness/provenance gated before the build | | **Integration** | Standalone; constraints deferred | Identity, access, latency/cost, observability, compliance scoped up front | | **Evaluation and reliability** | Judged on a happy-path demo | Dated, versioned eval suite; liveness distinguished from outcome; drift monitored | | **Ownership and operating model** | A champion and a deadline | A named owner, runbooks, on-call; "who owns this in 18 months?" answered before launch | | **Executive sponsorship** | Tied to the exciting demo | Sustained through the unglamorous integration-and-evaluation middle | | **Verification and provenance** | Output trusted because it looks plausible | Independently verifiable; human-checkable trail as a deliverable | IDC found only ~4 of every 33 AI proofs-of-concept reach production—approximately 88% do not—and the cause is consistently the conditions in the left column, not the model (IDC via CIO, March 2025). **Scope discipline is the under-discussed variable.** Enterprise AI deployments that succeed tend to start narrow enough that every failure mode is observable before expanding. The failure pattern—launching 33 pilots and hoping some survive—is the inverse of what actually reaches production. A single use case, delivered into a real operating environment with evaluation and ownership built in from day one, is the reliable unit of enterprise AI. Then the discipline—the data contract, the eval suite, the deploy-and-verify loop, the ownership model—is reused on the next system. **The operating model gap is the most consistently under-resourced.** Organisations that budget for the build and not for the run are common; they produce systems that work on launch day and degrade quietly thereafter, with no named team to notice. The cost of this is not just a failed system—it is a failed system with active downstream consequences that nobody is accountable for. One concrete illustration of the provenance problem: in October 2025, Deloitte's Australian member firm agreed to refund the final installment of its government contract—approximately A$97,000 of a ~A$440,000 engagement—after AI-generated citations to non-existent sources were discovered in a delivered report (Fortune, 7 October 2025). That is not an edge case; it is what happens when verification and provenance are treated as optional rather than as delivery requirements. If you suspect your pilot is already in this stall pattern, [the five-signal diagnosis](/has-your-ai-pilot-stalled-5-signal-diagnosis/) covers the warning signs. For the specific reasons pilots fail to advance to production, see [why AI pilots fail](/why-ai-pilots-fail/). --- ## How should an enterprise actually adopt AI? Start on a scope narrow enough to measure every failure. Prove the operating model before expanding the use case. Then scale on evidence rather than on enthusiasm. The sequence below applies regardless of which model, which platform, or which vendor. The sequencing problem is the same across all of them. 1. **Audit data readiness before writing a line of model code.** The production failure is almost always upstream of the model. Map the data sources, governance gaps, freshness and provenance requirements, and labelling needs for the specific use case before committing to a build timeline. If the data is not ready for production, the system will not be either. 2. **Define what "working" means in business terms before the first sprint.** Name the metric, the threshold, and the human-review process. A pilot that cannot state what success looks like in production—in terms of a business outcome, not a benchmark score—has no legible exit criterion and no way to justify scaling. 3. **Scope the first deployment to a single workflow, a single team.** Small enough to observe all failure modes. Large enough to be a real operating environment and not a sandbox. The constraints you discover on a real workflow are the constraints that matter; the constraints you discover in a sandbox are not. 4. **Build the operating model in parallel with the technical build.** Name the team that owns production. Define the re-training cadence. Set up the alerting. Establish the human-review cycle. None of this is post-launch work. By the time you launch, the operating model should already be running. 5. **Treat the first production deployment as a measurement exercise, not a launch.** Instrument everything. Run the human-review cadence from day one. Collect the failure taxonomy. Use it to scope the next expansion. A launch that is not measured is a pilot that reached production by accident. 6. **Expand scope only when the operating model has held at the current scope.** This is the discipline that separates enterprise AI from the pilot cycle. A system that is holding reliably at narrow scope, with its evaluation suite green and its ownership model functioning, is a system you can extend. One that is shaky at narrow scope will not become stable by expanding. For how NewGenApps structures an engagement to follow this sequence, see [AI consulting](/ai-consulting/). --- ## Frequently asked questions **What percentage of enterprise AI reaches production?** The delivery gap is well-documented. IDC's analysis of AI proofs-of-concept found only about four out of every 33 reach widescale production—approximately 88% do not (Ashish Nadkarni, IDC, *CIO Playbook 2025*; reported CIO, March 2025). Gartner predicted in July 2024 that at least 30% of GenAI projects would be abandoned after proof of concept, citing poor data quality, escalating costs, and unclear business value as the primary reasons (Gartner, 29 July 2024). The gap is a delivery and organisational problem, not a capability problem. **What is the biggest reason enterprise AI fails to reach production?** The most common reason is that the delivery conditions are not in place before the build begins: data that is not production-ready, evaluation criteria tied to benchmark accuracy rather than business outcomes, no integration plan for existing systems, and no named team to own the model in production. Informatica's 2025 survey of 600 CDOs found 43% cite data quality and readiness as the primary obstacle, with 97% reporting difficulty demonstrating GenAI business value (Informatica *CDO Insights 2025*, January 2025).\* The model is rarely the failure point. **How long does it take to get enterprise AI to production?** Scope, data readiness, and integration complexity are the dominant variables. A narrow, well-scoped use case with governed data and a clear evaluation framework can reach a limited production deployment in eight to twelve weeks. Broader or less-prepared deployments commonly take six to eighteen months, and many restart after the initial pilot fails. Starting narrow, measuring before expanding, and proving the operating model at each scope level is the approach most consistent with sustained delivery. **What does "meaningful" mean in the context of enterprise AI outcomes?** Meaningful means a measurable, attributed impact on a business metric—cost reduced, time saved, revenue influenced, error rate lowered—at a scale and reliability level that justifies ongoing operating cost. It excludes demos, one-off experiments, and pilots that generate insight but not operational change. **How is enterprise AI different from an AI pilot?** A pilot proves that a model can perform a task on representative data. Enterprise AI proves that the model can perform that task reliably, within the firm's existing systems, under real operating conditions, with a team accountable for its ongoing performance. The gap between the two is the delivery problem this page addresses. --- ## Getting enterprise AI to production is a delivery problem The enterprise AI problem in 2025–2026 is not that the models are insufficiently capable. It is that the delivery infrastructure—data, evaluation, integration, operating model—is not built around them. Gartner forecasts worldwide GenAI spending of approximately $644 billion in 2025, up roughly 76% year on year, with most of that sum in hardware and infrastructure rather than enterprise application value (Gartner, 31 March 2025). Spend buys access. It does not buy the delivery discipline that converts access into outcomes. That discipline is what NewGenApps is built around: a senior team that scopes the production-delivery requirements before the build, treats the evaluation framework and data contract as first-class deliverables, tests against real production failure modes rather than benchmark suites, and hands over a running system with the operating model to sustain it—with reliability independently verified rather than asserted from a demo. The people who scope the build are the people who see it through to production. That is what closes the gap between the two columns in the table above. Getting enterprise AI to production is a delivery problem, and a tractable one. See [AI consulting](/ai-consulting/) or [book a 30-minute working session](/book/). If you already have a stalled enterprise pilot, [AI Rescue](/ai-rescue/) is built for that. *NewGenApps. Stay a step ahead, always.* --- *Sources: IDC / Lenovo CIO Playbook 2025, Ashish Nadkarni, reported CIO 25 March 2025 · KPMG LLP + UT Austin McCombs, Harvard Business Review, 19 March 2026 · Gartner, 31 March 2025 (GenAI spend forecast) · Gartner, 15 January 2026 (total AI spend forecast) · Gartner, 29 July 2024 (GenAI project abandonment prediction) · Gartner, 25 June 2025 (agentic AI cancellation forecast) · Gartner, April 2026 (I&O AI ROI, 782 leaders surveyed Nov–Dec 2025) · MIT NANDA, The GenAI Divide, August 2025 · Informatica CDO Insights 2025, January 2025 (vendor-sponsored; 600 CDOs) · Fortune / Deloitte Australia, 7 October 2025.* *\* Informatica is a data management vendor; the CDO Insights 2025 survey was vendor-sponsored. The 43% and 97% figures are consistent with independent sources cited above.* ============================================================================== # How Do You Know an AI System Works in Production? URL: https://newgenapps.com/how-do-you-know-an-ai-system-works-in-production/ ============================================================================== **In short.** An AI system works in production when it produces evidence of **evaluated reliability**: measured performance on a frozen, representative, leak-free held-out set — which bounds, not guarantees, production performance, under the assumption that the held-out set matches live traffic — confirmed by outcome instrumentation on live traffic, and checked by someone other than its author. A confident demo, a green dashboard, and a high offline accuracy score are each correlated with a system that works. Each can also be entirely true while the system is failing. The standard matters because the gap between "looks right in the notebook" and "delivers the intended outcome at scale, consistently" is where most AI deployments quietly fail. OpenAI's evaluation guidance (developers.openai.com, confirmed live 2026-06-25) is direct: a common failure mode is "creating eval datasets that don't faithfully reproduce production traffic patterns" — meaning even teams who run evaluations often answer the wrong question. --- ## What does "works" even mean for an AI system? "Works" means the system produces the intended outcome reliably on the inputs it actually encounters — not on a curated sample, not on average across a test set that was tuned alongside the model. Three things get conflated, and the conflation is expensive: **Works in the demo** means: on a narrow set of inputs chosen by the person who built the system, the system produced output that looked right. A demo has no denominator. It shows you the system *can* succeed; it says nothing about how often it does. **Works on paper** means: on an offline test set the build team controlled, measured at some point during or after development, the system hit a target score. This is more than a demo. It is still measuring performance in a different universe than the one the system runs in. **Works in production** means: on the actual distribution of inputs the deployed system encounters, the system produces the intended outcome — and someone has the measurement to prove it. That third definition is the only one that matters for a system in service. Most AI projects never get an honest answer to this question. The [95% statistic](https://newgenapps.com/the-95-percent-ai-pilot-statistic-explained/) from MIT Project NANDA's 2025 research — that the vast majority of GenAI pilots show no measurable P&L impact — reflects, at least in part, a failure to distinguish these three. That said, production AI does work at scale for a minority of organisations. McKinsey's "State of AI in 2025" (published November 5, 2025; n=1,993 in 105 countries) found that roughly 6% of organisations attribute more than 5% EBIT impact to AI — a small but real cohort of high performers — and 39% attribute any measurable EBIT impact at all. LangChain's "State of Agent Engineering" (survey conducted November 18 – December 2, 2025; n=1,340) found that 57.3% of surveyed teams now have agents running in production. The question this page addresses is not whether production AI can work, but what distinguishes the teams who can demonstrate it from those who cannot. The answer is the evidence regime — evaluated reliability — not deployment alone. The NIST AI Risk Management Framework (AI RMF 1.0, January 2023) puts the requirement plainly: AI trustworthiness requires systems that are "valid and reliable," with test, evaluation, verification, and validation (TEVV) tasks spanning the entire AI lifecycle, not just the pre-launch gate. --- ## Why can't you tell from a demo, a dashboard, or accuracy alone? Each of the three common proof sources — a demo, a dashboard, and an offline accuracy score — measures something real. Each fails to measure evaluated reliability on live production inputs. **A demo** is a sample of size roughly one, on inputs the presenter chose. It has no denominator. What a demo hides is technically called the input distribution: it shows you the mode of the system's behavior, never the tail — and AI systems fail in the tail. A system that handles the ten prompts in the demo can degrade silently on the ten thousand input variations it encounters in practice. **A green dashboard** measures liveness: the service is up, latency is acceptable, error rate is within tolerance, throughput is normal. It does not measure outcome — whether the answers were correct, whether the decisions were good. A system can be fully live and entirely wrong: it can return a confident, well-formed, on-time response every time, and the response can still be incorrect. The full derivation of the liveness-versus-outcome distinction belongs to a separate treatment; the one-liner this page owns is: green means "it ran," not "it was right." See [production AI liveness vs outcome](https://newgenapps.com/production-ai-liveness-vs-outcome/) for the monitoring half of this question. **High offline accuracy** is the most deceptive of the three proxies, and where the precision matters most. A model can score high on its own test set and fail silently on production inputs, for three independent reasons: *Train-serving skew.* The data or feature pipeline differs between training and serving, so the model encounters different inputs in production than it was evaluated on offline. Google's engineering canon names this directly: Rule #37 of Martin Zinkevich's *Rules of Machine Learning* (Google for Developers, last updated 2025-08-25) states that "training-serving skew is a difference between performance during training and performance during serving," and recommends explicit monitoring because it otherwise goes unnoticed. *The offline-to-online gap.* Even with no skew, offline metrics are proxies for the online objective, and the two correlate weakly. Castells and Moffat, writing in *AI Magazine* (Wiley/AAAI, Vol. 43, No. 2, June 2022, DOI: 10.1002/aaai.12051), documented for recommender systems that "offline procedures do not yield results sufficiently correlated with" online controlled experiments — a finding that generalizes across ML systems to the extent that the same model sees different traffic offline and online. *Contamination and leakage.* If the held-out set is not truly held out — if near-duplicates or label-revealing features appear in training — the offline number measures memorization, not generalization. A 2026 arXiv paper on inference-time decontamination (Chai, Zhe, and Sakuma, arXiv:2601.19334, January 27, 2026) showed this can account for removing 22.9% of MMLU score inflation. The benchmark saturation problem compounds this: a 2026 study by Akhtar et al. and the Evaluating Evaluations Coalition (arXiv:2602.16763v1) found that 29 of 60 AI benchmarks examined show "high or very high saturation," meaning frontier models can no longer be statistically distinguished on them — the benchmarks themselves have degraded as measurement instruments. None of these three proxies is worthless. A demo is a reasonable first signal. A green dashboard is a necessary operations requirement. Offline accuracy is a prerequisite for any serious deployment. They are necessary, not sufficient. --- ## What evidence actually proves an AI system works? Three types of evidence together constitute proof. No single layer catches every failure mode; the goal is a measurement system where what slips past one check is caught by another. ### 1. A held-out evaluation result on a frozen, representative set The held-out (test) set is data the model never saw during training or selection, used to estimate how the system will perform on unseen inputs. Two requirements are non-negotiable: *Frozen and leak-free.* The set must be fixed before evaluation begins and must not be modified after. Temporal holdout — testing on data that is strictly later than the training cut — is the practitioner's primary defense against contamination, because static sets that persist in the public domain eventually leak into training corpora. The industry shift toward dynamic, time-refreshed evaluation (see Chen et al., arXiv:2502.17521, February 2025) reflects exactly this problem. *Representative of production traffic.* A frozen set that does not match the distribution of inputs the deployed system actually receives gives a precise answer to the wrong question. OpenAI's evaluation guidance identifies "creating eval datasets that don't faithfully reproduce production traffic patterns" as a named anti-pattern; its recommendation is to build and maintain eval sets from production data over time. The precise claim: a held-out evaluation result **bounds** expected production performance — under the assumption that the held-out distribution matches live traffic. The moment that assumption breaks (input drift, a new user segment, a changed feature pipeline), the bound is void. The copy for this does not say "guarantees" or "proves" or "ensures." It says: this is the best available estimate, stated honestly, with a denominator and a confidence interval. A metric without an interval and without a denominator is an impression, not a measurement. ### 2. Outcome instrumentation on live traffic Not "is it serving?" but "did it produce the right outcome?" These are different instruments. Cleanlab's "AI Agents in Production 2025" survey (August 2025), restricted to the cohort already running agents in production (n=95), found that fewer than one in three express satisfaction with their observability and guardrail solutions — and 62% plan to improve observability in the next year. Gartner, in a May 2026 press release, predicts that 40% of organizations deploying AI will implement dedicated AI observability tools by 2028. The implicit message of both: even as of mid-2026, most production AI is instrumented heavily for liveness and barely for outcome. The reason outcome monitoring is hard — and therefore skipped — is label latency: ground truth often arrives days or weeks after the prediction. Senior teams build the delayed-label join anyway, because a lagging truth is more valuable than a real-time guess at what "looks fine." See [production AI liveness vs outcome](https://newgenapps.com/production-ai-liveness-vs-outcome/) for the monitoring half of this question. ### 3. An independent check by someone other than the author The person who built the system has structural incentives to find it working. This is not a competence claim; it is an incentive-alignment claim. The build team, by definition, knows the inputs the system was designed for, the edge cases they handled, and the framing that makes the outputs look good. An independent evaluation — whether a separate internal team, an external review, or an adversarial red-team — is the control that catches the gaps the author's own testing structurally misses. The independent-check requirement is sharpest for LLM-based systems, where "LLM-as-judge" evaluation is increasingly common. Using a model to evaluate a model is legitimate and scalable, but it inherits systematic biases: Shi et al., in a 2025 AACL-IJCNLP study across 15 LLM judges (arXiv:2406.07791v8), found that simply swapping the presentation order of the answers changes which response a judge prefers — they quantify this as judges' position consistency and preference fairness varying substantially across models and tasks. Position bias, self-preference, and verbosity bias are not edge cases — they are consistent properties of unaudited LLM judges. An LLM judge that has not been calibrated against human labels, with order randomized and identity masked, is not an independent check; it is the same model family marking its own homework. Independence is a property of the control structure, not of the tool. The three types of evidence together — held-out result, outcome heartbeat, independent check — are the definition, not the evaluation harness mechanics. What to measure, which metrics, which thresholds, and how to build the infrastructure for continuous evaluation are the scope of the [eval-harness pillar](#) (see that piece when live for the mechanism behind this definition). --- ## "Looks like it works" vs "Proven to work" | Signal | Looks like it works | Proven to work | |---|---|---| | Demo passes | Inputs were chosen by the presenter | Held-out evaluation on a frozen, representative, leak-free production-input set — with a denominator and a confidence interval | | Dashboard is green | System is serving traffic (liveness) | Outcome heartbeat — a live signal that recent outputs were actually correct, not just served | | Offline accuracy is high | Measured on the team's own test set, at training time | A number from the running system on current traffic, closing the offline-to-online gap | | Model gives a confident answer | A confidence score is not calibrated reliability | Output checked against a ground-truth or human-review standard by an independent evaluator | | Ticket is closed / pilot is "done" | Work is done; whether it works is an open question | Freshness verified on inputs, and an end-to-end production probe confirms the live system still produces the right output today | --- ## Frequently asked questions ### Is there a standard definition of "AI evaluation" that I can cite? Yes. The NIST AI Risk Management Framework (AI RMF 1.0, January 2023) establishes "valid and reliable" as a core AI trustworthiness characteristic and defines TEVV — test, evaluation, verification, and validation — as a practice that spans the full AI lifecycle. OpenAI's evaluation guidance (developers.openai.com, confirmed live 2026-06-25) formalizes held-out evaluation and continuous evaluation (CE) on production-representative data as current practitioner norms. The working definition this page proposes is the most precise version of that accepted concept: evaluated reliability means measured performance on a frozen, representative, leak-free held-out set, bounded not guaranteed, confirmed by outcome instrumentation and an independent check. ### Can't the team who built the system also evaluate it? They can measure it; independent evaluation is needed to check it. The structural issue is incentive alignment, not competence. A build team knows which inputs the system handles well, which prompts were tuned, and which edge cases were already addressed. That knowledge shapes their test set, even unintentionally. An independent reviewer does not carry those biases. This is the same reason financial audits require an external auditor and clinical trials require an independent data safety monitoring board — not because internal teams are dishonest, but because the control structure requires separation. ### What if our AI system is too new to have production data to evaluate against? This is exactly when a frozen held-out set drawn from realistic synthetic or pilot inputs matters most. Pre-production, the goal is to construct a representative set from the best available proxy for real traffic: domain-expert-generated inputs, adversarial probes, and inputs sampled from any pilot or shadow-mode run. An honest evaluation on an imperfect-but-documented set is more useful than no evaluation, because it at least states explicitly what it does and does not cover. The eval-harness mechanics — how to build that set, what to measure, and how to maintain it as production traffic accumulates — are covered in the [eval-harness pillar](#) (see that piece when live). ### How is this different from traditional software testing? Traditional software tests deterministic behavior: given input X, the system must produce output Y. Pass or fail. AI evaluation tests probabilistic reliability across a distribution of inputs, which requires statistical thinking, not just pass/fail cases. The right question is not "did it get this one right?" but "what is the expected performance, with what variance, across the range of inputs it will encounter?" That shifts the measurement apparatus from a checklist to a sample, from a binary to a distribution, and from a one-time gate to a continuous instrument. The metric itself is a modeling choice driven by which type of error is expensive in the specific system — there is no universal "accuracy." ### How do I know the production system is still the version that passed evaluation? That is the deploy-verification question, distinct from the evaluation question. Evaluation tells you whether the validated version works. Deploy verification tells you whether the thing now serving traffic is the validated version — whether what was merged is what was shipped, and whether it remains what is running. Those are [different questions with different instruments](https://newgenapps.com/deploy-and-verify-ai/). --- ## The gap this question reveals If the answer to "how do we know it works?" in your organisation is currently "it passed the demo" or "it's been running for three months without complaints," the gap is not unusual. A system that doesn't throw errors is not the same as a system that is right. Models degrade quietly — they do not throw exceptions when their outputs become less reliable, which means silence is the default failure mode of an unmonitored system, not a signal of health. The starting point is agreeing on what "works" means for the specific system, then building the smallest evaluation that would give an honest answer. If you want to think through what that looks like for your context, [how we work](https://newgenapps.com/how-we-work/) describes the method — a small senior team, independent verification, evaluated reliability; not a demo and a handover. Or book an [AI working session](https://newgenapps.com/book/) to start from your system's current evidence state. *NewGenApps — production AI, proven. Stay a step ahead, always.* --- *Sources cited in this piece: NIST AI Risk Management Framework (AI RMF 1.0), January 26, 2023; OpenAI, "Evaluation best practices," developers.openai.com, confirmed live 2026-06-25; Zinkevich, M., Rules of Machine Learning (Rule #37), Google for Developers, last updated 2025-08-25; Castells, P. & Moffat, A., "Offline Recommender System Evaluation: Challenges and New Directions," AI Magazine (Wiley/AAAI), Vol. 43, No. 2, pp. 225–238, June 2022, DOI: 10.1002/aaai.12051; Akhtar, M. et al. and the Evaluating Evaluations Coalition, arXiv:2602.16763v1, 2026; Cleanlab, "AI Agents in Production 2025," August 2025, production cohort n=95; Gartner, AI observability press release, May 12, 2026; Shi, L. et al., "Judging the Judges," AACL-IJCNLP 2025 (ACL Anthology), arXiv:2406.07791v8; Chai, J., Zhe, and Sakuma, arXiv:2601.19334, January 27, 2026; Chen, S. et al., arXiv:2502.17521, February 2025; MIT Project NANDA, "The GenAI Divide," 2025 research (see the linked 95% pilot explainer for the full citation); McKinsey, "The State of AI in 2025: Agents, Innovation, and Transformation," published November 5, 2025, survey June 25 – July 29, 2025, n=1,993; LangChain, "State of Agent Engineering," survey November 18 – December 2, 2025, n=1,340.* ============================================================================== # AI Evaluation Harness: What to Measure and How URL: https://newgenapps.com/ai-evaluation-harness/ ============================================================================== # The AI evaluation harness: what to measure and how Most teams can build a demo that works once. Far fewer can prove a system still works after the next prompt edit, the next model upgrade, or the vendor swapping the model underneath them. The instrument that closes that gap is an evaluation harness — and it is the discipline most teams skip on the way to shipping. **In short** — An AI evaluation harness is a repeatable, automated system that scores model or agent output against a frozen, representative dataset using defined graders, and gates every release on the result — so that each change is measured against a fixed bar instead of judged by eye. It is not a demo, a one-off spot-check, or a human impression; a demo proves an input exists on which the system works, while a harness proves the system did not get worse on inputs you fixed in advance. The distinction matters because production AI systems are non-deterministic — the same input can yield different output across runs and across model versions you do not control — so "it looked good" is not a measurement. If this is well understood, why do most teams still ship without one? Because the work is invisible: in LangChain's *State of Agent Engineering 2025* survey (1,340 respondents, fieldwork November 18 – December 2, 2025), roughly 89% of teams had implemented observability for their agents, but only 52.4% ran offline evaluations and 37.3% ran online ones — evaluation discipline lags the tooling, and quality is the single most-cited barrier to production at 32%. NewGenApps builds these harnesses as the verification layer behind production AI — the mechanism that turns "we think it works" into a number you can defend. This piece covers what an evaluation harness is, what to measure, how to build one, why it is the hard part, and how it differs from the cheap alternatives teams reach for instead. --- ## What is an AI evaluation harness? **An AI evaluation harness is a repeatable, automated system that scores model or agent output against a frozen, representative dataset and gates releases on the result — it is not a demo, a one-off test, or a human impression.** It is the instrument that turns "it looked good" into "it scored X on the held-out set, versus Y last release." A harness has four components, each of which is load-bearing: - **A frozen, held-out dataset.** The eval set is held out and does not move while you iterate. The moment the set drifts with the system, you are grading against a moving ruler and the score is meaningless. "Frozen" is the word a brochure leaves out, and it is what makes the definition correct rather than generic. - **Defined graders.** Each task has an explicit way to decide whether an output is correct — deterministic and rule-based where outputs are checkable, and an LLM-as-judge grader where they are open-ended (covered below). - **Threshold logic that blocks release on regression.** A harness that reports a number but cannot block a release is a dashboard, not a harness. The gate is what makes it a control rather than a vanity metric — the same distinction as a passing test suite versus a coverage chart nobody enforces. - **Persistent tracking across versions.** Scores are stored per version so you can answer the only question that matters during iteration: did this change make it worse than what we shipped? It is worth separating two terms that get used interchangeably. The **harness** is the infrastructure; the **evals** are the individual measurements it runs. Arize AI, an observability vendor, frames the harness as "the infrastructure that lets you evaluate an AI system consistently as the system changes" (*The best eval harness for production AI and agents*, June 1, 2026) — consistency across model, prompt, and framework iterations is the whole point. The mental model worth planting: **an evaluation harness is regression testing for a non-deterministic system.** Everyone accepts that you do not ship code without tests that fail when you break something. An LLM or agent has no compiler and no deterministic unit test — so the harness is how you recover what tests gave you, a tripwire on "did this change make it worse," in a setting where exact-match assertions do not apply. A peer-reviewed framing of the same idea describes a readiness harness as an "integrated testing and validation system" that "establishes measurable gates preventing substandard systems from reaching users" (Maiorano, *LLM Readiness Harness: Evaluation, Observability, and CI Gates for LLM/RAG Applications*, arXiv preprint, May 22, 2026). This pillar is the measurement mechanism behind the system-level question of [how to know if an AI system works in production](/how-do-you-know-an-ai-system-works-in-production/) — that piece defines what "works" means; this one defines how you measure it. --- ## What should you measure when evaluating an AI system? **The metrics that matter in production are task-completion rate, confident-but-wrong rate, regression against a baseline, and latency-and-cost efficiency — measured together, not collapsed into a single aggregate score.** Measurement is a mechanism, not a number, and the harness enforces that all four dimensions clear their threshold before a release ships. **Task-completion rate — but define "success" before you measure it.** Computing the rate is the easy part; the grader is the hard part. Reach for graders in descending order of trustworthiness: programmatic and reference-based checks first (exact match, regex, schema-valid JSON, a passing unit test, retrieval hit@k) because they are cheap and deterministic; a small human-labeled gold set as the ground truth everything else is calibrated against; and an LLM-as-judge only for genuinely open-ended output. Most "we cannot evaluate open-ended output" claims collapse once the task is decomposed into checkable assertions. **Confident-but-wrong rate — as a measured rate, not an anecdote.** This is the production-specific failure mode a demo structurally hides: fluent, plausible, wrong output. Make it a first-class number — the fraction of outputs confidently asserted and factually or operationally incorrect — because the silent wrong answer is the expensive one. The rigorous backing is calibration: a well-calibrated model's stated confidence should match its empirical accuracy, and models trained with RLHF are documented to become miscalibrated, more overconfident than the base model (OpenAI, *GPT-4 Technical Report*, 2023, Figure 8). A separate analysis across nine models and three factual-QA datasets found that RLHF-tuned models can paradoxically suffer increased miscalibration on easier queries (Chhikara, *Mind the Confidence Gap*, arXiv preprint, February 2025). The agent-specific treatment of this failure mode is its own subject; here it is one metric among four. For the broader agent context, see [agentic AI](/agentic-ai/). **Regression against a frozen baseline — the axis that justifies the word "harness."** A single score is nearly useless; the value is the delta against the last known-good version. You are not asking "is it good," you are asking "did this prompt edit, model upgrade, or retrieval change make it worse than what we shipped." Two practitioner subtleties keep this honest. First, report per-slice regression, not just the aggregate — a change that lifts the average can quietly destroy a critical sub-population (the "it improved overall but broke refunds" failure). Second, because the system is stochastic, run each eval item multiple times and treat the score as a distribution; a single pass or fail conflates a real regression with sampling noise. **Latency and cost per successful task.** Not p50 latency and not raw token spend, but cost per task that actually succeeded — including retries, judge and guardrail calls, and any human review. For multi-step or agentic systems this compounds: every step is a call and every retry multiplies. A harness that tracks quality but not the cost of achieving it will happily approve a release that is three times more expensive for the same result. The synthesis: you are not measuring "how good is the model." You are measuring whether this change moved success, false-confidence, cost, or latency in the wrong direction, on inputs you froze in advance, across the slices you care about — and blocking the release if it did. --- ## How do you build an AI evaluation harness? **Building an evaluation harness takes five steps: assemble a frozen, representative held-out set; define your graders; set thresholds and a release gate; run the harness on every change; and track drift over time.** 1. **Assemble a frozen, representative held-out set.** Sample from real, production-shaped inputs, including the messy, adversarial, and edge cases — not a curated happy-path set. Freeze it and version it. Two failure modes deserve to be named explicitly. *Leakage:* never let an eval item appear in a prompt, a few-shot example, a fine-tuning set, or the retrieval corpus, or you are grading memorization rather than capability (this is the contamination problem, covered below). *Representativeness:* a clean set that omits the inputs that fail is the same lie as a curated demo. On size, practitioner convention holds that a production gold set typically contains 100–300 diverse, mutually exclusive prompt-response pairs to give statistical significance for regression metrics; treat that as a starting heuristic, not a rule, and prioritize coverage of the input distribution over raw count. 2. **Define your graders per task.** Reach for programmatic and reference-based graders first; reserve LLM-as-judge for genuinely open-ended output; anchor everything to a small human-labeled gold set. Write down what "success" means before you look at outputs — pre-registering the bar stops you from rationalizing the score after the fact. The grader-definition step is where most teams under-invest, and it is where the failure modes covered below live. 3. **Set thresholds and a release gate.** A threshold is the minimum acceptable score per metric; the gate is the automated block on release if any threshold fails. Distinguish an absolute floor ("must score at least X") from a regression threshold ("no drop greater than X versus the last-version baseline"). Use multi-metric gating: all metrics must clear, and failing one fails the release. Set the bar per critical slice, not only on the aggregate. 4. **Run the harness on every change — automatically, in CI.** Every prompt edit, model version, retrieval or index change, and preprocessing change triggers the harness, and each item runs multiple times for stochastic stability. The principle is that evaluation must be cheaper than a bad release, so the goal is to catch the silent regression the day it is introduced rather than in production. This is the seam to [how to integrate verification into deployment](/deploy-and-verify-ai/) — the harness is what the deploy gate runs. 5. **Track drift and re-validate the set over time.** A harness that runs only at release misses production drift. The world moves (concept drift), the model is deprecated under you, and the eval set slowly stops resembling live traffic — concept drift, where the relationship between inputs and the target shifts so a deployed model loses accuracy with no code changes, is the formal mechanism (Gama et al., *A Survey on Concept Drift Adaptation*, ACM Computing Surveys, 2014). Periodically refresh and augment the held-out set from current production inputs, re-check the human-gold anchor, re-freeze after each update, and never fold the current test items back into training or retrieval. Offline evaluation is the snapshot; its online counterpart is [outcome monitoring](/production-ai-liveness-vs-outcome/) — the two are a pair, covered in the FAQ below. --- ## Why is evaluation the hardest part of getting AI into production? **Evaluation is the hardest part because production AI systems are non-deterministic, the failure modes that matter most are rare and correlated, and the cost of skipping it only appears after deployment — by which point trust is already damaged.** The first reason is non-determinism. Standard software QA assumes a deterministic system: the same input yields the same output, and an exact-match assertion is a valid test. Neither holds for an LLM or agent. The same input yields different output across runs, and the vendor can change the model under you between Tuesday and Wednesday. The assertions software engineers rely on do not transfer, which is why you need a statistical apparatus — a frozen set, repeated runs, distributions — rather than a single green check. The second reason is the shape of the failures that matter. A model can pass a thousand random tests and still fail systematically on a class of inputs, because errors in these systems are rare and correlated rather than random and independent. The confident-but-wrong failure is the canonical case: the output is fluent and plausible, so it passes a casual look, and it is wrong in a way that only surfaces when a particular kind of input arrives at volume. Spot-checking is structurally blind to this; only a measured rate on a representative set sees it. The third reason is incentives. Demos are cheap, fast, and persuasive; harnesses are expensive, slow to build, and invisible until they catch something. The result is a documented adoption gap. In LangChain's *State of Agent Engineering 2025* (1,340 respondents, fieldwork November–December 2025), roughly 89% of teams had observability in place but only 52.4% ran offline evaluations and 37.3% ran online ones, with quality the top barrier to production at 32%. The cost of the gap is measurable: in Galileo's *State of Eval Engineering Report* (500+ enterprise practitioners, February 25, 2026), 84.9% of organizations experienced an AI incident within six months of production deployment, and teams that skipped evaluations for "low-risk" behaviors saw 2.3 times more production incidents than comprehensive testers — while teams with 90–100% evaluation coverage reported 70.3% "excellent reliability" against 32.4% for teams below 50% coverage, a 38-point gap. (Galileo sells evaluation tooling, so read its figures as a directional signal from a motivated source rather than a population census.) This is the defensible version of the point the industry keeps gesturing at when it calls evaluation the number-one blocker: the gap is real, it is dated, and it is attributable — and it is consistent with what NewGenApps observes in the stalled pilots it is asked to rescue, where the missing piece is almost always this layer. The response is not more demos. It is a harness that makes the failure modes visible and automated before they reach users. ### A note on LLM-as-judge — the limit a brochure pretends away The obvious fix for grading open-ended output is to use a model to grade the model. It works: the foundational study shows strong judges reach over 80% agreement with human preferences — about the level at which humans agree with each other (Zheng et al., *Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena*, NeurIPS 2023). But treating the judge as ground truth is the single most common evaluation mistake, and the failure modes are named and documented: - **Position bias** — the judge favors an answer by its order, first or second, rather than its merit. The effect has been systematized across 15 judges and roughly 150,000 instances and varies by judge and task rather than being random noise (Shi et al., *Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge*, IJCNLP/AACL 2025; first documented in Zheng et al., 2023). - **Verbosity bias** — longer, more fluent answers are preferred regardless of correctness (Zheng et al., 2023). - **Self-enhancement bias** — a judge favors outputs from its own model family, or simply the outputs it finds more familiar, over the more correct one (Zheng et al., 2023; further documented in Gu et al., *A Survey on LLM-as-a-Judge*, arXiv preprint, 2024). Adoption confirms the tension: in Galileo's February 2026 survey, 67% of teams used an LLM-as-judge approach and 93% of those reported major reliability problems, with 42.4% citing consistency failures specifically. The production-grade move is to calibrate the judge against the human-labeled gold set, report the judge-to-human agreement number, and control for the known biases — randomize answer order, length-normalize, and avoid grading a model with a judge from its own family. A judge you have not measured against humans is not a measurement instrument; it is a second opinion with the same blind spots as the thing it grades. This is the case for independent verification at the grading layer: an evaluation owned and tuned by the same team that built the system, with an uncalibrated judge, tends to produce the score the team hoped for. It also argues for grader parsimony — reach for the simplest grader that holds before adding a model-based one, since each added model in the loop adds its own un-audited biases to control for (the general principle of adding model complexity only when it measurably improves the outcome is set out in Anthropic, *Building Effective Agents*, December 2024). ### A note on leakage — the data-integrity failure inside the eval The eval set has its own data-integrity contract, and the worst way to break it is invisible: test-set leakage. If any eval item has been seen by the model — in pre-training, fine-tuning, few-shot examples, or the retrieval corpus — the score measures memorization, not capability, and you will ship a regression the harness told you was an improvement. This is not hypothetical: a controlled study probed modern benchmarks with a missing-option guessing protocol and found that on MMLU, ChatGPT and GPT-4 reconstructed the masked answer 52% and 57% of the time respectively — a signature of memorized test data rather than reasoning over it (Deng et al., *Investigating Data Contamination in Modern Benchmarks for LLMs*, NAACL 2024). The survey literature formalizes the problem and points to post-cutoff, continuously refreshed eval sets as the standard mitigation (Xu et al., *Benchmark Data Contamination of LLMs: A Survey*, arXiv preprint, 2024). Two consequences follow. First, public benchmark scores are an upper bound, not your number — a vendor's MMLU or GSM8K figure says little about your task on your data, partly because of contamination and partly because it is not your distribution. Your held-out set is the only score that predicts your production behavior. Second, the harness is the model-error gate; a data-integrity contract is the data-error gate. An eval can be flawless and the system still fail because a stale or wrong input fed it, and a perfect data contract cannot save you from a model that regressed. They are a pair. --- ## How does an evaluation harness compare to vibes, spot-checks, and demos? **The difference is systematic, repeatable signal versus a one-time impression: a harness runs the same graded tests on every change against a frozen baseline and gates release, while a spot-check, a demo, or a "vibes" review produces no comparable baseline and cannot detect regression.** Teams reach for the cheap alternatives because they are fast and feel like enough — until they are not. | Approach | What it actually tests | What it cannot catch | Why teams reach for it | |---|---|---|---| | **Vibes / spot-checking** | A handful of outputs someone eyeballed once | The distribution of outputs; whether the last change regressed; the silent-wrong-answer rate | Zero setup; feels fast; "it looked fine" | | **A single one-off test** | One input, one moment | Variance in stochastic output; any input you did not pick; drift after the test | Looks like rigor; is an anecdote with n=1 | | **A polished demo** | That there exists an input on which it works | Unchosen inputs; the curated-away failure cases; cost and latency at volume | Persuasive to stakeholders; built to impress, not to falsify | | **Public benchmark scores** | The model on someone else's distribution | Your task and data; contamination inflates the number; not a release gate | Authoritative-looking; vendor-supplied; free | | **Offline accuracy alone (no gate)** | One aggregate number, once | Per-slice regression; the change that helps the average and breaks a sub-population; nothing blocks a bad release | Feels like "we have metrics" | | **Evaluation harness** | Every change, against a frozen representative set, with calibrated graders, per slice, gated | The point: it is built to catch exactly the regressions and tail failures the rows above hide | Setup cost; needs a held-out set and an owner | *A demo can be optimized; a harness cannot be faked. The harness runs the same graded tests on every change against a frozen baseline — the others do not.* Every cheap alternative fails the same way: it produces a watched, single, chosen observation when production correctness requires a measured rate on unchosen inputs, with the tail characterized and re-run on every change. The hidden cost is that the cost of failure is deferred to production, where it is more expensive to fix and more damaging to trust. The harness is not QA you add at the end; it is the instrument that makes iteration possible at all, because without it you cannot tell improvement from regression. A system you cannot measure on a frozen set is not "almost shippable" — it is unmeasured, and an unmeasured AI system is an unverified claim, not a product. --- ## FAQ **What is an AI evaluation harness?** An AI evaluation harness is a repeatable, automated system that scores model or agent output against a frozen, representative dataset using defined graders and gates releases on the result. It is the infrastructure that lets you measure an AI system consistently as it changes, so every version is graded against a fixed bar rather than judged by eye — distinguishing it from a demo, a spot-check, or a human impression. **What metrics should an AI evaluation harness measure?** It should measure at least four axes together: task-completion rate, confident-but-wrong (false-confidence) rate, regression against a frozen baseline, and latency and cost per successful task. Measured together they answer the production question — did this change move any of them in the wrong direction on inputs fixed in advance — which a single aggregate score cannot. **How many test cases do you need in an evaluation dataset?** Coverage of the real input distribution matters more than raw count: the set must include the messy, adversarial, and edge cases that fail, not just the happy path. As a practitioner heuristic, production gold sets often run 100–300 diverse, mutually exclusive prompt-response pairs to give regression metrics statistical significance, but treat that as a floor to refine against your task, not a target to hit. **What is an LLM-as-judge grader and when should you use it?** An LLM-as-judge grader uses a language model to score another model's output, and it is the right tool for genuinely open-ended output where deterministic checks do not apply — strong judges reach over 80% agreement with human preferences (Zheng et al., NeurIPS 2023). Use it only after calibrating it against a human-labeled gold set and controlling for documented biases (position, verbosity, self-enhancement); an uncalibrated judge is a second opinion with the same blind spots as the system it grades, not a measurement instrument. **When does an evaluation harness replace outcome monitoring?** It does not. An evaluation harness is offline and pre-release — it runs on a frozen set before you ship and answers "is this version good enough to release, and is it worse than the last one." Outcome monitoring is online and post-release — it watches the live system on real, unchosen traffic and answers "is what we shipped still working now." They are complements: the harness gates what you release, [outcome monitoring](/production-ai-liveness-vs-outcome/) watches what you released, and neither substitutes for the other. --- ## Build the layer your pilot is missing Most teams that "can't tell if it works" are missing exactly this layer. We build evaluation harnesses as part of [how we work](/how-we-work/) — or as the first phase of [AI Rescue](/ai-rescue/) when a pilot stalled for want of one. [Book an AI working session](/book/). *NewGenApps — production AI, proven. Stay a step ahead, always.* ============================================================================== # How to Evaluate an AI Agent Before It Reaches Production URL: https://newgenapps.com/how-to-evaluate-an-ai-agent/ ============================================================================== # How to Evaluate an AI Agent Before It Reaches Production **In short.** Evaluating an AI agent means measuring the trajectory — the tools it called, the order it called them, the data it used — and the outcome together, not just the final answer. Two agents can produce the same correct output; one reached it through a valid reasoning path on fresh data, the other by a lucky guess on stale information. Only measuring the output tells you nothing about which one will hold up at scale. The metric that separates production-ready agents from demo-ready ones is the **confident-but-wrong rate**: how often does the agent produce a fluent, plausible-sounding answer that is factually or operationally incorrect? An agent that fails loudly is fixable. An agent that is fluently, confidently wrong is the production risk that matters. --- ## How do you evaluate an AI agent? Evaluate trajectory and outcome together — measure the tools the agent called, the order and validity of those calls, the freshness of the data it retrieved, and the correctness of the final result — and use a verifier the agent cannot itself game. The distinction matters from the first question. Evaluation is offline and pre-release: it runs on a frozen case set before you ship, and tells you whether this version of the agent is good enough to release and whether it has regressed against the previous one. That is different from what happens after release. Production outcome monitoring watches the live agent on real, unchosen tasks in deployment — it is the detection layer, not the release gate. Both are necessary; neither substitutes for the other. For the online layer, see [Production AI: Liveness vs Outcome](/production-ai-liveness-vs-outcome/). Two axes determine whether an agent's result is production-worthy: **trajectory correctness** (did it use the right tools, in the right order, on valid data) and **outcome correctness** (is the result accurate, current, and validated). A passing result requires both. An agent in the "right answer, wrong path" quadrant — correct output, flawed trajectory — is a latent incident: it will produce the same output by the same broken path until the day the path breaks, and no final-answer check will warn you before that day. The independent-verifier requirement follows from this. If the agent can assess its own trajectory — generating its own test cases, grading its own outputs, operating in a loop where the grader and the agent share the same model family and the same failure modes — the evaluation is circular and gives no production signal. The verifier must be independent of the generator. For the general mechanics of building a measurement harness — frozen sets, grader design, release gates — see [AI Evaluation Harness](/ai-evaluation-harness/). This page applies those mechanics specifically to agents: trajectory, tool-call correctness, and the confident-but-wrong rate. For what "working in production" means at the system level, see [How Do You Know an AI System Works in Production?](/how-do-you-know-an-ai-system-works-in-production/) --- ## Why is evaluating an agent harder than evaluating a single model output? A single model output has one step to grade; an agent has a trajectory — multiple tool calls, branching decisions, accumulated state — and a correct-looking final answer can mask a chain of flawed or unsafe steps that will fail differently under different conditions. Three compounding reasons make agent evaluation structurally harder than scoring a single LLM response: **1. A right answer reached the wrong way is still a production failure.** An agent can return the correct number having read a stale cache, skipped a validation step, or called a write tool it should never have touched — and the final answer looks identical to the one reached correctly. Final-answer-only scoring passes both. In production, the second agent is a latent incident: the day the lucky guess stops being lucky, or the day the unsafe tool call has a side effect, it fails — and the eval said it was fine. **2. Tool calls are where agents actually fail, and they are checkable.** An agent's distinguishing capability is calling tools; its distinguishing failure is calling the wrong one, with malformed arguments, in an invalid order, or fabricating a tool that does not exist. The Berkeley Function-Calling Leaderboard (Patil, Mao, Yan et al., 2024–25) formalizes this, evaluating function calls across simple, parallel, and multi-turn settings, measuring hallucinated-function detection and irrelevance-avoidance as distinct categories. The practitioner takeaway: "we can't evaluate an agent because its behavior is open-ended" collapses the moment you decompose the trajectory into checkable tool-call assertions. Those assertions — right tool, valid arguments, valid order — are mostly programmatic, not judge-dependent. **3. State and non-determinism mean a single passing run proves almost nothing.** The τ-bench benchmark (Yao, Shinn, Razavi, Narasimhan, arXiv:2406.12045, June 2024) introduced `pass^k` — the probability of succeeding on all k independent attempts at the same task — to expose exactly this. The paper reports state-of-the-art function-calling agents dropping sharply from `pass@1` to `pass^k` at higher k; succeed-once is meaningfully more common than succeed-every-time. The arithmetic is plain without citing the specific numbers: a ten-step agent at 95% per-step reliability is roughly 60% reliable end-to-end (0.95^10 ≈ 0.60). A chain of "almost always right" steps is "usually wrong" end-to-end. An agent eval must therefore run each case multiple times and treat the score as a distribution, not a single pass/fail. There is, in practice, still no universally adopted methodology for verifying that a non-deterministic agent has not regressed after a change to its prompts, tools, model, or orchestration — the field is actively building one, which is itself evidence that the problem is real. This is part of why the overwhelming majority of AI proof-of-concepts never reach widescale deployment. According to IDC's *Lenovo CIO Playbook 2025* (February 2025), roughly 88% of AI proofs-of-concept do not advance to widescale deployment — about four in every thirty-three reach production. The evaluation gap is one of the structural causes. For the production engineering view, see [Multi-Agent Orchestration in Production](/multi-agent-orchestration-production/) and [Agentic AI](/agentic-ai/). --- ## What should you measure in an agent evaluation? The five metrics that distinguish a production-grade agent eval from a demo-grade check are: task success on representative real cases, trajectory and tool-call correctness, the confident-but-wrong rate, cost and latency per task, and safety and guardrail adherence — measured against a frozen held-out case set, not against the training distribution. ### Final-Answer Eval vs Agent (Trajectory + Outcome) Eval | Evaluation dimension | Final-answer eval only | Agent eval: trajectory + outcome, on a frozen set, independently verified | |---|---|---| | **What it scores** | Whether the last output matches the gold answer | The output *and* the path: tools called, order, argument validity, data freshness, plus the validated result | | **Catches the right-answer-wrong-path failure?** | No — a lucky guess and a sound reasoning chain look identical at the endpoint | Yes — the trajectory grade fails an invalid path even when the final answer is coincidentally correct | | **Catches the confident-but-wrong rate?** | No — a fluent, plausible, confidently-asserted wrong answer scores the same as any other wrong answer, and is not tracked as a rate | Yes — confident-but-wrong is a first-class measured rate; the trajectory grade often reveals the cause (stale data, wrong tool, skipped check) | | **Catches a stale-data success?** | No — if the stale value matches the gold answer, it passes | Yes — tool-call grading checks whether the right data source was queried at the right decision point, not only what number was returned | | **Production-readiness signal** | Weak — "it produced the right answer on the cases we checked," blind to path, freshness, and silent confidence | Strong — on a frozen representative set, run k times and gated per slice, the agent reaches correct, current, validated results by valid paths, and the confident-but-wrong rate is a stated number | Each metric deserves a working definition: **Task success on representative real cases.** Whether the agent completed its intended task on inputs drawn from the actual production distribution — including edge cases and adversarial inputs — not a curated happy path. This is the baseline floor, not the ceiling. **Trajectory and tool-call correctness.** Whether the agent called the right tools, in a valid order, with well-formed arguments, at the right decision points. This is the evaluation dimension most teams skip and the one most predictive of production failure. The Berkeley Function-Calling Leaderboard provides a formal measurement methodology; at the practice level, most tool-call assertions are programmatic rather than judge-dependent. **The confident-but-wrong rate.** The fraction of agent outputs that are asserted fluently and without hedging while being factually or operationally incorrect. Defined in full in the next section — this is the page's central concept. **Cost and latency per task.** Often skipped, and one of the dimensions that decides whether production economics hold. An agent that solves a task in forty tool calls at twelve seconds and three dollars has a different production cost profile from one that solves the same task in six calls at two seconds and fifteen cents. Both may pass a task-success eval. **Safety and guardrail adherence.** Whether the agent stayed within its sanctioned action boundaries — did not call a write tool on a read-only task, did not expose restricted data, did not take an irreversible action without confirmation. Rule-based assertions at the tool-call level are the right check here. For the general harness mechanics that underpin these metrics, see [AI Evaluation Harness](/ai-evaluation-harness/). --- ## What is the confident-but-wrong rate and why does it matter? The confident-but-wrong rate is the fraction of an agent's outputs that are fluent, plausible-sounding, and factually or operationally incorrect — the metric that distinguishes a production-safe agent from one whose errors accumulate undetected. It is not the same as the hallucination rate in the general sense. It is specifically the rate at which the agent produces wrong outputs *with high expressed confidence* — no hedging, no abstention, no error signal. The contrast makes the definition precise: an agent that errors loudly or says "I cannot verify this" is recoverable. A team sees the failure and corrects it. An agent that is fluently wrong produces outputs that users, downstream systems, and reviewers accept and action. **The failure that hurts in production is not the agent that errors — it is the agent that is plausibly, fluently wrong.** The asymmetry that makes this dangerous is not symmetric noise. An agent that is equally likely to under- and over-state its confidence is merely imprecise; the production risk is an agent that is *systematically* overconfident about its failures — high expressed certainty on exactly the tasks it gets wrong, while the loud, recoverable failures stay rare. That is the shape the confident-but-wrong rate is built to expose: not how often the agent is wrong, but how often it is wrong *and sure*. The mechanism is not incidental. Leng et al. (arXiv:2410.09724, accepted ICLR 2025) documented that RLHF fine-tuning — the standard technique used to improve instruction-following in deployed models — introduces systematic verbalized overconfidence. Reward models used in RLHF training exhibit inherent biases toward high-confidence scores regardless of actual response quality. The implication for evaluators: asking the agent whether it is sure queries a miscalibrated instrument, not a reliable confidence signal. You cannot rely on the agent's self-reported certainty as a proxy for correctness. In agentic systems, the problem compounds across steps. Research into agentic hallucination dynamics describes the failure mode as cascading: a minor logical divergence in an early search or planning step propagates through subsequent tool calls, and the agent's high linguistic coherence masks the underlying functional misalignment — the chain of thought remains grammatically intact while the agent fabricates non-existent API arguments or ignores environment feedback. A wrong early step poisons later steps; the fluent delivery continues throughout. This trajectory connection is the reason the confident-but-wrong rate belongs with trajectory evaluation rather than with final-answer grading. The failure almost always traces upstream: the agent retrieved a stale document and reported it as current; called the wrong tool and trusted its output; skipped a verification step and asserted the unverified result. The final-answer grade cannot see the cause. The trajectory grade can. **How to measure it.** The confident-but-wrong rate is a rate on a held-out set, not an anecdote. Numerator: outputs that are asserted without hedging and are wrong on the frozen gold standard. Denominator: all outputs (or all confident outputs, stated consistently before you measure). Calibration — the gap between an agent's expressed confidence and its actual accuracy — is a formally measurable quantity, not a vibe: the same expected-calibration-error and Brier-score machinery used to score probabilistic classifiers applies once you treat each trajectory's success as the outcome and the agent's stated confidence as the predicted probability. Use the underlying structure: calibration is measurable as a formal score, not just an anecdote. **The honest boundary.** You cannot drive the confident-but-wrong rate to zero. Chasing zero by making the agent abstain on everything relocates the failure to uselessness. The production-grade move is to make abstention a rewarded output in your grader — an agent that says "I cannot verify this" scores better than one that confidently guesses wrong — and to target a measured, bounded rate that is traded off against task success on the cases the business actually requires. The eval makes that trade-off legible; it does not eliminate the need to make it. If you cannot yet state your agent's confident-but-wrong rate, you have not evaluated the agent — you have watched it succeed once. Breck, Cai, Nielsen, Salib and Sculley's *ML Test Score* (IEEE Big Data 2017) operationalizes production-readiness as the presence of specific tests and monitoring: "it looks good" scores zero. --- ## How do you build an agent evaluation before production? Building a production-grade agent evaluation requires five steps: freeze a representative case set, grade trajectory and tool calls alongside the final answer, measure the confident-but-wrong rate explicitly, set a regression gate that runs on every change, and keep an independent verifier that the agent cannot grade past. ### How to Evaluate an AI Agent Before Production **Step 1: Freeze a representative case set from real, agent-shaped tasks.** Sample from production-shaped inputs — including the messy, adversarial, and multi-step cases the agent will actually encounter, not a happy-path set. For an agent, a case is a *task with a known-good trajectory and outcome*, not just an input/answer pair: capture which tools a correct run should call, in what order, on what data, and what the validated result is. Freeze it and version it. Two failure modes to watch for: (i) *leakage* — never let eval cases appear in the agent's prompt, few-shot examples, fine-tuning data, or retrieval corpus, or you are scoring memorization, not capability; (ii) *false representativeness* — a clean set that omits the inputs that fail is the same lie as a curated demo. Lù et al. (AgentRewardBench, arXiv:2504.08942, April 2025) found that the rule-based final-state evaluation used by common web-agent benchmarks tends to *underreport* agent success — a valid alternative path is scored as a failure — and that expert human reviewers agree at a high rate on whether a trajectory succeeded. The evaluation criteria are tractable to define; the difficulty is automating them, not defining them. **Step 2: Grade trajectory and tool calls, not just the final answer.** Score three layers. First, outcome: is the result correct, current, and validated? Reach for programmatic or reference-based graders first — a schema-valid output, a recomputable check, a retrieval hit. Second, trajectory and tool-call correctness: right tools, valid order, well-formed arguments, no hallucinated or unsafe calls. These are mostly deterministic assertions; an LLM judge is not required here. Third, the open-ended residue: only where the output is genuinely free-form should you use an LLM-as-judge, and only if it is calibrated (see step 5). Write down what "success" means before you look at outputs — pre-registration stops you rationalizing the score after the fact. A case where the agent produces a correct final answer via an invalid trajectory is a *failure*, not a pass, because it tells you the agent will produce the same answer by the same invalid path until the path breaks. **Step 3: Measure the confident-but-wrong rate as a first-class number.** For every case, record not just right/wrong but *confidently wrong*: an output the agent asserted without hedging that is factually or operationally incorrect. Track it as a rate, per slice of the task distribution. The right framing is the abstention trade-off: make "I cannot verify this" a rewarded output so the agent that abstains under uncertainty scores better than the agent that confidently guesses. Because RLHF training is documented to induce verbalized overconfidence (Leng et al., ICLR 2025), do not assume the agent's expressed confidence correlates with its actual accuracy. **Step 4: Set a regression gate and run it on every change.** A score without a threshold is a dashboard, not a control. Define the bar — an absolute floor on trajectory correctness and confident-but-wrong rate, and a "no regression beyond tolerance vs. the last shipped version" clause — set it per critical slice, not only on the aggregate, and wire it so a failing run blocks the release. Trigger it on every prompt edit, model version change, tool schema change, and retrieval change. Because the agent is stochastic, run each case k times and treat the score as a distribution. A 1-of-1 result conflates a real regression with sampling noise. The right gate is statistical, not a single comparison: treat "did the agent regress?" as a hypothesis test over repeated trials, which naturally admits a third verdict — inconclusive — when the runs you have done cannot yet separate a real regression from noise, telling you to sample more rather than ship or block on a coin flip. The tooling for this is still being built across the field; the engineering principle is in place. **Step 5: Keep an independent verifier the agent cannot grade past, and re-validate the set over time.** The grader must be independent of the generator: a separate model, deterministic code, or a held-out check — never the agent grading its own trajectory. The same model and context that produced the trajectory also produce the self-assessment, so the two errors are correlated. A verifier that shares the generator's failure modes confirms the story the generator already told itself. Beyond the grader, the eval set itself decays. The world moves — the input-to-target relationship shifts without any code change, a phenomenon documented in concept drift literature (Gama, Žliobaitė, Bifet, Pechenizkiy and Bouchachia, ACM Computing Surveys, 2014). Tools change, dependencies change, the case set stops resembling live traffic. Refresh it from current production tasks, re-freeze after each update, and never fold current test cases back into training or retrieval. These steps presuppose the general measurement infrastructure described in [AI Evaluation Harness](/ai-evaluation-harness/). That page owns the harness mechanics; this page applies them to agents specifically. Offline evaluation pairs with online liveness and outcome monitoring — see [Production AI: Liveness vs Outcome](/production-ai-liveness-vs-outcome/) — as complements, not alternatives. --- ## Frequently Asked Questions **Is testing an AI agent the same as testing a standard software function?** No. A standard function has deterministic outputs for a given input. An agent produces outputs via a non-deterministic multi-step trajectory — the same input can produce different tool-call sequences across runs, and a correct output on one run does not guarantee a correct trajectory on the next. Testing must cover path validity, not just output value. **Can I use an LLM to grade my agent's outputs?** Yes, with caveats. An LLM-as-judge grader can score semantic correctness faster than human review at scale, and Zheng et al. (NeurIPS 2023) found that a well-calibrated judge achieves greater than 80% agreement with human preferences — about the rate at which humans agree with each other. But documented failure modes make an uncalibrated judge dangerous for agent evaluation: position bias (the judge favors answers by order rather than merit, not by actual quality — documented by Shi et al. across 15 judges and approximately 150,000 evaluation instances, AACL-IJCNLP 2025), verbosity bias (longer, more fluent answers preferred regardless of correctness — which rewards exactly the confident-but-wrong output this page is trying to catch), and self-enhancement bias (a judge favors outputs from its own model family). A grader that is the same model as the agent, or in the same model family, shares failure modes and will miss the agent's systematic blind spots. Validate your grader's false-positive rate on labelled cases before treating its scores as ground truth. Use it for genuinely open-ended output; reach for deterministic tool-call assertions first. **How many test cases do I need to evaluate an AI agent?** There is no universal minimum. The case set must represent the real task distribution — including tail cases and adversarial inputs — not just modal examples. A small set of 20–30 carefully constructed real cases, graded on both trajectory and outcome, gives more production signal than 200 synthetic prompts graded only on final answer correctness. Size matters less than representativeness and the integrity of the freeze. **What is the difference between offline evaluation and production monitoring for an AI agent?** Offline evaluation runs on a frozen case set before release — it is a gate on the path to production. Production monitoring observes the live agent's behavior and outputs in real deployment — it is a detection layer after release. Both are necessary; neither substitutes for the other. A passing eval does not prove the production agent works as inputs shift and dependencies drift. A green health check does not prove agent outcomes are correct. For the online layer, see [Production AI: Liveness vs Outcome](/production-ai-liveness-vs-outcome/). **How is agent evaluation different from the general AI evaluation harness?** A general evaluation harness (see [AI Evaluation Harness](/ai-evaluation-harness/)) measures model output quality — accuracy, relevance, safety — for discrete queries. Agent evaluation adds trajectory grading: the sequence and validity of tool calls, intermediate state, and the path to the answer. The discipline is explicit in well-regarded practitioner guidance: start with the simplest thing that works and add complexity only when it measurably improves outcomes, which applies directly here — reach for a deterministic tool-call assertion before an LLM judge, and add a judge only where the output is genuinely open-ended. The general harness is the foundation; agent evaluation extends it to multi-step, tool-using systems. --- If you cannot yet state your agent's confident-but-wrong rate, that is the measurement layer that is missing — and it is the layer that separates production AI, proven, from a pilot that has only succeeded once. We build agent evaluations as part of [how we work](/how-we-work/) and as the first phase of [AI Rescue](/ai-rescue/) when an agent pilot has stalled for want of one. [Book an AI working session](/book/). ============================================================================== # Deploy-and-Verify for AI: Why 'It Merged' Isn't 'It Shipped' URL: https://newgenapps.com/deploy-and-verify-ai/ ============================================================================== # Deploy-and-Verify for AI: Why "It Merged" Isn't "It Shipped" **In short:** deploy-and-verify is the discipline of proving an AI change is live and correct by checking the *running system* across four layers — source, environment, running process, and real outcome — rather than trusting a merged pull request or a green status light. A merge updates the description of the code; it does not, by itself, update the process serving traffic. The most expensive failures in AI deployment are not the ones that throw an exception. They are the ones where every dashboard is green, the pull request is merged, the pipeline reports success — and the system in production is quietly doing the wrong thing, or the old thing, or nothing at all. This piece is about the discipline that closes that gap: treating evidence from the running system, not the state of the repository, as the only acceptable definition of shipped. It is not a posture we adopted for the current wave of AI work; it is one we developed running production workloads on cloud infrastructure from its earliest days, where being wrong is immediate and costly. The provenance is taken up at the end; the discipline comes first. ## Why is "it merged" not the same as "it shipped"? A merge updates the description of the code; it does not update the running process. A Git repository is a description of intent. It says what the code *should* be. Production is something else entirely: a set of long-lived processes, holding state in memory, reading configuration that was loaded at some point in the past, authenticated with credentials that were valid at some point in the past, against data that is fresh or stale depending on a hundred things the repository cannot see. The two drift apart constantly, and the drift is structural, not accidental. Between "merged to main" and "the new behaviour is what users experience" sits a chain of steps — build, artefact, deploy, restart, config reload, cache invalidation, credential refresh — and every link in that chain can silently fail while reporting success upstream. This is not a fringe risk: in general-software delivery, Google Cloud's DORA *Accelerate State of DevOps Report 2023* found only about a third of teams kept their change-failure rate at or below roughly five per cent, and only 17% recovered from a failed deployment in under an hour (DORA / Google Cloud, *2023 Accelerate State of DevOps Report*, 2023). A deploy completing tells you little about whether the change is correct — and that figure understates the AI case, because AI failures are additionally *silent*. This matters more for AI systems than for ordinary software, for a specific reason. Conventional code tends to fail loudly: a missing function throws, a bad type crashes. An AI component fails *plausibly*. A model served from a stale process returns confident, well-formed answers — they are simply the answers of the old prompt, the old retrieval index, the old tool definitions. A retrieval pipeline pointed at an un-updated vector store returns relevant-looking context that is three weeks out of date. There is no stack trace for "correct shape, wrong substance." The output looks exactly like success. The hidden costs of production ML are well documented: the foundational study by Sculley et al., *Hidden Technical Debt in Machine Learning Systems* (NeurIPS 2015), named unstable data dependencies and "pipeline jungles" among the dominant hidden costs in real ML systems. This is why MLOps cannot borrow its definition of "done" from a CI badge. ## The failure mode, concretely Consider a routine change to a production AI assistant: you tighten a system prompt to stop a category of unwanted responses, add a guardrail, and update a tool schema. The change is reviewed, approved, merged. CI is green. The deploy job exits zero. By every signal a team normally trusts, it is shipped. Now walk the layers down to where the work actually happens, and watch where it can break without anyone noticing: - **Stale process.** The deploy pushed new artefacts to disk, but the long-running inference service was never restarted — or it was restarted by a supervisor that silently fell back to the last-known-good image when the new one failed a health probe. The process in memory is still running last week's prompt. The repository and the disk agree; the live behaviour does not. - **Un-applied config.** The new guardrail lives behind a feature flag or a config file that the running process reads only at boot. The file on disk is correct. The value in memory is not. The guardrail is "deployed" and inert. - **Partial deploy.** Behind a load balancer sit several replicas. Three took the new version; two are wedged on the old one. Roughly forty per cent of traffic gets the un-fixed behaviour, intermittently — the hardest kind of bug to reproduce, because half your probes hit a healthy node. - **Expired credential.** The updated tool needs a token that rotated overnight. The service starts cleanly, serves traffic, and fails *only* on the path that calls the tool — returning a graceful, plausible fallback instead of the intended result. Liveness is perfect. The outcome is wrong. Every one of these passes a naive check. The file is on disk, so `grep` confirms the fix is "there." The service answers a ping, so the health check is green. The PR is merged, so the ticket is closed. And the system is not doing what you shipped it to do. ## What is deploy-and-verify for AI? Deploy-and-verify for AI is proof-based deployment: you stop verifying the *description* and start verifying the *running system*, layer by layer, until you reach the only layer that matters — the real outcome. We verify four: 1. **Source.** The intended change is actually in the branch that production builds from — not in a fork, not behind an unmerged dependency, not reverted by a later commit. This is the layer most teams stop at. It is necessary and badly insufficient. 2. **Environment / disk.** The built artefact on the production host contains the change. The config file, the model weights, the prompt template, the index — the bytes that the process will read — are the new bytes. This catches the partial deploy and the silent rollback. 3. **Running process.** The process *currently serving traffic* is the one built from those artefacts, and it has loaded the new configuration into memory. Process start time after the deploy; config value observed at runtime, not read from the file on disk; model or prompt version reported by the live endpoint itself. This catches the stale process and the un-applied config. 4. **Real outcome.** The system, exercised end to end on a representative input, produces the intended result — the guardrail actually blocks, the tool actually returns, the fresh data actually appears. This is the layer that subsumes all the others, and the only one that proves the change *shipped* rather than merely *deployed*. Each layer proves a different thing, and skipping any one leaves a specific failure undetected: | Layer | What it checks | What it proves | What fails silently if you skip it | |---|---|---|---| | Source | The change is in the branch production builds from — not a fork, not behind an unmerged dependency, not reverted by a later commit | The intent exists in the right place | A "fixed" branch that never reaches the build; a later commit that quietly reverts it | | Environment / disk | The built artefact, config, weights, prompt and index on the production host are the new bytes | The deploy moved the right bytes to the host | A partial deploy or a silent rollback to last-known-good | | Running process | The process *serving traffic* was built from those artefacts and has loaded the new config into memory — checked by start time, runtime config value, and version reported by the live endpoint | The live process is the new one | A stale process on last week's prompt; a config read only at boot and never reloaded | | Real outcome | End to end on a representative input, the system produces the intended result | The change actually shipped | Plausible, well-formed output that is the *old* behaviour, an expired credential silently returning a fallback, or silently stale data | The order matters because each layer localizes failure. If the outcome is wrong but the process is new and config is loaded, the bug is in the change itself, not the deploy. If the process is stale, you have a deployment problem, not a code problem. Walking the layers turns "it's broken somewhere" into a precise diagnosis instead of a guessing game. Two refinements make this hold under pressure. First, **liveness is not outcome**: instrument and alarm on whether the work *completed correctly* — fresh data written, right answer returned — not on whether a port is open. A service can be flawlessly up and producing nothing of value. Second, hold a **data-integrity contract** alongside the deploy check: every source that feeds a decision carries a freshness guarantee, and when a source is unavailable the system says so explicitly rather than serving a confident stale answer. A perfect deploy of code that reads a silently stale feature is still a wrong outcome. ## The trade-off, stated honestly This is not free. Four-layer verification adds steps to every release, and runtime probes are more work to build than a green checkmark. The honest framing is that you are buying down a specific, expensive risk — plausible-looking wrong output in production — and paying for it in deploy-time effort. For a low-stakes internal tool, that trade may not be worth the full ceremony; verifying source and outcome may be enough. For anything that informs a decision, touches money, or reaches a customer, the asymmetry is stark: the verification costs minutes, and the undetected failure costs trust you do not get back. Skipping it does not remove the risk; it relocates the risk to whoever is harmed by the wrong answer, and defers discovery to the worst possible moment. This is also why verification is the differentiator rather than a tax on the unlucky. The deployment gap that strands most GenAI pilots short of measurable impact is, on the evidence, a delivery gap and not a model gap — the structural reasons are taken up in [the 95% pilot statistic, explained](/95-percent-ai-pilots-fail/). What that body of work points to is narrow: the discipline of proving a change works is what separates the systems that hold from the ones that stall. Verification is the practice that lands a deployment on the right side of that line. The other cost is organizational, and it is the one most teams resist. The person who built a change cannot be the final authority that it works. ## Why must the builder not be the verifier? Because self-grading confirms the story you have just told yourself. The engineer who wrote the fix knows what it was *supposed* to do, and that knowledge quietly biases every check they run toward the path they expect to succeed. Independent verification — a separate check, read-only, run against production — is what catches the partial fix, the incomplete deploy, the regression in an adjacent path. This is not a comment on competence; it is a structural property of who can see what. The same logic applies when a model does the grading. A generative system evaluated by another model — "LLM-as-judge" — inherits that judge's own documented, measurable biases (Zheng et al., *Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena*, NeurIPS 2023). A model judge is a screen, not a verdict: anchor it to a human-labelled gold set and never let the system under test grade its own output. The grader needs a grader, or the loop is circular. The mechanics of building one that holds — which biases to control for, and how — are the subject of [the evaluation harness](/ai-evaluation-harness/). In practice this is within reach. You can put the four-layer check into the deploy pipeline as a gate that runs after the rollout and queries the live system — process start time, runtime config value, version reported by the endpoint, a canonical end-to-end probe — and refuses to mark the release done until the evidence is in hand. You can have an automated verifier, separate from whatever generated the change, exercise the running system and report the outcome. Orchestrating the work across models suits this division of labour: one role builds, another independently confirms. Our own engineering depth here is agentic delivery — we build with Claude Code and a library of custom skills, the agent-driven workflow that lets a small senior team ship and verify production AI fast — model-flexible by design and fluent across the stack, running on Amazon Bedrock, SageMaker, and GPU compute accelerated by NVIDIA. The point is not the tooling. It is the principle: capability earns its way to "shipped" through evidence from the running system, gathered by someone — or something — other than its author. ## Where does this deploy-and-verify discipline come from? Our deploy-and-verify posture predates the current wave of AI investment by more than a decade. It was built running production workloads on cloud infrastructure when that infrastructure was itself brand-new. Amazon S3 reached general availability on 14 March 2006; Amazon EC2 entered a limited beta — open only to existing AWS customers — in August 2006, opened to unlimited public beta in August 2007, and did not reach general availability, with a formal service-level agreement, until 23 October 2008 (Timeline of Amazon Web Services; AWS News Blog). We were running production workloads on AWS by 2009, within months of that general-availability milestone, before the platform conventions most teams take for granted had settled. As of 2026, that is more than fifteen years of operating with no margin for "it probably shipped." That timing matters less as a vanity credential than as the explanation for the posture. Deploy-and-verify is not a 2025 talking point invented alongside LLM consulting. It is what you develop when the consequence of a wrong answer in production is immediate and measurable, and when the platform underneath you is too young to have a runbook for every failure. You learn very quickly that a deploy that "completed" is a claim, not a result — and that the only trustworthy signal is evidence pulled from the running system itself. The good news is that the discipline is not a NewGenApps invention, and it does not have to be taken on faith. The gap between "deployed" and "verified" is a named, rubric-scored engineering problem: Breck et al., *The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction* (IEEE Big Data, 2017), defines 28 specific tests and monitoring needs across data, model, infrastructure, and monitoring — including explicit tests for training/serving skew, one of several classes of skew the rubric covers. The deploy gap this article addresses is primarily a runtime and deployment-drift problem: the live process diverging from what was merged. We did not invent the problem. We have been practising the answer to it since the first generation of cloud. ## How do you verify an AI change in production? You verify an AI change in production by querying the live system, after the rollout, for evidence at each layer — and refusing to mark the release done until that evidence is in hand. The procedure: 1. **Confirm source.** Establish that the change is in the exact commit production builds from, and that no later commit reverted it. 2. **Confirm the artefact on the host.** Check that the deployed bytes — config, prompt template, weights, retrieval index — match the new version, catching a partial deploy or a silent rollback. 3. **Confirm the running process.** Read the process start time (it should post-date the deploy), the config value *as held in memory* rather than as written on disk, and the model or prompt version reported by the live endpoint itself. 4. **Exercise the real outcome.** Run a canonical end-to-end probe on a representative input and confirm the intended result — the guardrail blocks, the tool returns, the fresh data appears. 5. **Verify independently.** Have someone other than the change's author run those probes, read-only, against production, so an expected-path bias cannot hide a partial fix. 6. **Alarm on outcome, not liveness.** Instrument whether the work *completed correctly* and keep a freshness guarantee on every source, so a healthy port never masquerades as a correct result. See [why liveness is not outcome](/production-ai-liveness-vs-outcome/). A passing check at any layer is a timestamp, not a guarantee. Production input distributions move, and the relationship a model learned can move under them — covariate shift and concept drift, the canonical formalisation of which appears in Quiñonero-Candela et al., *Dataset Shift in Machine Learning* (MIT Press, 2009). "It worked in March" is not a claim about June, which is why verification is a standing posture run on a cadence and on triggers, not a launch-day gate. If your question is not merely whether an AI system started but [how you know it works in production](/how-do-you-know-an-ai-system-works-in-production/), that re-verification cadence is the answer. This is the same arc as our broader delivery method — source → environment → running process → real outcome, confirmed independently — described in [how we work](/how-we-work/). When a pilot has stalled short of this bar, recovering it to verified production is exactly the scope of [AI Rescue](/ai-rescue/). To put it plainly: a merged pull request is a claim. A green health check is a claim. A slide is a claim. Only the running system, exercised on real input and observed independently, is proof — and in production AI, where failure wears the face of success, proof is the only thing worth trusting. --- If you are putting AI into production and want a second pair of eyes on how you prove a change is live and correct — not merely merged — that is a conversation we are glad to have. Start with [an AI working session](/book/), or read how we approach [AI consulting](/ai-consulting/). *NewGenApps — production AI, proven. Stay a step ahead, always.* ============================================================================== # 'Done' Means the Running System: Why Liveness Is Not Outcome in Production AI URL: https://newgenapps.com/production-ai-liveness-vs-outcome/ ============================================================================== # 'Done' Means the Running System: Why Liveness Is Not Outcome in Production AI > **In short:** A liveness check confirms a process is running. An outcome check confirms the AI system produced a fresh, valid, independently verifiable result — on time. The two are not the same property. A system can pass every liveness probe while silently delivering stale, incorrect, or unvalidated outputs that cost real money, because a wrong answer is shaped exactly like a right one. The discipline that closes the gap is outcome monitoring: measuring what the system actually delivered, not whether it appears to be alive. The most expensive failures in production AI are not the loud ones. A crashed service pages someone within seconds; everyone knows. The failures that cost real money are the quiet ones — the service that is up, answering health checks, consuming compute, and producing nothing of value. This article is about one discipline: instrumenting and alarming on the *outcome* of the work, not on the *liveness* of the process. We arrived at it the hard way — running our own AI systems in production under a strict data-integrity contract, where a silently stale value is treated as a defect rather than tolerated — and have since applied it in client delivery. The engineering principle generalizes cleanly, and it is the one most MLOps and AI monitoring setups get wrong. ## What is the difference between liveness and outcome in production AI? Liveness is whether the *process* is running; outcome is whether the *work the system exists to do* actually got done — correctly and recently. A green health check confirms liveness and says nothing about outcome, and in production AI the two come apart because a wrong answer is shaped exactly like a right one. **Liveness** is the answer to "is the process running?" — a TCP port accepts connections, a container has not exited, an HTTP `/health` endpoint returns 200, CPU is being consumed. These are infrastructure signals. They were designed for an era of stateless web services, where a process that is up is, to a first approximation, a process that is working. **Outcome** is the answer to a different question: "did the work that this system exists to do actually get done, correctly, and recently?" Did the nightly feature pipeline write fresh rows? Did the model produce a prediction for *this* request, or fall back to a default? Did the agent complete the task, or politely give up after one tool call? Did the number on the dashboard come from a computation that ran in the last five minutes, or from a cache populated last Tuesday? The gap between these two questions is where AI systems fail silently. A traditional CRUD service is roughly outcome-faithful: if it is up, it is mostly doing its job, and if it stops doing its job, it usually crashes. AI systems break that assumption along four specific seams: - **Multi-stage pipelines.** Ingest → feature compute → model → post-process → serve. Any single stage can stall while the *endpoint* the health check hits stays up. Liveness is checked at the door; the failure is three rooms back. - **Deliberate fallbacks and defaults.** Resilience patterns — serve-last-known-value, default prediction, retry-with-stale — are added on purpose, and they convert a hard failure into a *silent* one. The fallback is doing exactly what it was built to do; that is the problem. - **Upstream data the system does not own.** Freshness depends on feeds outside the process boundary, so the process can be perfectly healthy while its inputs are hours stale. - **A wrong answer is shaped exactly like a right one.** A model returns a clean, plausible number whether or not the inputs behind it were valid. There is no exception, no stack trace, no non-200. This is the seam with no analogue in a service that throws when it breaks. Liveness was the right proxy for stateless web services, which is exactly why teams trust it reflexively in a setting where its core assumption — up implies working — no longer holds. That is not a tooling gap. It is a category error: instrumenting the process when the thing you care about is the output. ### Liveness signal vs. outcome signal The differentiating column is the fourth — the failure each liveness signal cannot see. The bottom row is the one that has no analogue in ordinary software: there is no liveness signal for "the answer was wrong." | What you are checking | Liveness signal (what teams instrument) | What it answers | The failure it cannot see | Outcome signal (what to instrument instead) | |---|---|---|---|---| | Is the process up? | TCP port accepts connections; container has not exited | The OS still has a process | A process that is up and doing nothing | Did a fresh, valid result appear within the required interval? | | Is the endpoint responding? | `/health` returns 200; p50 latency normal | The web handler answers | A shallow health handler answering while the work loop is wedged | An outcome heartbeat: "I completed real work successfully at time T" | | Is it consuming resources? | CPU/memory busy; GPU utilized | Compute is being spent | Compute spent on a retry storm or a spin-loop producing nothing | Work-completed count rising, not just utilization | | Are inputs flowing? | Upstream connection open; bytes received | A pipe exists | A live pipe carrying stale, last-Tuesday data | Freshness-and-provenance on every decision-grade value (age + source) | | Did the system produce the right answer? | *(no liveness signal exists for this)* | Nothing | The clean, plausible, *wrong* answer | Independent, read-only verification of the live output, separate from the producer | ## Why does a green health check prove nothing? Because a health check tests the *process*, not the *result* — and an AI system has at least three ways to stay green while producing nothing of value. A green check confirms a true, useless sentence: "the service is up." It is silent on whether the work got done. This is not a NewGenApps coinage; it is established monitoring doctrine. The standard guidance in Google's *Site Reliability Engineering* (Beyer et al., O'Reilly, 2016) is that "it's better to spend much more effort on catching symptoms than causes," and that an alert should fire on a condition that is "actively or imminently user-visible." A liveness probe is a *cause* check — a process detail. "The expected result did not appear, fresh and correct, on time" is the *symptom*. Even the defining infrastructure tool admits its own narrow remit: the Kubernetes documentation describes a liveness probe as something that "could catch a deadlock, where an application is running, but unable to make progress," whose only action on failure is to restart the container — never to ask whether the last answer was correct (Kubernetes, *Configure Liveness, Readiness and Startup Probes*). Consider a non-proprietary but realistic example: a system that ingests a stream of input signals, computes derived features on a schedule, runs a model against those features, and surfaces a result to downstream consumers — an operator dashboard, an automated action, another service. Three distinct outage modes can occur while every health check stays green. **Running but hung.** The worker process is alive — the supervisor sees it, the port is open — but the main loop is wedged on a lock, a never-returning network call, or a deadlocked dependency. It will never crash and never restart, because nothing has technically died. Liveness says healthy; the work queue silently backs up behind a process that is "up" and doing nothing. This is precisely the case the liveness probe is *advertised* to catch and frequently does not, because a shallow `/health` handler answers while the real work loop is wedged. **Serving a stale cache.** A resilience pattern most teams add deliberately: if the fresh computation is unavailable, fall back to the last known value rather than error out. This is reasonable for a one-off blip and catastrophic as a steady state. The upstream feed dies at 02:00; the system serves yesterday's cached features all day; the model dutifully produces predictions on stale inputs; the dashboard shows live-looking numbers that are hours old. Every component reports success. The output is wrong but plausible, which is worse than no output — a blank panel makes someone investigate, a plausible stale number makes someone act. A stale-cache fallback is not a bug; it is a feature failing in a way that looks safe. **The silently latched gate.** A safety check or circuit breaker trips — a validation gate, a rate limiter, a "pause if anomalous" guard — and then never resets, or resets only on a code path that no longer runs. The unit is alive, healthy, and has quietly done nothing for six hours. Because the gate's whole job is to *prevent* output, its closed state is indistinguishable from "there was simply no work to do." No alarm fires, because nothing failed; the work just stopped. In all three cases the same words get said: "the service is up." It is. That sentence is true and useless. Uptime was never the thing you cared about. The deeper reason this class of failure exists is structural: silent degradation is a defining, named property of machine-learning systems. Sculley et al., in *Hidden Technical Debt in Machine Learning Systems* (NeurIPS, 2015), name unstable data dependencies, hidden feedback loops, and "undeclared consumers" as dominant hidden costs in real ML systems — the reasons a model can serve cleanly and still be wrong. The most dangerous output is the plausible wrong one, and it is invisible to error-rate monitoring *by construction*: error rate counts thrown exceptions, and this failure throws nothing. There is empirical weight behind the size of this gap. A 2026 peer-reviewed study of microservice monitoring (Sabetta et al., *Vitality Assurance in Microservice Architectures*, Frontiers in Computer Science, June 2026) found that passive infrastructure checks — services returning HTTP 200 — detected only 58.3% of injected production failures, while outcome-level synthetic journey testing that asked "did the user workflow actually succeed?" detected all of them. The authors call the undetected cases "grey failures": services that are technically alive but experientially dead. The terminology maps precisely onto the argument here. The honest caveat is that liveness probes still catch most events — in one 2025 study of real production LLM inference incidents, the majority were auto-detected by health probes before customer reports (arXiv:2511.07424, October 2025). The point is not that liveness is useless; it is that the residual class — system up, output wrong, no alarm — is structurally invisible to infrastructure monitoring and is the class that costs the most. ## How do you monitor an AI system's outcome, not just its liveness? Instrument the work, not the process, in three moves. The fix is a discipline, not a tool, and it follows from one rule: **define "done" as the running system producing correct, fresh output — and instrument that definition directly.** **1. Emit an outcome heartbeat, not a process heartbeat.** Instead of (or alongside) "I am alive," each unit should assert "I completed my work successfully at time T." The monitor's job is then to check that T is recent enough — that a fresh, valid result appeared inside the interval the business actually requires. A hung worker stops emitting; a stale-cache fallback emits a result *flagged as a fallback*, which the monitor counts as a miss; a latched gate stops producing completions. The wedge, the staleness, and the silent gate all become visible as the same observable symptom: *the expected outcome did not appear on time.* This is an AI-shaped service-level objective. Instead of "99.9% of requests return 200," the objective is "a fresh, valid result appears within N minutes," and you alert on the burn of *that* budget — the same error-budget burn-rate discipline set out in Google's *Site Reliability Workbook* (Beyer et al., O'Reilly, 2018), which is explicit that good alerts come from SLOs that "measure the reliability of your platform, as experienced by your customers." **2. Carry a freshness contract on every value that informs a decision.** A number that reaches a human or triggers an action must travel with its provenance and its age. The reader's contract is not "is this value greater than zero?" but "was this value produced recently enough to be trusted, by a source that was actually live?" A plausible non-zero number is precisely the failure a presence check waves through. When a source is unavailable, the honest behavior is to say so explicitly — surface "data unavailable" — rather than serve a confident wrong answer dressed as a live one. (The mechanics of that contract are a topic of their own; see our [data integrity contract](/ai-data-integrity-contract/).) **3. Verify the outcome independently of the thing that produced it.** The component that does the work is the worst-placed judge of whether the work was done; self-reporting confirms the story the system just told itself — a hung loop, a stale-cache fallback, and a latched gate all *report success* from inside. A separate, read-only check — looking at the actual output, on the live system — catches the partial completion, the stale write, and the latched gate that the producer cheerfully reports as success. Builder is not verifier. The model-orchestration layer is no exception: the conductor coordinating the models still needs an outside observer confirming that fresh, correct output actually landed, not merely that a call returned. ## What outcome monitoring costs you This discipline is not free, and pretending otherwise would be exactly the kind of overclaim this register avoids. Outcome instrumentation requires you to *define the outcome*, which is harder than exposing a port. You must state, per unit, what "done" means, what "fresh enough" means, and what the valid range of a correct result looks like — and those definitions are domain knowledge, not boilerplate. They will be wrong at first and need tuning. It also costs an alerting budget. Outcome alarms are more sensitive than liveness pings by design, so a careless rollout produces noise, and noise trains operators to ignore alarms — which is worse than having none. The thresholds (how stale is too stale, how many fallbacks before it's an incident) have to be set against the real business tolerance, not pulled from a default. And there is a build-versus-buy reality: most off-the-shelf monitoring ships liveness and resource metrics out of the box and leaves outcome instrumentation as custom work you have to design. The effort is the point — it is precisely the work that the easy path skips. ## Where outcome monitoring sits next to evaluation Outcome monitoring is the online, post-release half of a pair. An evaluation harness is offline and pre-release: it gates *what you ship*, on a frozen set, before traffic. Outcome monitoring watches *what you shipped*, on real unchosen traffic, now. This page owns the second. The reason both are needed is that a passing eval is a timestamp, not a guarantee — production distributions move (concept drift, the formally studied phenomenon in which live data wanders away from the data a model was validated against, degrading accuracy without any code change; Gama et al., *A Survey on Concept Drift Adaptation*, ACM Computing Surveys, 2014), which is why online outcome monitoring is a continuing obligation rather than a launch-day check. For the offline half — what to measure before release — see our work on the [evaluation harness](/ai-evaluation-harness/). For the discipline of proving a change is actually running in production before you trust it, see [deploy and verify](/deploy-and-verify-ai/), where "merged" is not the same as "shipped." ## How to apply it You do not need a platform migration to start. A pragmatic sequence: - **Write down the outcome for each unit.** One sentence: "this unit is doing its job if [a fresh, valid X] appears [at least every N minutes]." If you cannot write that sentence, you have found the first problem. - **Instrument that sentence.** Have each unit emit a success-with-timestamp signal on real completion — not on loop entry, not on a healthy port — and alarm when the latest success ages past N. - **Attach freshness to every decision-grade value** and make readers check age and provenance, not just presence. Make "data unavailable" a first-class, visible state. - **Add one independent check** that reads the live output and confirms it, separate from the code that produces it. Start with your highest-stakes output and expand. - **Tune against business tolerance,** then treat every silent-failure incident as a missing outcome alarm and close that gap. Over time your alarms converge on the things that actually matter. The shift is small to state and demanding to live by: stop asking whether the system is up, and start asking whether it did its job. "The service is running" is the beginning of the investigation, not the end of it. If you want a sharper test of the whole question — [how do you know your AI system is actually working?](/how-do-you-know-your-ai-works/) — the answer always reduces to the outcome, observed independently. In production AI, the only honest definition of done is the running system producing the right result, recently — and proven, not assumed. --- If your AI is in production and your monitoring still answers "is it up?" rather than "did it work?", that is a gap worth closing deliberately. We work through exactly this kind of instrumentation — outcome heartbeats, freshness contracts, independent verification — in a focused [AI working session](/book/), or as part of longer-term [AI consulting](/ai-consulting/). We don't just say it; we show it, against the running system. *NewGenApps — production AI, proven. Stay a step ahead, always.* ============================================================================== # The Data-Integrity Contract: No Synthetic Data in Anything That Decides URL: https://newgenapps.com/ai-data-integrity-contract/ ============================================================================== # The Data-Integrity Contract: No Synthetic Data in Anything That Decides Most AI systems do not fail because the model was wrong. They fail because the data behind the model was wrong in a way nobody noticed — a placeholder that survived into production, a cache that quietly went cold, a feed that stopped updating while the dashboard kept rendering yesterday's number as though it were live. The model did exactly what it was asked. It was asked over bad inputs. This piece argues for a specific, enforceable discipline — a data-integrity contract — as the foundation of production AI reliability, and walks through how stale-as-live failures actually happen and how to engineer them out. *Last reviewed: June 2026.* ## What is a data-integrity contract? > **In short:** A data-integrity contract is an enforced engineering guarantee that no synthetic, mocked, or silently-stale data is allowed into anything that informs a decision or reaches a user. Every decision-facing value carries explicit provenance (where it came from) and a freshness budget (the maximum age this decision tolerates); a value that cannot meet its budget is treated as no value at all, and the system fails closed — returning an honest "data unavailable" rather than a confident wrong answer. It is worth being precise about two words, because they are what make the definition correct rather than generic. The first is **"contract," not "check."** A check is a line of code someone might add. A contract is a property the system must hold before any decision logic is allowed to run — stated, enforced, and verified, the same way a type signature or a service-level objective is enforced. It is a precondition, not a courtesy. The second is **"freshness budget," not "fresh."** Freshness is not a binary, and not a global setting. It is a per-source, documented tolerance window — sub-second for a fast-moving signal, hours for something that genuinely changes slowly. The discipline is that the tolerance is stated and enforced per source, not assumed. The useful mental model is that a data-integrity contract is a **service-level objective for data, enforced at read time.** Site reliability engineering already has the vocabulary: Google's *Site Reliability Workbook* defines a data-freshness SLO — "the oldest data is no older than Y minutes" — as a measurable, alertable target distinct from a data-correctness SLO (Google, *Site Reliability Workbook: Data Processing Pipelines*). The contract takes that pipeline-level discipline and moves it to the decision boundary: not "did the pipeline meet its freshness SLO somewhere upstream," but "is this specific value, at the moment I am about to decide on it, within its budget and provenanced — and if not, do I refuse to act?" ## Why is data — not the model — the leading cause of AI pilot failure? Because a correct model reasoning over a stale, mocked, or synthetic input produces a wrong answer with full confidence and no error flag. It is the worst possible failure, because it is indistinguishable from a right one until someone acts on it. The survey evidence makes the scale concrete. In Informatica's 2025 survey of 600 chief data officers and chief data and analytics officers across the U.S., Europe, and Asia, data quality, completeness, and readiness was the most-cited obstacle to moving generative AI from pilot to production — named by 43% of data leaders, tied with technology adoption challenges — while two-thirds (67%) said they had not been able to move even half of their generative AI pilots into production (Informatica, *CDO Insights 2025*, January 2025). The number worth holding is that 67% gap: most pilots are not reaching production, and the most-cited reason is the state of the data, not the sophistication of the model. That is not a one-survey artifact. The most rigorous academic backing is peer-reviewed: in a study of AI practitioners building high-stakes systems, 92% reported experiencing one or more "data cascades" — compounding, downstream failures that originate in data — and 45.3% reported two or more (Sambasivan et al., *"Everyone wants to do the model work, not the data work": Data Cascades in High-Stakes AI*, CHI 2021). The failure surface that practitioners actually hit, repeatedly, is the data work that the field systematically under-invests in. To be accurate about scope: the 43% figure is tied with technology in the Informatica data, not the sole named cause, and not every AI deployment depends on the kind of freshness-and-provenance contract argued for here — a low-risk, static-analytics use case can succeed on imperfect data. The contract earns its keep most strongly in real-time, decision-facing systems, where recency is load-bearing and a wrong answer is acted on before anyone notices it was wrong. That is the scope this page is about. This is also the natural place to state a pairing that is easy to confuse. The **evaluation harness is the model-error gate**: it runs offline, before release, on a frozen held-out set, and answers "did this version of the model get worse?" The **data-integrity contract is the data-error gate**: it runs online, at decision time, on live inputs, and answers "is the data this decision is about to consume real, provenanced, and fresh enough?" The two gates fail independently, and neither covers the other — a flawless eval cannot save you from a six-hour-old input, and a perfect data contract cannot save you from a model that silently regressed. A production AI system that decides needs both, because a confident wrong answer can originate on either side of the model boundary. See the [evaluation harness](/ai-evaluation-harness/) for the model-error side; this page is the data-error side. Both are part of [why AI pilots fail](/why-ai-pilots-fail/). ## Why data integrity is the real reliability problem When teams talk about AI reliability they usually mean the model: hallucination, accuracy, drift, evaluation. That is a real surface, but it is the second-order one. The first-order surface is the data the model and the surrounding system consume. A correct model reasoning over a stale, mocked, or synthetic input produces a wrong answer with full confidence and no error flag — which is the worst possible failure, because it is indistinguishable from a right one until someone acts on it. This is harder than ordinary data quality because of where AI sits in the stack. A traditional report that draws on stale data looks stale — the timestamp is on the page, a human reads it, judgement intervenes. An AI system collapses that distance. It ingests the input, reasons over it, and emits a decision or a number in a single motion, often with no human between the bad input and the consequence. The model also launders the staleness: it wraps a six-hour-old value in fluent, present-tense, authoritative prose. The very competence that makes the system useful is what makes a bad input dangerous, because it removes the human cue — visible staleness, hedged language — that would otherwise have caught it. So the contract is not a data-hygiene nicety bolted on at the end. It is a property the system must hold before any decision logic is allowed to run. Stated plainly: **no synthetic, mocked, or silently-stale data in anything that informs a decision or reaches a user.** Every data source carries a freshness guarantee, and when a source cannot meet it, the system says so explicitly instead of serving a confident wrong answer. There is a deeper reason existence checks fail, and it is worth naming because it is the technical spine of this whole argument. Data quality is not one axis. The canonical framework — Wang and Strong's *Beyond Accuracy: What Data Quality Means to Data Consumers* (Journal of Management Information Systems, 1996) — established that data quality decomposes into multiple dimensions, and that practitioners systematically over-index on *accuracy* while data consumers care just as much about *timeliness*: quality relative to the task at hand. Timeliness is a named, distinct dimension. A value can be perfectly accurate and worthless because it is stale. An existence check measures completeness, and nothing about timeliness. It is testing a different quality dimension from the one that is load-bearing for the decision. ## How stale-as-live failures actually happen The failure mode is rarely a dramatic outage. It is a value that is structurally present but semantically dead. Consider a generic, non-proprietary example: a service writes the current value of some external signal to a cache every few seconds, and a downstream component reads that cache to make a call. One day the writer dies — a network blip, an expired credential, an upstream rate-limit, a deploy that didn't restart the right process. The cache still holds the last value it ever received. The reader is checking only whether a value exists and is greater than zero. It is. So the reader proceeds, treating a value frozen at the moment the writer died as the live state of the world. Nothing alarms, because every component is technically "up." The dashboard is green. The number is wrong, and it will stay wrong, confidently, until someone independently notices the world has moved and the value hasn't. Stale data is specifically the dangerous failure mode because it is the one that looks healthy. Unlike a missing or corrupted record, a stale value renders normally — dashboards draw, jobs complete, no errors fire — which is why it is documented as a silent failure that standard error monitoring structurally cannot see (IBM, *What Is Stale Data?*). Process monitoring answers "did the job run?"; freshness monitoring answers "did fresh data actually arrive?" The gap between those two questions is where stale-as-live incidents live. There are three recurring shapes here, and they compound: - **The mock that outlived the demo.** A placeholder or fixture wired in during development to get the pipeline flowing, never removed, now silently feeding a production decision. It survives because it works — it returns plausible data on every call, so nothing ever fails loudly enough to get it noticed. - **The cache with no expiry semantics.** A value cached for performance, read with an existence check (`value != null`, `value > 0`) rather than a freshness check. Existence is not liveness. A value can exist and be hours stale; the read path cannot tell the difference unless freshness is part of the contract. - **The liveness/outcome confusion.** "The service is up" answers a different question from "fresh, correct data was just produced." A health check that pings a process tells you the process is running, not that it is doing its job. You can have a perfectly live service emitting perfectly dead data. This is the [liveness vs. outcome](/production-ai-liveness-vs-outcome/) distinction, treated in full on its own page; the data-integrity contract is one reason a liveness probe is not an outcome signal. In each case the root cause is the same: the system was built to check *presence*, when the property that matters is *recency and provenance*. Presence is cheap to verify and reassuring; recency is the thing that was actually load-bearing. ## Existence check vs. freshness check: what each one tests and what it misses This is the distinction at the centre of every stale-as-live failure. The methods below are not wrong — they are testing the wrong dimension. The column that matters most is the failure each method silently allows. | Read pattern | What it actually verifies | The failure it silently allows | The honest alternative | |---|---|---|---| | `value != null` (existence) | A value is present in the read path | A mock or fixture that returns plausible data on every call; a cache frozen the instant the writer died | Require provenance: which source produced this, and is it the real one? | | `value > 0` (sanity) | A value is present and in a plausible range | A stale-but-plausible value — last good price, yesterday's count — that is in-range and completely wrong now | Attach a timestamp and check it against a per-source freshness budget | | "The service is up" (liveness ping) | A process is running | A live service emitting dead data: green health check, stale output | Alarm on "fresh, correct data was produced," not on "the process responded" | | "The job ran" (orchestration success) | A pipeline step completed | A job that finished and wrote a stale or empty result; an upstream that silently stopped feeding it | A data-freshness SLO — oldest data no older than Y — checked at read time, not just at job exit | | Freshness + provenance check (the contract) | Value, plus origin, plus age within the decision's tolerance window | This is the point: it is built to catch exactly the stale-as-live and surviving-mock failures the rows above pass | When the budget cannot be met, fail closed — return an explicit "data unavailable" and decline to decide | Two of those rows are catching genuinely different problems, and conflating them is a common mistake. **Freshness answers "how old?"; provenance answers "from where?"** A value can be perfectly fresh — written one second ago — and still be wrong because it came from a mock that is updating happily on a timer. Freshness checks catch the dead-writer and cold-cache cases; provenance checks catch the surviving-mock and wrong-source cases. The contract needs both stamps, age and origin, because they fail independently. ## The principle: freshness guarantees and fail-closed by default The fix has two parts, and both are engineering commitments rather than aspirations. First, **every market-facing or decision-facing value carries explicit provenance and a freshness budget.** A read is not "is there a value?" but "is there a value, where did it come from, and was it produced within the window this decision tolerates?" That window is a deliberate, documented choice per data source — sub-second for a fast-moving signal, minutes or hours for something that genuinely changes slowly. The point is that the tolerance is *stated and checked*, not assumed. A value that exists but falls outside its freshness budget is treated as no value at all. Second, **fail closed.** When a source cannot meet its guarantee, the system returns an honest, explicit "data unavailable" — and the decision logic declines to act on absent data — rather than substituting a stale value, a default, or a synthetic stand-in. This is not a contrarian instinct. It is the data-plane instance of a fifty-year-old secure-design principle: fail-safe defaults, from Saltzer and Schroeder's *The Protection of Information in Computer Systems* (1975), which holds that when a mechanism fails or cannot reach a decision, it should fail to the safe, deny state rather than the permissive one. Applied to data: when a source cannot prove freshness and provenance, the safe default is to deny the value, not to substitute a stand-in. Most systems are built the other way — to keep producing output under degradation, because an empty screen feels like a failure. But in anything that decides, a confident wrong answer is strictly worse than a visible gap. "I don't currently have a reliable value for this" is a safe state. A six-hour-old number presented as live is not. This is also where independent verification matters. A multi-component AI system can only be trusted with a number once the data plane beneath it is provable — the model is only ever as honest as its inputs. We are AI-led and model-flexible; orchestrating the components is the mechanism, but the contract is what makes the result defensible. It is deliberately model-neutral, because the failure mode lives in the data path rather than in any one model. (For how that verification fits a delivery model, see [how we work](/how-we-work/).) ## The trade-offs, stated honestly This discipline has real costs, and a contract that hid them would not be worth signing. Fail-closed means the system will sometimes refuse to answer when a looser design would have guessed — and a guess is right more often than not, which makes the discipline feel pessimistic. That is exactly the trap: a design calibrated by the common case is blind to the expensive tail. The cases where the guess is wrong are exactly the cases that matter, and you cannot tell them apart at the moment of the guess. You are trading a small amount of availability for a large amount of trustworthiness, and in any decision system that is the correct trade. Fail-closed also produces visible gaps, and a user can read a visible gap as the system being broken — which can raise perceived unreliability even when the system is behaving exactly as designed. That is a real cost, and the answer is not to hide the gap but to make the "data unavailable" state legible: say what is unavailable and why, so the honest state reads as honesty rather than breakage. Freshness budgets add engineering surface: every source needs a defined tolerance, every write needs to stamp provenance and time, every read needs to enforce the check. This is more code and more discipline than an existence check. It is not free. But it is the cheapest place to pay, because the alternative — discovering a stale-as-live failure after a decision has been made on it — is paid in incidents and in eroded confidence, which is far more expensive and arrives without warning. ## How to apply it You do not need to rebuild a system to adopt the contract. Start where decisions are made and work outward. - **Inventory the decision-facing data paths.** For each value that informs a decision or reaches a user, ask one question: if its upstream source died right now, how would this code know? If the answer is "it wouldn't," you have found a stale-as-live risk. - **Replace existence checks with freshness checks.** Anywhere a read tests only for presence, add provenance and a timestamp, and define the tolerance window explicitly. Make the window a reviewed decision, not an accident of caching. - **Audit for surviving mocks and fixtures.** Search the production path for placeholders, sample data, and development stand-ins. Anything that returns plausible data without touching a real source is a candidate to fail loudly or be removed. - **Instrument the outcome, not the ping.** Alarm on "fresh, correct data was produced," not on "the process is running." A green health check that coexists with dead output is the signal you most need to catch. - **Make "data unavailable" a first-class state, and verify it under real failure.** Build the explicit, honest unavailable path deliberately — then prove it triggers under a real, induced source failure, against the running system, checked by someone other than the author. The fail-closed branch is the one path that only fires under failure, so it is the least exercised in normal operation and the most likely to be quietly broken. A fail-closed path you have not watched fail closed is an assumption, not a guarantee. This is what production AI, proven actually means in practice: not a claim that the system is safe, but independent evidence that the safe path fires when it has to. Every failure in the table above fails the same way: it verifies presence — a value exists, a process responds, a job finished — when the property that is load-bearing for a decision is recency-and-provenance: this specific value, fresh enough for this decision, from the source it claims. A system that checks whether a value exists, but not whether it is still true, is not "mostly reliable" — it is confidently wrong on exactly the inputs that matter, and it cannot tell you which ones those are. That is why the data-integrity contract is the gate to production: it is the discipline that turns "the number is there" into "the number is true, and the system would have said so if it weren't." --- If you are putting AI behind real decisions and want a clear-eyed read on where stale-as-live and synthetic-data risks hide in your own stack, we can work through it directly in an [AI working session](/book/), or as part of longer-term [AI consulting](/ai-consulting/). The proof is the fail-closed path firing on demand, watched by someone other than the author — evidence, not assurance. *NewGenApps — production AI, proven. Stay a step ahead, always.* ============================================================================== # How to Choose an AI Consulting / Delivery Partner (2026 Buyer Checklist) | NewGenApps URL: https://newgenapps.com/how-to-choose-an-ai-partner/ ============================================================================== # How to Choose an AI Consulting or Delivery Partner **In short:** vetting an AI partner comes down to three questions. Can they prove production delivery, not just pilots? Are senior practitioners doing the actual work? And can they show you *how* they verified the results — not just what the results were? Those map to three tests: **production-proof** (a real PoC-to-production conversion rate, stated as a number), **senior delivery** (the people who scope the work are the people who build it), and **independent verification** (someone other than the builder confirms the system runs on real data). A firm that cannot name clients under NDA is not automatically a red flag; a firm that cannot show you its evaluation method is. This page gives you the questions to ask, the signals to read, and a scoring table to apply to any partner on your shortlist. Most AI initiatives do not fail on the idea or the model. They fail in the gap between a convincing demo and a system that runs in production, on real data, under load. The vetting questions below are designed to surface, early, whether a prospective partner can cross that gap — before you have spent a quarter and a budget line finding out. ## What should you ask an AI consulting firm before you sign anything? The questions that matter most fall into three categories: production proof, delivery accountability, and independent verification. Everything else — frameworks, certifications, methodology slides — is downstream of those three answers. 1. **Production-proof.** Evidence that their work reaches and survives production, stated as specifics, not adjectives — a PoC-to-production conversion rate as a number, a live system you can have described, a measured outcome. 2. **Senior delivery.** The people who scope the work are the people who build it. No junior pyramid learning AI on your budget; no team you meet in the pitch and never see again. 3. **Independent verification.** Someone other than the builder confirms the system runs, on real data — an evaluation harness, a read-only check the builder does not control, instrumentation of the business outcome rather than a server ping. These are the tests. The vetting table further down scores each partner against them, and every question on this page ladders back to one of the three. The 2026 market makes this screening urgent. The failures are widespread, and the published record attributes them to delivery, not model quality: data quality, evaluation, integration, and ownership. Across 600 chief data officers, **43% cite data quality, completeness and readiness as the top obstacle keeping GenAI initiatives from the finish line** (Informatica, *CDO Insights 2025*, January 2025, n=600). That is the single most-cited cause — which is exactly why the questions that predict success are delivery questions, not model questions. These map directly to how we run engagements; the method behind them is on our [how we work](/how-we-work/) page. ## What are the green flags and red flags when evaluating an AI partner? The clearest way to separate a production-ready partner from a pilot-stage vendor is to ask for evidence of each, then score what you get back. A red flag is rarely a single bad answer — it is a pattern of leading with the model or the platform instead of the outcome, and being unable to show, in concrete terms, that the work reaches and survives production. The table below contrasts the signals that separate a delivery partner from a demo shop. Each row maps to one of the three tests, so you can see *why* it is on the list. | Signal | Green flag (a partner who ships) | Red flag (a partner who stalls) | Which test | |---|---|---|---| | Production track record | States a PoC-to-production conversion rate as a number and a timeline; can narrate the unglamorous middle | "Most of our clients are happy"; pivots to logos when asked for a rate | Production-proof | | Who does the work | The senior people who scope it are the people who build it | Partner sells, then juniors deliver; the team you met is not the team you get | Senior delivery | | Verification method | A documented evaluation harness; an independent, read-only check confirms the result on the live system | "We test it ourselves"; self-graded; no separate verification step | Independent verification | | Data discipline | A data-integrity contract; says plainly when a source is stale or missing | Built on a hand-cleaned demo extract; no answer for stale or malformed data | Independent verification | | Definition of "done" | A verified outcome on the running system | "It merged" / "the demo worked" / a green status light | Production-proof | | Reference posture | Offers method-proof when names are under NDA — harness, verification step, anonymized-but-specific account | "All references are NDA'd" with nothing else offered | Independent verification | | Scope honesty | Will tell you, with evidence, when a use case should stop | Says yes to everything; claims capability across every AI use case | Senior delivery | | Ownership | Transfers capability, runbooks, and operating discipline to your team | Designs for permanent dependence | Senior delivery | | Model stance | Model-flexible; uses the right model and keeps it replaceable | Locked to one platform or one model as the answer to everything | Production-proof | | Pricing structure | Fixed or outcome-based, backed by a verification method | Open-ended time-and-materials with no reliability gate | Production-proof | ## Is "all our clients are under NDA" a red flag when choosing an AI partner? Not by itself — but it becomes a red flag the moment it is the only answer a firm gives to the question "how do I know this worked?" The tension is worth stating accurately, because the published checklists are more precise than the loose version suggests. The 2026 buyer guides do not, as a class, flag "the partner won't name clients" as the warning sign. What they flag is the inability to produce *any* evidence of a live production system. Rocket Farm Studios' *Complete Buyer's Guide* names as a warning sign a partner who "can't connect you with a past client whose AI system is live and generating business value right now" (Rocket Farm Studios, *How to Choose an AI Development Agency*, March 2026). A May 2026 practitioner guide is explicit that the line is opacity, not anonymization: "some level of client anonymization is normal, even expected," but an agency with no live demos, no named clients, no reference call, and no working tool you can try "is an agency with no verifiable track record" (Kalvium Labs AI, *How to Vet an AI Agency*, May 2026). The honest reading is sharper than the loose one: **the red flag is no proof, not no names.** Many serious enterprise partners — operating for banks, insurers, or regulated-market operators under NDA — genuinely cannot name a client. That constraint is real, and it is not, by itself, a warning sign. The resolution is to **shift the proof from names to method.** A partner who cannot name the client can still show you: - the **evaluation harness** they used to decide the system was good enough to ship; - the **independent verification step** that confirmed it ran on real data; - the **data-integrity contract** that governs what the system is allowed to treat as true; and - an **anonymized-but-specific account** — sector, scale, geography — of what they built, what broke, and how they proved the fix. Demand the proof. Accept that for an NDA-constrained partner it may arrive as method-proof rather than logo-proof — and reject a partner who can offer neither. This is the *more* rigorous test, not the lenient one: a named logo proves a contract existed; a verification method proves a system worked. ## Why does "demand the proof" beat "trust the label"? Because a label is not evidence, and the cost of trusting one is borne by the buyer — and the 2025–2026 record makes that concrete from both directions. On the capability side: Builder.ai, a London startup once valued near $1.5 billion with high-profile backers, entered U.S. bankruptcy in 2025 after it was reported to have been marketed as more autonomous than it was, with much of the app-building work done by human engineers (Rest of World, 2025; TechCrunch, May 2025). The exact "AI versus humans" share is contested — The Pragmatic Engineer pushed back on the viral "faked AI" framing — so the precise, undisputed facts are the 2025 insolvency and the AI-washing pattern. The buyer-side lesson is not contested: buyers paid for a capability they could not independently verify. On the output side: in October 2025, Deloitte's Australian member firm agreed to a partial refund on a roughly A$440,000 (about US$290,000) government report after AI-generated citations to non-existent sources were found in it (Fortune, October 2025). The brand was impeccable; the deliverable was not verified. Same failure, opposite end — in one case the capability was unverified, in the other the output was, and in both a trusted label substituted for an inspectable method. The durable protection in vendor selection is not a brand you trust but a method you can inspect — an evaluation harness, an independent verification step, a human-checkable trail. That is also why the three tests are model-agnostic: they apply equally to a global firm, a boutique, or an in-house build. (For the model comparison itself, see [boutique vs. Big-4 AI consulting](/boutique-vs-big-4-ai-consulting/).) ## How do you check whether an AI partner has actually delivered in production? Ask for three things: a specific system description, a measured outcome, and the verification method — if any one of the three is missing, ask why. Production capability shows up as specifics, not adjectives. Run this checklist in a single discovery call: 1. **Ask them to describe one live system** — not a demo, a deployed integration — and what problem it solves. A partner who ships can walk you through the part the pitch skips: how the pilot's hand-cleaned data met the production feed, what broke at the integration boundary, how quality was measured before and after. A demo shop's story ends at the demo. 2. **Ask what the measured outcome was and how it was measured** — internal audit, A/B, integration-test coverage — not "the client said so." 3. **Ask who on the delivery team ran the verification, and whether it was the same team that built it.** The person who builds a change should not get the final word on whether it works. A separate, read-only check on the real system catches the partial fix and the incomplete deploy that self-grading misses. 4. **Ask what happened when something went wrong.** Production partners have post-mortems; pilot firms do not. "The service is up" is an infrastructure signal; "fresh, correct output appeared on real data" is a business one — a production-grade partner instruments the outcome, not just a ping. 5. **Ask about data integrity:** how is the training or fine-tuning data audited, and what does the contract say about data lineage? This is non-optional precisely because data quality is the most-cited cause of pilot failure (Informatica, *CDO Insights 2025*, January 2025). If the answers are concrete, you are likely looking at a partner who ships. If they are adjectives, you are looking at a partner who demos. The same root causes that sink in-house pilots — data, evaluation, integration, ownership — are exactly what a recovery engagement like [AI Rescue](/ai-rescue/) is built to fix when a pilot has already stalled. ## What questions should you ask about team structure and seniority? The single most reliable structural question is: who specifically will work on my engagement, and what have they individually shipped in production? Ask these in a discovery call, before you sign: 1. **Who is the named practitioner leading this engagement, and what have they built and deployed?** 2. **What is the ratio of senior to junior practitioners on a typical engagement of this size?** 3. **At what point in the project does a senior practitioner hand off to a junior one — and who makes that call?** 4. **What is your escalation path if the senior practitioner becomes unavailable mid-engagement?** 5. **Can I speak with the delivery lead — not the sales lead — before I sign?** There are no universally correct answers, but there are coherent ones and incoherent ones. A firm that cannot give you a named delivery practitioner at proposal stage has already answered the first question. The reason this test is more than a slogan is engineering reality: the work that turns a demo into a system — failure paths, distribution shift, reasoning about a probabilistic system's reliability as a distribution rather than a value, integration with identity and data access and observability — is senior work, and it is most of the work. A junior pyramid under-reaches production for the same reason a cold in-house build does: the production-delivery skill is the bottleneck, and it does not scale by adding headcount. ## Should you build an in-house AI team or hire a partner? That is a separate decision from vetting, with its own economics — and it is worth making deliberately, because the right answer depends on your team's current capability and your intended time-to-production horizon. The most reliable order of operations is **partner → transfer → build**: bring in a senior partner to cross the production gap on your first real system, have them transfer the capability and operating discipline to your team as they go, then have your in-house team take it from there — from working software, not a standing start. We treat the trade-offs, costs, and when each model makes sense in full in [should you build an in-house AI team or hire a partner?](/build-vs-buy-ai-team/). Whichever path you choose, the three tests on this page — production-proof, senior delivery, independent verification — are how you vet it. ## How NewGenApps maps to this checklist These three tests are the standard we hold our own work to, so it is fair to apply them to us before you apply them to anyone else. - **Production-proof.** We treat evidence from the running system — not a merged pull request, not a green status light — as the only acceptable definition of done. More than fifteen years of cross-domain delivery, on a public dated record, is the experience behind that, including adopting cloud infrastructure early — running our own site on AWS back in 2009. - **Senior delivery.** A small, senior team does the work end to end. The engineer who scopes your build is the one who ships it and verifies it in production — no junior pyramid, no bait-and-switch. - **Independent verification.** A separate, read-only check confirms the result on the real system, under a data-integrity contract that forbids synthetic or silently-stale data. Our engineering depth is agentic delivery — we build with Claude Code and a library of custom skills, the agent-driven workflow that lets a small senior team ship and verify production AI fast — running on AWS (Bedrock, SageMaker, GPU compute, Kiro) accelerated by NVIDIA. We stay model-flexible; the method is the point, not the model. We are NDA-constrained, so we lead with method where we cannot lead with names — which is exactly what this checklist asks every serious partner to do. ## Frequently asked questions about choosing an AI partner The questions below are the ones buyers ask most often in active vetting — and the ones that most often expose the gap between a firm's marketing and its delivery reality. **What is the single most important question to ask an AI vendor?** "What is your PoC-to-production conversion rate, as a number — and walk me from one demo to one live system." The number screens for production-proof; the walkthrough screens for whether the people answering have actually crossed the gap. A vague answer or a pivot to logos is the tell. **What is the difference between an AI consulting firm and an AI delivery partner?** A consulting firm typically advises on strategy and architecture; an AI delivery partner takes accountability for building and deploying a working system. In practice, firms do both — but the distinction matters for scoping accountability. Ask any candidate: do you own the deployment, or do you hand off a specification? **How long should an AI engagement take before I see results in production?** The honest answer depends entirely on scope and data readiness. A narrowly scoped integration against clean, well-documented data can reach production in weeks; a platform-level transformation takes months. Be skeptical of any firm that gives you a production timeline without first auditing your data estate. **Is an NDA a legitimate reason not to share client references?** Yes — in regulated industries, client confidentiality is often contractually required, and some anonymization is expected (Kalvium Labs AI, May 2026). The resolution is to shift from logo-proof to method-proof: ask the firm to show you its evaluation harness, its verification step, and a specific anonymized account of what was shipped and what the measured outcome was. That evidence is available even when the client name is not. (See "Is 'all our clients are under NDA' a red flag?" above.) **What is an independent verification step, and why does it matter?** Independent verification means the accuracy, reliability, and output quality of a delivered AI system is assessed by a method or party that is not the team that built it — an internal audit function, a third-party technical review, or a documented evaluation protocol run before handoff. It matters because AI systems can appear to work in testing and fail in production in ways that are subtle and expensive. A partner who cannot describe their verification method is asking you to trust their assessment of their own work. **How do you evaluate an AI partner's data practices before you hire them?** Ask two questions: what happens when a data source is stale or malformed, and what does the contract say about data lineage? A serious partner has a written data-integrity contract — it should specify that no synthetic data or silently-stale data may inform a production decision. Data quality is the single most-cited cause of GenAI pilot failure among enterprise practitioners (Informatica, *CDO Insights 2025*, January 2025, n=600). A partner who has no answer for stale data has not crossed the gap from demo to production. **What is the difference between a boutique AI firm and a Big-4 consultancy for this kind of work?** Both can deliver; the structural differences are in how seniority and accountability flow through the engagement. The question to ask any firm — boutique or global — is the same: who specifically does the work, and how do you verify it? We treat the comparison in full on [boutique vs. Big-4 AI consulting](/boutique-vs-big-4-ai-consulting/). For more answers, see the full [FAQ](/faq/). --- *Vetting a shortlist, or want a straight read on whether a stalled pilot belongs in production? Put the three tests to us first. [Book a 30-minute working session](/book/) — no deck, no pitch — or read more on [how we work](/how-we-work/) and [AI consulting](/ai-consulting/).* *NewGenApps — production AI, proven. Stay a step ahead, always.* ============================================================================== # What Does an AI Consulting Firm Actually Do? URL: https://newgenapps.com/what-does-an-ai-consulting-firm-do/ ============================================================================== # What does an AI consulting firm actually do? **In short:** An AI consulting firm scopes the right problems, designs and builds working solutions, and — the distinction that matters — takes that work through verified production so your team can run it. The firms worth hiring keep senior practitioners on the problem from first scoping call to live system; many sell a strategy document and hand the build to a separate team. The difference between those two models is the difference between a recommendation on a slide and a system your business operates and trusts. --- ## What does an AI consulting firm actually do? An AI consulting firm identifies which AI problems are worth solving for a specific business, designs and implements a solution, and verifies that the solution works reliably before it goes live — then ensures the client's team can operate and maintain it. The job has three parts. First, **problem selection**: deciding which use cases justify the cost and risk of AI, which are better served by conventional software, and which should not be built at all. Most organizations have more ideas than viable candidates. The first filter is elimination. Second, **solution design and build**: architecture, data pipelines, model selection, integration with the systems the use case actually touches, and the evaluation harnesses that let you measure whether it works on real inputs rather than curated ones. Third, **production verification and handover**: evidence that the system holds up under real conditions — not under demo conditions — and documentation your team can operate from after the engagement closes. The defining characteristic of a firm that does this well is that the senior practitioners who scoped the work are the ones who build and prove it. No reassignment to a team you never met during the sale; no separate delivery vendor brought in after the strategy is signed. The point where a strategy firm hands the build to someone else is exactly where the constraints discovered in code stop reaching the person with the authority to change the design. If you want to understand how this applies to an actual engagement, start with [AI consulting](/ai-consulting/). --- ## What is the difference between AI consulting and AI development? AI consulting decides *what* to build and *whether* to build it; AI development builds it. The two overlap at the implementation phase — and the best AI consulting firms do both. | | AI consulting (pure strategy model) | AI consulting + development (full-arc model) | |---|---|---| | **Decides what to build** | Yes | Yes | | **Builds the solution** | No — hands off to a development vendor | Yes — same practitioners | | **Verifies it in production** | Rarely | Yes — part of the engagement | | **Leaves your team able to run it** | Depends on handoff quality | Yes — explicit handover gate | The line between the two roles blurs in practice, and that blurring is the problem. A consulting firm that stops at "decide what to build" and passes the work to a separate development shop creates a coordination gap. The team who designed the solution is not the team who discovers its limitations in code, and the constraints that surface during build never make it back to the person with the authority to rethink the design. The firms that carry both roles are not doing two jobs — they are closing a gap that exists structurally in the two-vendor model. For more on what the build-vs-buy decision looks like in practice, see [what "production build" actually means](/pilot-vs-production-ai-what-changes/). --- ## What does a good AI consulting engagement look like, step by step? A structured AI consulting engagement runs in four stages: opportunity selection, proof of concept, production build with evaluation, and operational handover. Each stage is a filter. Each one is allowed to kill the project. The willingness to stop is what separates a delivery firm from a vendor protecting a contract. 1. **Opportunity selection.** The engagement starts before any model is written. The consulting team audits candidate use cases against three criteria: data availability, measurable business value, and AI-appropriate risk profile. A use case where a confident wrong answer is expensive — a customer-facing claim, a financial figure, a legal or compliance statement — demands a different, costlier design than one where a wrong answer is cheap to catch. The output is a ranked shortlist with an explicit success metric for each viable case and a written reason the deferred cases were deferred. 2. **Proof of concept.** A working prototype scoped to validate the core technical assumption — not a demo, not a slide. The POC runs against real, messy data, not a curated set optimized for the room. It is evaluated, not just demonstrated: measured against the success metric from stage one on inputs the system has not been tuned to. A demo that works in the room and a POC that holds up on held-out real data are categorically different artifacts. Conflating them is the most common reason a "successful pilot" never reaches production. Gartner predicted, in a July 2024 press release, that at least 30% of generative-AI projects would be abandoned after proof of concept by end of 2025, citing poor data quality, inadequate risk controls, and unclear business value — a figure that points squarely at the gap between a POC passing and a POC being production-ready (Gartner, 29 July 2024). 3. **Production build with evaluation.** This is where most of the real engineering lives, and where advisory-only firms stop. Production-grade delivery adds, at minimum: integration with the systems and data the use case actually touches (production data is incomplete, inconsistent, and changing — a notebook on a fixed file and a live pipeline are different engineering problems); evaluation harnesses that score outputs continuously against the success metric, not once at launch; guardrails and failure handling calibrated to the cost of being wrong — grounding and retrieval where factual accuracy matters, input/output validation, human review on high-stakes paths, and fallbacks for model uncertainty. The [OWASP Top 10 for LLM Applications (2025)](https://owasp.org/www-project-top-10-for-large-language-model-applications/) catalogs the failure classes a serious build must defend against — prompt injection, sensitive-data leakage, excessive agency among them; and monitoring instrumentation so the team can see accuracy, groundedness, and cost degradation before users do. McKinsey found, in its November 2025 *State of AI* report, that only about 7% of organizations had fully scaled AI across the enterprise, with most stuck between pilot and production (McKinsey QuantumBlack, November 2025). Orchestrating retrieval, evaluation, guardrails, and model components is the mechanism of this stage, not its purpose. The purpose is narrower and harder: a system that holds up under real conditions, with a documented record that it was verified before users depended on it. 4. **Operational handover.** The engagement closes when the client's team can run the system without the consulting firm on call. This means documented architecture, runbooks, the evaluation suite handed over so the client can re-prove the system after a change, and at least one operated production cycle with the internal team driving. Proven is not a launch-day event; it is a capability the client keeps. A firm that cannot hand over the means of verification has delivered something the client cannot re-check — which is to say, something the client cannot fully trust. --- ## Who actually does the work on an AI consulting engagement? At most firms, a senior practitioner scopes the work and a more junior team delivers it — the standard professional-services arrangement, where senior time is sold and supervised at the top and execution happens further down. That arrangement is well suited to structured analysis a capable generalist can produce under review. Production AI is a different kind of deliverable. The judgment that decides whether it succeeds shows up in the build, not the plan: choosing which use cases are feasible, designing an evaluation that measures the right thing on real data, deciding which failure modes need guardrails and which are cheap to catch, recognizing when a proof of concept's apparent success is an artifact of a curated test set. These are senior calls, and in a delivery model that pushes execution to the most junior tier, they land with the people least equipped to make them — and least likely to escalate when something is off. NewGenApps keeps the named senior practitioners on the work rather than supervising it. The person who scopes the use case is the person committing production code and signing off on the evaluation. Scoping holds up better because the person who discovers the constraints during build is the person accountable for the design they came from. Read more about how we staff engagements on [how we work](/how-we-work/). --- ## What should you ask an AI consulting firm before you hire them? Before committing to an engagement, ask four questions that reveal whether the firm takes work through production or stops at strategy. 1. **Who will be doing the technical work day to day — and can I meet them before we sign?** This reveals whether the senior people you met in the sales process are the delivery team, or whether you are buying access to a partner's reputation and receiving a junior team's time. 2. **How does the engagement end — what does handover look like?** A firm with a genuine production model can describe the handover gate specifically: what documentation is produced, what the evaluation suite looks like, and how the client team is trained to operate the system. A firm that exits after a POC will not have a clear answer. 3. **How do you evaluate that the system works before it goes live?** This reveals whether the firm has a structured evaluation practice — defined success metrics, held-out test sets, red-teaming, edge-case coverage — or relies on informal judgment. A firm that says "we test it and it looks right" has not built the evidence a production system needs. 4. **What does a past engagement that went into production look like — without naming the client?** A firm with a genuine production track record can describe the architecture, the evaluation approach, and the challenges encountered — without disclosing client identity. One that has not taken work to production cannot describe what that experience is actually like. Sector, scale, and geography descriptors are sufficient; confidentiality is not a reason to be unable to characterize the work at all. --- ## Frequently asked questions **Is AI consulting the same as AI development?** No. AI consulting determines which problems justify an AI solution and how to design it; AI development builds the solution. The most capable firms combine both: the same team that scopes the problem builds and verifies the production system. Firms that only consult hand the build to a separate vendor, creating a structural gap between design intent and delivered system. **How long does an AI consulting engagement typically take?** A scoped proof of concept can run four to eight weeks. A production build — which includes evaluation, integration, and operational handover — typically adds eight to sixteen weeks on top of that, depending on integration complexity and the maturity of the client's data infrastructure. Engagements that promise AI in production in two weeks are either narrowly scoped or skipping the evaluation phase. **What kinds of problems do AI consulting firms typically work on?** The most common categories are: document processing and extraction (turning unstructured text into structured data), decision-support systems (AI-assisted triage, prioritization, or classification), conversational interfaces grounded in a company's own data, and workflow automation where deterministic rules fail on edge cases. The common thread is that these are problems where the output needs to be accurate enough to be operationally useful — which is why evaluation is the critical gate, not the build itself. **What makes an AI consulting firm worth hiring?** Three things: senior practitioners doing the delivery work, not supervising it; a structured evaluation practice — they can describe how they verify a system works before it goes live; and a production track record — they can describe, without naming clients, engagements that went live and that the client's team runs today. A firm that can only describe strategy work or proofs of concept has not closed the production gap. **How is AI consulting different from hiring an AI vendor or buying an AI product?** An AI vendor sells a pre-built product configured for your context. An AI consulting firm diagnoses the problem first and determines whether a pre-built product fits, a custom build is warranted, or the problem should not be solved with AI at all. The consulting function is the evaluation and selection layer; the development function is the build layer when custom work is justified. A competent firm will sometimes conclude the right answer is to buy, not build — and say so. --- If you want to see what this looks like as an engagement, start with [AI consulting](/ai-consulting/) or read [how we work](/how-we-work/). ============================================================================== # Boutique vs Big-4 AI Consulting: An Honest Comparison URL: https://newgenapps.com/boutique-vs-big-4-ai-consulting/ ============================================================================== # Boutique vs Big-4 AI Consulting: An Honest Comparison A buyer searching "Accenture AI consulting alternative" has already formed a question. They suspect the large-firm model has a gap, and they are looking for an honest answer — not a sales page dressed as an analysis. This article attempts that. Both models are described on their own terms, because the honest comparison is the only version that is useful. --- **In short:** Neither model is better in the abstract. A Big-4 or major strategy firm is the right call when a program needs board-level brand authority, multi-continent delivery capacity, or cross-practice transformation management that no boutique can staff. A senior-led boutique is the right call when the goal is a working AI system in production — built by the practitioners who scoped it, on a short timeline, with independent verification that it actually performs. The deciding question is not prestige; it is which delivery model matches the job you actually need done. --- ## What is the difference between a boutique and a Big-4 AI consultancy? A Big-4 or major strategy firm brings global delivery infrastructure, multi-practice integration, and recognized brand authority. A senior-led boutique concentrates senior practitioners on a smaller set of engagements, with a shorter chain between the person who scoped the work and the person who builds it. **Big-4 / major strategy firm / SI.** Firms like Accenture, Deloitte, McKinsey, BCG, and Capgemini run global AI practices with thousands of practitioners across multiple seniority bands. They are strong in transformation programs that span technology, process, and change management simultaneously. Their delivery model is the classic leverage pyramid: a senior partner or principal leads the relationship and overall design; a team at varying seniority levels executes under structured methodology and review layers. The pyramid works because it amortizes senior time across large programs and gives clients a single contractual counterparty at scale. **Senior-led boutique.** A boutique — by the useful definition — is a smaller practice where the practitioners who assessed the problem and defined the architecture are the same people who build, deploy, and verify the system. Engagement scope is narrower. Specialization tends to be deeper. The delivery chain is shorter. There is no large bench behind the pitch team. Neither is inherently superior. They are optimized for different jobs, and the honest differences are easier to read in the table below. --- ## When should you hire a Big-4 or strategy firm instead of a boutique? Hire a Big-4 or major strategy firm when the engagement requires board-level brand authority, multi-continent delivery capacity, or cross-practice transformation management that no boutique can staff. Specific conditions where the large-firm model has a genuine structural advantage: 1. **Enterprise-wide transformation programs.** When organizational change management, legal, finance, technology, and compliance need to be coordinated under one contract, the large firm's multi-practice structure is doing real work — not just overhead. 2. **Regulated industries with procurement prerequisites.** Some sectors require a counterparty with global insurance capacity, audit-firm standing, or recognized compliance credentials as a condition of procurement, not a preference. 3. **Political cover.** When a program's success depends partly on internal legitimacy — where "we hired McKinsey" or "Deloitte signed off" is a meaningful signal to the board or to enterprise-wide stakeholders — the brand name is doing measurable work that a boutique cannot replicate. 4. **Multi-geography deployment at scale.** When a program genuinely needs to staff across many regions simultaneously, bench depth matters. A boutique is not the right answer. Conceding this cleanly is important. A buyer whose program fits one of these descriptions should hire a large firm. If the above describes your program, [our AI consulting page](/ai-consulting/) is a useful next step for understanding where the models divide. --- ## When does a senior-led boutique outperform a Big-4 for AI? A senior-led boutique outperforms when the objective is a working system in production with the evidence to prove it — not a strategy deck, not a pilot, not a roadmap — and when the buyer needs the practitioners who scoped the engagement to be the ones who build and verify it. The structural advantage is the delivery chain. In a large firm, the partner who carried credibility in the sales cycle typically hands delivery to a team the buyer has not met. In a boutique, the seniors present at scoping remain present at delivery. For AI specifically, this matters for three reasons. **AI systems surface edge cases that require judgment.** A pilot is judged on a curated happy path. Production AI has to handle malformed inputs, distribution shift, partial outages, retries, and the long tail of real user behavior. The work that turns a demo into a system — evaluation harnesses, guardrails, fallbacks, monitoring, rollback — is most of the cost and almost none of the pilot. The judgment calls that determine how to handle those failure modes require the contextual understanding built during scoping. That context dissipates when the delivery team changes. **Independent verification requires senior involvement.** Testing whether a production system actually performs as designed — not just in the demo environment but under real conditions — is not a junior analyst task. It requires the practitioner who set the design criteria to be present at the verification stage. **Speed to first value is a function of decision-making latency.** Fewer layers between the question and the decision means faster iteration. In a fast-moving model landscape, the cost of slow first value is not just budget — it is the strategy decisions made on stale assumptions while waiting. The headline evidence for the production gap sits in the incumbents' own research. McKinsey's *The State of AI* (March 2025, n=1,491 across 101 countries) found that only approximately 39% of organizations report any EBIT impact from generative AI at the enterprise level — and of those, most said less than 5% of EBIT was attributable to AI. Only roughly one-third of organizations report scaling AI across the enterprise. Among those experimenting with AI agents, fewer than 10% have scaled them in any single function (McKinsey, *The State of AI*, March 2025). The pattern is consistent across the major firms' own research. BCG's *Where's the Value in AI?* (October 2024, n=1,000 CxOs across 59 countries) found 74% of companies had yet to show tangible value from their use of AI, with only 4% generating significant value across functions. These are not assertions from a competitor. They are the incumbents' own measurement of the gap between AI launch and AI value at scale. The question they raise is which delivery model is built to close that gap. For a closer look at why pilots stall, see [why AI pilots fail](/why-ai-pilots-fail/) and [what changes between a pilot and production](/pilot-vs-production-ai-what-changes/). --- ## Why do so many large-firm AI engagements stall before production? The most commonly cited factors are misaligned success metrics, the partner-to-junior-delivery transition, and the structural incentive to expand scope rather than verify results. Each is a description of a structural dynamic, not an accusation. **1. Metric misalignment.** Large programs are typically contracted and measured on deployment milestones: launch a pilot, onboard users, deliver a roadmap. Production impact — verified EBIT contribution, measurable reduction in error rate, confirmed operational change — is a different metric, and frequently appears in a follow-on statement of work rather than the original contract. A program that ends at launch is not a failed program by its own terms; it simply was not contracted to go further. **2. The delivery-chain transition.** The partner who carries credibility in the sales cycle typically hands delivery to a mixed team. For a transformation program with strong methodology and review layers, this is workable. For a probabilistic AI system where the hard calls happen during the build, the handoff cost is higher. Traditional system integration assumes deterministic components: you can spec them, test them once, and sign off. LLM-based systems are probabilistic — the same input can yield different outputs, quality drifts as upstream models and data change, and "correct" is often a distribution rather than a value. You cannot acceptance-test a probabilistic system with a deterministic checklist. Teams staffed for deterministic delivery face a different challenge than the one they were trained for. **3. Scope expansion as a revenue model.** A large firm's economic model rewards program expansion. The incentive structure is not aligned with "smallest system that solves the problem and is verified to work." **4. Unverified output as delivery risk.** In October 2025, Deloitte's Australian member firm agreed to repay the final installment of a government contract after a researcher found multiple AI-generated fabrications in the delivered report — including citations to non-existent academic papers and a fabricated court quote (Fortune, 7 October 2025; The Register, 6 October 2025). The lesson is not that AI is unreliable. It is that unverified AI output reached a client deliverable. A production-grade system requires provenance, verification, and a human-checkable trail as part of the deliverable, not an afterthought. This risk is not unique to large firms, but it is visible when the output of a big-firm engagement cannot be independently checked. The same dynamic appears in AI-washing cases: Builder.ai entered insolvency in May 2025 after revenue was reportedly inflated approximately 300%; its platform relied on human engineers performing work marketed as AI automation (TechCrunch, May 2025). Buyers paid for capability they could not verify. These are not arguments against using large firms for the jobs they are structurally good at. They are arguments for knowing which job you are hiring for and specifying the deliverable — verified production system, not just launch — in the contract. See [what changes between a pilot and production](/pilot-vs-production-ai-what-changes/) for the engineering specifics. --- ## Which model fits which job — the comparison The table below reflects structural differences between the two models, not a ranking. Dated sources are cited where a published figure exists. | Dimension | Senior-led boutique | Big-4 / strategy firm / SI | Which job each fits | |---|---|---|---| | **Who delivers your work** | The seniors who scoped it remain on the build; the chain from decision to execution is short | Partner-led sale; delivery team is typically more junior and leverage-based; a structural feature of the pyramid model | Boutique when the hard judgment calls happen during the build. Big firm when the program needs broad, repeatable staffing at scale. | | **Reaches production / scaled value** | Built production-first; evaluation, guardrails, and verification are part of the system | Strong at launching programs. Industry-wide context: ~39% of organizations report any EBIT impact from GenAI; ~6% are "AI high performers" (McKinsey, *State of AI*, March 2025); 74% show no tangible value yet (BCG, *Where's the Value in AI?*, October 2024) | Boutique when the deliverable is a system that must run in production. Big firm when the deliverable is an enterprise-wide mandate or transformation program. | | **Independent verification** | Verifiable system and human-checkable trail as part of the deliverable | Methodology and review layers; verification quality varies by engagement type and team | Insist on it from whichever model you choose; an unverified deliverable is a liability regardless of the firm's brand. | | **Pricing model** | Senior-dense, fewer bodies; pay for judgment and continuity | Leverage-priced; includes brand cover, indemnification, and global reach baked into the rate card | Big firm when you need risk transfer and global reach. Boutique when you do not. | | **Speed to first value** | Fewer decision layers; working slice typically in weeks; iterate from real output | First production system often in quarters across a multi-workstream program; governance has its own value when governance is the product | Boutique when slow first value is itself a strategic cost. Big firm when the program is genuinely enterprise-wide and change management takes time. | | **Scale and global footprint** | Built to go deep on a few systems; not structured to staff a hundred-person program across regions | Global delivery capacity; handles multi-continent, multi-practice programs | Big firm when geography, language, or regulatory footprint requires it. | | **Brand and board cover** | Recognized by delivery quality and track record | Brand name carries weight in internal approval processes and regulated procurement | Big firm when the vendor's name is part of what the engagement is purchasing. | *McKinsey source: McKinsey QuantumBlack, "The State of AI: How Organizations Are Rewiring to Capture Value," March 12, 2025 (survey: July 16–31, 2024; n=1,491; 101 countries). BCG source: BCG, "Where's the Value in AI?," October 24, 2024 (n=1,000 CxOs; 59 countries).* --- ## How do you evaluate which model is right for your AI program? The deciding question is not firm size or brand — it is whether you need a system running in production, independently verified to perform, or a transformation program with board-level organizational backing. A practical checklist for the scoping conversation: 1. **Define your success metric before choosing a vendor.** If success is "a working system in production that has been independently verified to perform as designed," write that into the brief. If success is "a roadmap approved by the board," that is a different contract. 2. **Ask who will be on the delivery team.** Get names. Ask whether the people in the room during scoping will be the people building and deploying the system. 3. **Ask how the firm defines production.** Not deployment, not launch — measurable impact on operations or outcomes, in the live environment, under real conditions. 4. **Assess your governance requirements honestly.** If the program requires cross-functional change management at scale, or multi-geography simultaneous rollout, that is a genuine large-firm advantage. Do not hire a boutique for a job it is not built to do. 5. **Assess your timeline.** A governance-heavy engagement model has genuine value when governance is the product; it has cost when speed-to-production is the binding constraint. The full vetting checklist — applicable regardless of which model you choose — is at [how to choose an AI partner](/how-to-choose-an-ai-partner/). --- ## Frequently asked questions **Is a boutique AI consultancy better than a Big-4 firm?** Neither is categorically better. They solve different problems. A Big-4 or strategy firm has structural advantages in large-scale transformation programs that require organizational change management, multi-practice integration, and recognized brand authority. A senior-led boutique has structural advantages when the deliverable is a production system whose performance is independently verified, built by the same seniors who scoped it. The deciding factor is which job you actually have. **Why do so many AI consulting engagements fail to reach production?** McKinsey's *State of AI* (March 2025) found that only approximately 39% of organizations report any EBIT impact from generative AI, and only about one-third are scaling AI across the enterprise. BCG (*Where's the Value in AI?*, October 2024) found 74% of companies had yet to show tangible value from AI. Contributing factors include metric misalignment — contracts measured on launch rather than production outcomes — delivery-chain transitions from senior to junior practitioners, and scope expansion that extends programs without verifying results. **What does "senior-led delivery" mean in AI consulting?** Senior-led delivery means the practitioners who assessed the problem, defined the architecture, and scoped the engagement remain the practitioners who build, deploy, and verify the system. The alternative — common in large leverage-pyramid firms — is that a senior partner scopes the work and hands it to a more junior team the client has not met. For AI specifically, this matters because production edge cases require the contextual judgment built during scoping. That judgment does not reliably transfer in a mid-engagement handoff. **When is a Big-4 AI firm the right choice?** When the engagement requires multi-continent delivery capacity, cross-practice transformation management, regulated-industry brand authority, or internal political cover that a recognized firm name provides. These are genuine structural advantages. A boutique cannot replicate them and should not claim to. **How do you verify that an AI system is actually working in production?** Independent verification means testing the system against the original design criteria in the production environment — not the demo environment. It involves measurable performance thresholds, defined failure modes, and a sign-off process that is distinct from the team that built the system. Ask any prospective firm how verification is structured and what the tangible deliverable is at the end of the verification stage. An answer that amounts to "trust the output" is not an answer. **What is "production AI" versus a pilot?** A pilot demonstrates that an AI approach is feasible under controlled conditions. Production AI runs in a live operational environment, integrates with real systems and real data, handles edge cases at scale, and has been verified to perform as designed under those conditions. The gap between pilot and production — in engineering complexity, integration work, monitoring, and failure-mode handling — is where most AI programs stall. See [what changes between a pilot and production](/pilot-vs-production-ai-what-changes/). --- If you need a system that runs in production — built by the seniors who scoped it, with an independent check that it performs as designed — that is what NewGenApps does. See [AI consulting](/ai-consulting/), or [book a 30-minute working session](/book/). If the Big-4 model is the right fit for your program, the checklist at [How to choose an AI partner](/how-to-choose-an-ai-partner/) applies regardless of which firm you are evaluating. *NewGenApps — production AI, proven. Stay a step ahead, always.* ============================================================================== # Build an in-house AI team or hire a partner? The economics URL: https://newgenapps.com/build-vs-buy-ai-team/ ============================================================================== # Build an in-house AI team or hire a partner? The economics *Published: 2026-06-25 · Author: [NewGenApps](/how-we-work/) · Estimated read: ~8 min · Last reviewed: 2026-06-25* --- ## In short Most teams that ask this question are looking for permission to hire fast. Building in-house is not the wrong choice — it is slower and more expensive than most business cases acknowledge, and it reaches production at roughly half the rate of the alternative. In-house AI builds reach production roughly one-third of the time — about 33%. Engagements run by expert delivery partners succeed approximately **67%** of the time — roughly twice as often. That is the finding of MIT's *The GenAI Divide: State of AI in Business 2025* (Project NANDA, August 2025), which reviewed approximately 300 publicly disclosed AI initiatives, conducted 52 structured interviews with enterprise leaders, and surveyed approximately 153 senior leaders at industry conferences. The model the evidence supports is not "always build" or "always partner." It is **partner, then transfer, then build**: bring in a senior team to cross the production gap on your first system, have them transfer the capability and the operating discipline, then let your in-house engineers take ownership from working software rather than a standing start. You still end up owning the capability — you simply do not pay the first-build failure premium to get there. --- ## Should you build an in-house AI team or hire a partner? The case for building in-house holds up. Long-term ownership, institutional knowledge, cost dilution across multiple systems, a competitive moat — these are real advantages. The business case that justifies "hire an AI team" is not wrong about what building eventually gets you. What it usually gets wrong is three costs that rarely appear on a spreadsheet: the hiring timeline, the turnover rate, and the production-delivery gap. The spreadsheet answers a well-defined question: what does it cost in salary and compute to run an AI team for twelve months? The question it skips is harder: who on this team has shipped a production AI system before, and how long does it take to hire them? Those two sub-questions determine whether the in-house path gets you to a working system in year one or an expensive, stalled build that needs rescuing in year two. **The production-delivery gap** is the structural difficulty. Model access is a commodity — most teams can get an API key in minutes. The scarce input is the accumulated experience of taking a system from "demo that works in a notebook" to "runs reliably in production, handles real inputs, recovers from failure modes, is observable and evaluable, and delivers the business outcome it was commissioned for." That experience is acquired engagement by engagement. A team building its first AI system acquires it on the organization's own budget and timeline. A partner has already paid for it. Three archetypes of buyer clarify the decision: - **Teams ready to build from day one.** These are organizations that already include engineers with a track record of production AI delivery — not AI familiarity, but verifiable prior systems in production. If you have that, the partner-first step may be shorter or different in form. - **Teams who will build, but need a first success first.** This is the majority of the buyers who ask this question. The right sequence is partner for the first system, transfer the discipline, then build from there. - **Teams for whom partnering indefinitely is the right model.** If in-house ownership is not a strategic goal — you are a focused product company with a narrow AI use case — ongoing partnership may be more efficient than maintaining an AI engineering team. Self-diagnosis starts with one question: does anyone on your team have a production AI system — not a demo, not a pilot, but a running system in a real environment — on their resume? --- ## How much does building an in-house AI team actually cost? A 4–5 person in-house AI team typically runs in the range of $520K–$840K in year-one fully-loaded costs. That figure is a directional planning range derived from 2025–26 AI-hiring research, not a quote — validate it against current market data for your geography and role mix before using it in a business case. It is also a conservative floor, for the reasons below. **Compensation is the visible cost, and it is high.** Senior AI/ML engineers in the US command base salaries of $180K–$280K; at the staff and principal level, total compensation regularly clears $350K–$600K (KORE1 AI Engineer Salary Guide, 2026 — a specialist-recruiter compilation drawing on Built In, Glassdoor Feb 2026, and Levels.fyi Jan 2026 data). PwC's 2025 Global AI Jobs Barometer (June 2025, primary research across close to a billion job postings on six continents) found that workers with AI skills earn a 56% wage premium over peers in the same roles without those skills, up from 25% in 2024 — a premium that reflects both supply scarcity and employer competition for a small pool. **The fully-loaded cost is materially higher than the salary line.** The standard planning multiplier is approximately 1.25–1.4× base salary for benefits, payroll taxes, equipment, software, and overhead (2025 benchmark). A team modeled at base salary alone understates the true first-year cost by roughly 25–40% before a line of production code ships. **The hire cycle is longer than most timelines assume.** Senior AI/ML engineering roles typically take 45–75 days to fill in the current market (KORE1 Cost to Hire AI Engineer, 2026 — a specialist-recruiter market analysis). Acceler8 Talent (a specialist recruiter, 2025 market commentary) cites data indicating searches can extend to 114 days when the offer is below the market-rate threshold of approximately $200K for senior roles. Sequentially hire four to five people — which is how budget and approval cycles usually work — and first capability is months away before onboarding begins. Acceler8 Talent (2025 market commentary) reports that approximately 76% of firms cite a severe shortage of AI/ML specialists; PwC's 2025 AI Jobs Barometer (June 2025, primary research) independently confirmed that AI-skill job postings grew 7.5% year over year even as total job postings fell 11.3%. The frequently cited directional composite of approximately 142 days from "decision to build" to "team in place and productive" is derived from per-role hire cycles plus sequential hiring and onboarding ramp — it is a planning composite, not a published statistic. **Recruiting costs add to the first-year tab before anyone ships.** Agency fees for senior AI hires typically run 20–25% of first-year base salary — roughly $28K–$56K per senior hire depending on role level (KORE1 market guide, 2026 — specialist-recruiter market commentary). For a five-person team, that is $140K–$280K in recruiting costs alone, on top of the salary-plus-load figure. **Turnover compounds every cost above.** Software engineer median tenure runs approximately two years across the tech industry (LinkedIn workforce data, 2024; BLS background data) — meaning roughly half of a small in-house team can be expected to turn over within the first two years. AI engineering roles, competing actively against frontier-lab and hyperscaler compensation, are not more stable than that baseline. No dated primary measures AI-engineer annual turnover at a precise rate; treat any specific figure in this area as a directional planning estimate rather than a measured statistic. The economic logic of turnover is less about the rate itself and more about what leaves: when one of three engineers departs at month nine, the build does not lose a third of its capacity — it loses the system knowledge that lived in that person's head, restarts a 45–75-day hire cycle, and pays the onboarding ramp again. This is structural in a small in-house build, and it does not improve until the institutional knowledge is in documented systems rather than people. *Directional figures footnote: the $520K–$840K range and the ~142-day composite are planning ranges derived from 2025–26 AI-hiring and workforce research. They are not quotes, guarantees, or representations of any specific engagement. Validate against current market data for your geography, role mix, and seniority composition before using in a business case.* --- ## How often do in-house AI builds reach production, versus partnerships? In-house AI builds reach production roughly one-third of the time — about 33%. Engagements run by expert delivery partners succeed approximately 67% of the time — roughly twice as often. This is the finding of MIT's *The GenAI Divide: State of AI in Business 2025* (Project NANDA, August 2025). The report's own wording: *"external partnerships with learning-capable, customized tools reached deployment ~67% of the time, compared to ~33% for internally built tools"* — and, in the report's summary framing, *"external partnerships see twice the success rate of internal builds."* The MIT NANDA report's methodology: approximately 300 publicly disclosed AI initiatives reviewed; 52 structured interviews with enterprise leaders; and a survey of approximately 153 senior leaders at industry conferences. "Success" in this context means deployment beyond pilot with measurable KPIs tracked at six months — not reaching a demo, and not financial ROI alone. One important caveat, which the evidence pack for this piece flags explicitly: the split likely reflects organizational capability and readiness as well as approach choice. Organizations that successfully manage external partnerships may differ systematically from those that build cold — in AI maturity, executive sponsorship, and procurement discipline. The correlation does not establish that the partnering approach *causes* the higher success rate. What it establishes is that organizations with the discipline and capacity to manage a partnership reach production more often — and that is precisely the point: the first-build experience gap is the thing the partner → transfer → build model addresses. The broader production gap is corroborated by independent research. S&P Global Market Intelligence's *Voice of the Enterprise: AI & Machine Learning, Use Cases 2025* (survey fielded October–November 2024, published 2025, 1,006 respondents across North America and Europe) found that the share of companies abandoning most of their AI initiatives jumped from 17% to 42%, with the average organization scrapping approximately 46% of proof-of-concepts before production. Updated Gartner research found that by the end of 2025, more than half of GenAI projects were abandoned after the proof-of-concept stage. These figures are not disaggregated by build versus partner approach — they establish the size of the POC-to-production gap that any first in-house build must cross. RAND Corporation interviews with 65 experienced data scientists and ML engineers (published August 2024) found that more than 80% of AI projects fail to deliver their promised business value — roughly twice the failure rate of non-AI IT projects. The RAND study is a qualitative study of a specific interview pool, not a large-N representative survey — treat the directional finding as corroborating context, not as a precise measured rate. The consistent root cause identified across those interviews was delivery and organizational readiness, not model access. The consistent root cause across all of these studies is the same: delivery, not model access. The organizations that reach production are the ones that can scope and evaluate, handle failure modes, maintain observability, and govern output reliably. That experience does not come from an API key — it comes from having shipped before. --- ### Build in-house vs hire an expert partner: the economics compared | Factor | Build in-house (first AI system) | Hire a senior partner | |---|---|---| | **Success rate to production** | ~33% (about one-third of the time) — MIT NANDA, *The GenAI Divide*, August 2025 | ~67% — roughly twice as often as in-house builds — MIT NANDA, *The GenAI Divide*, August 2025 | | **Time to first capability** | ~45–75 days *per* senior hire, made sequentially; months to a working, productive team (KORE1, 2026; directional composite) | Senior team engaged in days to weeks; delivery work begins at contract start | | **Year-one cost** | Directional planning range: ~$520K–$840K fully loaded for a 4–5 person team — a conservative floor at blended US seniority (2025–26 AI-hiring research) | Scoped to the work; no standing payroll, recruiting costs, or ramp time carried before output | | **Key-person risk** | Concentrated in 2–3 people; median software-engineer tenure of approximately two years (LinkedIn, 2024) means roughly half of a small team may turn over before the system is fully stable — and each departure restarts a 45–75-day hire cycle | Spread across a partner team with established continuity; institutional knowledge is in documented systems, not individuals | | **Production-delivery experience** | Acquired on your budget and your timeline, on your first system | Already paid for across prior engagements; the failure modes were met on earlier systems, not yours | | **Long-term ownership** | Yours from day one — if the build reaches production | Transferred to your team over the engagement; you own it from working software, not a blank repository | *Directional-figures note: year-one cost (~$520K–$840K) and hire cycle composite (~142 days) are planning ranges from 2025–26 AI-hiring and workforce research. They are not quotes or guarantees. Validate for your geography, role mix, and current market conditions. NewGenApps' starting prices for each scoped engagement are published on the [engagements page](/engagements/); compare any partner's price, ours included, to the full all-in year-one cost of an in-house team, including the ramp period before output and the probability-adjusted cost of the first build not reaching production.* --- ## What is the "partner, then transfer, then build" model? The partner, then transfer, then build model is a three-phase approach to building in-house AI capability. In phase one, an experienced delivery partner takes your first AI system to production. In phase two, the partner runs a structured transfer of the codebase, operating discipline, and institutional knowledge to your team. In phase three, your in-house engineers take ownership of the live system and build the next one from working software rather than a standing start. Most organizations that try to build in-house on a first system go directly to phase three. They hit the one-third production success rate. The sequence below is built to avoid that. **Phase 1: Partner — cross the production gap once.** Engage a senior AI delivery team to run the first real production build end-to-end: scoped, integrated, evaluated, and verified in a production environment. The goal is not a demo or an extended pilot — it is a system that runs reliably, delivers a measurable business outcome, and has the observability and governance infrastructure to be maintained by whoever comes next. This is the phase that carries the highest risk and the lowest success probability for a first in-house build. Pay for it once, with people who have crossed it before. The production-delivery experience the partner brings — knowing how to handle distribution shift, malformed inputs, latency constraints, compliance requirements, and the edge-case tail that only appears at scale — was earned on prior systems, not on yours. One concrete illustration of what production discipline requires: in October 2025, Deloitte's Australian member firm agreed to a partial refund on a ~A$440,000 government report after AI-generated citations to non-existent sources were found (Fortune, 7 October 2025). The lesson is not specific to that firm — it is architectural. Provenance, citation verification, and a human-auditable output trail are not edge-case features; they are delivery requirements. A team shipping its first AI system rarely has this discipline yet. **Phase 2: Transfer — make the capability portable, not the people.** As the system approaches production, the partner runs a structured handover. The deliverable is not only working software — it is the operating discipline that keeps it working: the evaluation suite, the data-integrity checks, the deploy-and-verify protocol, the incident runbooks, and the model-governance playbooks. Transfer has acceptance criteria and a named completion point; it is not a vague "knowledge sharing" addendum. Done correctly, the in-house team can evolve and maintain the system safely without the partner present. That is the test: not whether they can read the code, but whether they can change it with confidence. **Phase 3: Build — take ownership from working software.** The in-house team you hire now joins a running system with a test harness, an evaluation suite, documented failure modes, and a paved path to production. They take over system one and design system two from a foundation of earned experience rather than theory. This is where in-house economics finally favor building: the highest-risk, lowest-success-rate first crossing is behind you, and the capability compounds from here. The long-term arithmetic: the cost of phase one — partnering to production — is typically recoverable in phase three, where the in-house team delivers subsequent systems at lower cost and higher reliability than a cold build would have achieved. You still own the capability. You did not pay the first-build failure premium to get there. This is the operating model behind NewGenApps' engagements — production AI, proven. A small senior team delivers into production, proves it works through independent verification rather than a status report, and hands the running system and the operating discipline to your team. See [how we work](/how-we-work/) for how this is structured in practice. --- ## When does building in-house from the start make sense? Building in-house from day one is defensible under three conditions. **First: prior production-delivery experience on the team.** The relevant credential is not AI familiarity — it is a track record of production AI systems. Engineers who have used AI tools widely are common; engineers who have shipped a production AI system with reliability, observability, and incident-response discipline are not. If your team includes someone with that track record on a comparable system, the partner-first step may be shorter or different in form. **Second: a hiring pipeline that can close roles faster than the market average.** If you have a strong brand in the AI engineering community, an existing network, or a specialist recruiter with a live candidate pool, you may be able to compress the 45–75-day-per-role cycle to something that keeps the project on schedule. Most organizations cannot. **Third: tolerance for the probability distribution.** A first cold in-house build reaches production roughly one-third of the time on the MIT NANDA evidence. If your organization can absorb a failed first attempt — financially, politically, and competitively — and treat it as the learning investment that buys the in-house experience, building cold is a rational choice. The question worth pressure-testing is whether the organization actually has that tolerance, not whether it believes it does in the planning phase. The internal build that works in practice tends to be small (two to three engineers), narrow in scope (a well-defined use case with a clear definition of "production" agreed upfront), and anchored by at least one engineer who has shipped a comparable system before. Scaling from that base is straightforward once the first system is running. EY's *Work Reimagined Survey 2025* (November 2025, 15,000 employees across 29 countries) found that only 12% of employees were receiving sufficient AI training to unlock productivity benefits. The gap between AI deployment and AI capability is not a technology problem — it is a delivery and skills-transfer problem. Whether you build or partner, that gap is where the investment needs to go. --- ## If the math points to partnering first The model that works is partner, then transfer, then build — that is exactly how NewGenApps structures engagements. See [AI consulting](/ai-consulting/) or [book a 30-minute working session](/book/). If you want senior capability without hiring it, [an embedded AI team](/embedded-ai-team/) compares a monthly retainer with a fixed project and a hire. If your in-house build has already stalled, [AI Rescue](/ai-rescue/) is built for that — the cost of recovering a stalled build is, on the published evidence, frequently higher than crossing the production gap with a partner the first time. Once you have decided to partner, [how to choose an AI partner](/how-to-choose-an-ai-partner/) covers what to look for in a vetting process. --- ## Frequently asked questions **What is the production success rate for in-house AI builds versus partnerships?** In-house AI builds reach production roughly one-third of the time — about 33%. Engagements run by expert delivery partners succeed approximately 67% of the time — roughly twice the rate. These figures come from MIT NANDA's *The GenAI Divide: State of AI in Business 2025* (August 2025), which reviewed approximately 300 publicly disclosed AI initiatives, conducted 52 structured interviews with enterprise leaders, and surveyed approximately 153 senior leaders at industry conferences. "Success" means deployment beyond pilot with measurable KPIs tracked at six months. **How much does it cost to build an in-house AI team?** A 4–5 person in-house AI team typically costs between $520K and $840K in year-one fully-loaded costs, based on 2025–26 AI-hiring and workforce research. This directional planning range covers salaries, benefits, tools, compute, and onboarding at a blended US seniority. It does not include recruiting fees (20–25% of first-year base per senior hire, per KORE1 2026), the opportunity cost of the approximately 45–75-day hiring cycle per role, or the probability-adjusted cost of a first build not reaching production. Validate against current market data for your geography and role mix before using this in a business case. **What is the "partner, then transfer, then build" model?** Partner, then transfer, then build is a three-phase approach to building in-house AI capability. In phase one, an experienced delivery partner takes your first production AI system across the production threshold — scoped, integrated, and verified. In phase two, the partner runs a structured transfer of the codebase, operating discipline, evaluation suite, and runbooks to your team. In phase three, your in-house team takes ownership of the live system and builds the next one from working software rather than a standing start. The end state is owned in-house capability; the sequence is designed to reach that end state without paying the first-build failure premium. **Why do in-house AI builds fail to reach production?** The most consistent cause across the published evidence is the production-delivery gap: the team has AI familiarity but not the accumulated experience of shipping systems that run reliably at scale. This experience — how to handle distribution shift, malformed inputs, latency constraints, the observability gaps that only appear in production, and the governance requirements that a compliant deployment demands — is acquired engagement by engagement. A team building its first system acquires it on the organization's own budget and timeline. RAND Corporation interviews with 65 experienced AI engineers and data scientists (August 2024) found that the failure causes are consistently delivery-related, not model-related. **When should you hire an AI partner instead of building in-house?** Hiring an AI partner is most defensible when: (1) your in-house team has not previously shipped a production AI system at the complexity level you need, (2) the per-role hiring cycle — typically 45–75 days for senior AI engineers, multiplied across sequential hires — would cause unacceptable strategic delay, or (3) the fully-loaded year-one cost ($520K–$840K directional) is not justified for the scope in question. The partner, then transfer, then build model allows you to partner for the first system and build in-house from the second — with working software and transferred operating discipline as the starting point. **What is the key-person risk in a small in-house AI build, and why does it matter?** No dated primary measures AI-engineer annual turnover at a single reliable rate — figures in this area vary widely and should be treated as directional planning estimates. What the evidence does support: software engineers broadly carry a median tenure of approximately two years (LinkedIn workforce data, 2024), and AI engineering roles, competing actively against frontier-lab and hyperscaler compensation, are not more stable than that baseline. The structural risk is less about the attrition rate itself and more about what leaves: in a small in-house build, system knowledge lives primarily in two or three people. When one departs, the build does not lose a proportional share of capacity — it loses the undocumented design decisions that engineer held, and restarts a 45–75-day hire cycle (KORE1, 2026) before that knowledge gap can be addressed. --- *NewGenApps — Stay a step ahead, always.* ============================================================================== # Orchestrating Multi-Agent AI Systems in Production URL: https://newgenapps.com/multi-agent-orchestration-production/ ============================================================================== # Orchestrating Multi-Agent AI Systems in Production **In short:** agentic AI in production is the engineering discipline that makes AI agents do useful work the thousandth time, not just the first — on real data, under load, with failures you can see, attribute, and contain. A production-grade multi-agent system is built from bounded-scope agents, typed output contracts, hard time limits, and an independent verifier that no agent can grade past. A demo proves an agent *can* perform a task once; production asks whether a multi-agent system performs it when an upstream source goes stale, a model behaves differently than it did yesterday, and a step half-finishes. The hard problem in [agentic AI](/agentic-ai/) is not making one agent clever. It is making a collection of AI agents do useful work, repeatedly, on real data, under load, without quietly producing a confident wrong answer. This piece sets out the engineering discipline that closes that gap — a discipline drawn from operating a production AI system at the hard end of reliability, where a wrong answer is expensive and immediate, then generalized for client delivery. It is the same method behind every engagement, described in [how we work](/how-we-work/). ## What does "agentic AI in production" actually mean? Agentic AI in production means an AI agent or multi-agent system that takes actions over real data, repeatedly and under load, where the system — not a human watching a demo — catches its own failures before they reach a user. The difference between agentic in a demo and agentic in production is not capability; it is legibility: in production you can see when a step half-finished, attribute which agent caused it, and contain the damage before it spreads. An agent that performs a task once on clean data is a demo. An agentic system whose mistakes are visible, attributable, and contained on the thousandth run is in production. The gap between those two states is a systems gap, not a model gap, and a single capability number hides it. "The agent succeeded" is usually a measure of one attempt — `pass@1`, in benchmark terms. Production runs on `pass^k`: the probability the agent succeeds on *every* one of k attempts at the same task, when nothing should have changed. Those are different numbers, and they diverge sharply even for strong models. The τ-bench tool-agent benchmark (Yao et al., Sierra, June 2024) introduced the `pass^k` metric precisely to expose this: a model that looks reliable on a single run is far less reliable across repeated runs of the same task. Capability is what a demo measures; reliability across repeats is what production requires, and the first predicts surprisingly little about the second. ## Why do agentic pilots fail to reach production? Agentic pilots fail to reach production because the work breaks in delivery, not in the model: 88% of AI proof-of-concepts do not reach widescale deployment — about four in every thirty-three make it to production (IDC, *Lenovo CIO Playbook 2025*, February 2025). The demo runs once on clean data; production runs continuously on data that goes stale, models that behave differently over time, and steps that half-finish. The gap is not a smarter prompt; it is the engineering that makes failure legible. The most detailed public evidence that this is a systems problem, not a model one, is the Multi-Agent System Failure Taxonomy (MAST): a study of more than 1,600 annotated traces across seven multi-agent frameworks that catalogs 14 distinct failure modes, finds that system-design issues account for 44.2% of failures, and concludes that failures stem from poor system design rather than model performance ("Why Do Multi-Agent LLM Systems Fail?", UC Berkeley et al., arXiv 2503.13657, 2025). Our own working version of that map is set out in the [production AI failure taxonomy](/production-ai-failure-taxonomy/). The forward-looking picture is no gentler: Gartner projects that more than 40% of agentic AI projects will be canceled by the end of 2027, citing escalating cost, unclear business value, and inadequate risk controls (Gartner, June 2025). Both findings point at the same cause — architecture, not model quality. The pattern across stalled pilots is consistent. A single agent is handed the whole job and works well until an input changes shape. There is no contract for what each step returns, so downstream code parses free text and breaks. And the only check on the result is the agent's own opinion of it, which is no check at all. Data readiness sits among the top-cited obstacles to moving GenAI from pilot to production — named by 43% of data leaders, tied with technology, and ahead of people, process, and regulation (Informatica, *CDO Insights 2025*, January 2025, n=600). The disciplines that catch these failures — evaluation and observability against real data — are exactly what a demo skips. The fix is not novel; it is applied [deploy-and-verify](/deploy-and-verify-ai/) discipline carried into a multi-agent topology. ## Why does one agent doing everything break in production? The instinct, when you first wire an LLM to tools, is to give a single agent the whole job. One long prompt, a fistful of tools, an open-ended instruction: read the data, decide what matters, act, and report back. It works in the demo. It works for the next dozen runs. Then it starts to drift in ways that are hard to see. Consider a concrete, non-proprietary example. An analytics agent is asked to summarize the day's operational metrics and flag anomalies. On most days it is excellent. On the day a data feed goes stale, the agent does not know the feed is stale — it sees numbers, and numbers look like data. It produces a clean, well-written summary built on yesterday's values, and because nothing errored, nothing alarms. The output is plausible, fluent, and wrong. Worse, the same agent that produced the summary is the one you would ask "are you confident?", and it will reassure you, because a model grading its own work confirms the story it just told itself. This is the structural weakness of the monolithic agent. It conflates roles that should be separated: gathering, deciding, acting, and checking. It has no bounded scope, so its failures have no boundaries either — one bad input contaminates the whole chain. It has no contract for what it returns, so downstream code parses free text and breaks on the first reword. And it has no independent check, so the only verification is the agent's own opinion of itself. These are not prompt-engineering problems. They are systems-design problems, and they are why "one agent does everything" does not survive contact with production. There is arithmetic underneath this. A multi-step agent's end-to-end reliability is the *product* of its per-step reliabilities, not the average. Ten steps at 95% each is roughly 60% end-to-end — 0.95 to the tenth power is about 0.60. Long-horizon agents therefore degrade faster than any single step's accuracy suggests, and one unverified step contaminates everything downstream of it. This is why bounded scope and per-step verification are not stylistic preferences; they are how you stop the product of probabilities from collapsing. ## The principle: legible failure, not a cleverer agent The goal is a system whose failures you can see, attribute, and contain. The mechanism that buys it is decomposition: splitting the work into **domain-specialized agents with bounded scope**, coordinated by a layer that owns sequencing, checks, and the final say. A capable conductor does not play every instrument; the conductor assigns parts, sets the tempo, and listens for the section that has drifted. Applied to agentic AI, orchestration is the means, not the end — what matters is the performance, not the conducting. This approach is model-flexible: it cares about the result, not the brand of model behind each agent, though it goes deepest on the models we know best. Five engineering rules make the parts play in time. **1. Bounded scope and failure-class ownership.** Each agent does one kind of thing and owns one class of failure. A retrieval agent owns "the data is missing or stale." A reasoning agent owns "the interpretation is wrong." An action agent owns "the side effect failed or was unsafe." When scope is bounded, a failure is *attributable* — you know which agent to interrogate — and *contained* — it does not silently propagate. The stale-feed problem above stops being invisible the moment a retrieval agent whose entire job is freshness refuses to pass data forward without a timestamp it trusts. **2. Structured output contracts.** Agents communicate through typed, validated structures, not prose. Each agent declares the shape of what it returns — fields, types, required values, a freshness stamp — and the orchestrator validates against that schema before the next agent runs. A contract turns "the model phrased it differently today" from a production incident into a caught validation error. It is the same reason we use interfaces between software modules: the boundary is the thing you can reason about. **3. Hard time bounds.** Every agent and every step runs under an explicit deadline. An agent that has not produced a valid result within its bound is treated as having failed, deterministically, rather than being allowed to hang the pipeline or burn budget exploring. Time bounds convert the open-ended nature of agentic reasoning into something an operations team can plan around. They also expose a quiet failure class — the agent that is *technically* still working but has stopped making progress. **4. Independent verification — an agent never grades its own work.** This is the rule that most distinguishes a production system from a demo. The agent that performs a step does not get the final word on whether it succeeded. A separate verifier — a different agent, or deterministic code, reading the actual result against the actual criteria — confirms it. Self-grading fails for a structural reason, not a lazy one: the same model, prompt, and context that produced the answer also produce the self-assessment, so their errors are correlated. A check that shares the generator's failure modes cannot be an independent measurement. Research on models trained with human feedback documents a further compounding factor: stated confidence stops tracking accuracy as models are fine-tuned, with the model reporting higher certainty than the base model on the same questions (OpenAI, *GPT-4 Technical Report*, 2023). The instinct to "ask the agent if it's sure" is therefore worse than it looks. The production move is a separate verifier — a different agent or deterministic code, grading against the actual criteria on the actual result. The builder is never the verifier. **5. Liveness is not outcome.** "The agent ran" is an infrastructure signal. "The agent produced a correct, fresh, validated result" is the business signal — and it is the one you instrument and alarm on. A green health check tells you a process is alive; it tells you nothing about whether the work was done. Measuring the outcome, not the heartbeat, is what lets you trust a multi-agent system without watching it. A common seam where these rules earn their cost is retrieval. A retrieval-augmented agent that *looks* grounded can be confidently wrong in two ways a demo never surfaces. First, retrieving the right document is not the same as using it: models attend to a passage far better when it sits at the start or end of a long context than in the middle, and accuracy on multi-document tasks drops when the relevant passage lands in the middle (Liu et al., *Lost in the Middle*, TACL 2024). Second, retrieval reduces but does not eliminate unsupported claims — the answer can still assert details the source does not contain, which is why a faithfulness check decomposes the answer into claims and tests each against the retrieved source (Es et al., *RAGAS*, EACL 2024). Both failures are this page's thesis in a new domain: a fluent, plausible, wrong answer that no error log catches. The verifier checks the answer against the source, not the agent's confidence. The freshness and fail-closed mechanics behind this are the subject of our [data-integrity contract](/ai-data-integrity-contract/). ## The loop that holds it together: understand, design, implement, verify Bounded agents and contracts describe the *shape* of the system. The *motion* of it — how a single unit of work flows through — follows a loop: **understand → design → implement → verify.** A task is first understood (what is actually being asked, against what data, with what success criteria). It is then designed (which agents, in what order, with what contracts between them). It is implemented (the agents execute within their bounds). And it is verified independently against the criteria set at the start, not against the implementer's own report. The discipline is that the loop does not short-circuit. The most common production failure is skipping straight from "implement" to "declared done" — the agent reports success, the log is green, and no one closed the loop by checking the live outcome. "It ran" is not "it worked," in exactly the way "it merged" is not "it shipped." Treating evidence from the running system as the only acceptable definition of done is what makes the loop more than a diagram. ## When does multi-agent orchestration belong in production? Orchestration is not free, and pretending otherwise would be the kind of overclaim we avoid. A multi-agent system has more moving parts than a single prompt: more components to build, more boundaries to define, more latency from passing work between specialized agents, and an orchestration layer that itself must be engineered and operated. It also costs more to run — one credible public engineering account of a production multi-agent system notes that a multi-agent design can consume on the order of fifteen times the tokens of a single chat interaction, and is harder to evaluate and debug because behavior emerges across agents rather than sitting in one place (Anthropic, *How we built our multi-agent research system*, June 2025). For a genuinely simple, low-stakes, single-step task, a single well-scoped agent is the right tool, and adding agents is over-engineering. The calculus changes with stakes and complexity. As the cost of a wrong answer rises, and as the work spans multiple steps, sources, and side effects, the monolithic agent's hidden costs — undiagnosable failures, silent staleness, no separation between doing and checking — come to dominate. Orchestration trades up-front engineering for *legible* failure and *earned* trust. You pay in design effort, latency, and tokens; you are repaid in a system whose mistakes you can see, attribute, and contain. The judgment is knowing which regime you are in, and not paying for orchestration you do not need. | In production… | Single well-scoped agent | Orchestrated multi-agent system | |---|---|---| | **Best fit** | Simple, single-step, low-stakes work | Multi-step work across sources and side effects where a silent wrong answer is costly | | **Failure containment** | One scope, one failure class — legible enough on its own | Failures are attributable to a bounded agent and contained before they propagate | | **Cost and latency** | Low — one prompt, one call | Higher — more components, more hops between agents, and (on one public engineering account) up to ~15× the tokens of a single chat interaction | | **Verification** | Adequate when the agent's own output is the whole result | An independent verifier grades each step against real criteria; the builder is never the verifier | | **What it buys you** | Speed and simplicity | Legible, attributable, contained failure on the thousandth run | The full decision framework — when one agent is enough and when orchestration earns its cost — lives in [single-agent vs multi-agent](/single-vs-multi-agent/). ## What does production-grade multi-agent actually require? Production-grade multi-agent requires five things a demo agent lacks: a graded eval set, enforced guardrails, outcome observability, independent verification, and a data-integrity guarantee. The difference is not capability — a demo agent can be impressively capable — it is whether the system catches its own failures before they reach a user. The table makes the distinction concrete. | Dimension | Demo agent | Production-grade agent | |---|---|---| | **Evals** | "It worked when we tried it" — anecdotal, unrepeatable | A graded test set with agreed success criteria; regression-tested on every change | | **Guardrails** | Open scope, full tool access, trusts its own output | Scoped permissions, write actions gated, fail-closed on uncertainty, input/output validation | | **Observability** | Green health check ("the agent ran") | Outcome instrumented and alarmed — fresh, correct, validated result, not a heartbeat | | **Independent verification** | The agent grades its own work | A separate verifier — different agent or deterministic code — reads the real result against the criteria | | **Data integrity** | Numbers look like data, so stale passes silently | Every source carries a freshness stamp; an unavailable source says so rather than serving a confident wrong answer | None of these is exotic. Each is the unglamorous production engineering that separates a system you can trust with a number from one you cannot — the same contract described in our work on the [data-integrity contract](/ai-data-integrity-contract/). ## How do you evaluate a multi-agent system in production? You evaluate a multi-agent system in production by measuring the outcome, not the process — did fresh, correct, validated output appear — and by having something other than the agent that produced it confirm it. Self-grading confirms the narrative the model just told itself; only an independent check catches the partial result and the plausible-but-stale answer. A workable evaluation has four parts: 1. **A graded test set on real cases.** Agreed success criteria scored against actual examples, not a single happy-path demo run. 2. **Regression tests on every change.** So you know the moment a result slips, rather than discovering it from a customer. 3. **Outcome instrumentation, not liveness.** Alarm on "the work completed correctly and fresh," never on "the process is alive" — the distinction between liveness and outcome is the most common blind spot, covered in depth under [production AI: liveness vs outcome](/production-ai-liveness-vs-outcome/). 4. **An independent verifier in the loop.** A distinct, non-skippable step that reads the live result against the criteria set at the start. The builder is never the verifier. The measurement machinery behind this — the harness that runs graded evals on every change — is set out in our work on the [AI evaluation harness](/ai-evaluation-harness/), and the agent-specific version of it, including how to measure a confident-but-wrong rate, is covered in [how to evaluate an AI agent](/how-to-evaluate-an-ai-agent/). ## How to apply it Start by decomposing the job into roles, not tasks: what must be gathered, what must be decided, what must be acted upon, what must be checked. Give each role its own agent with a bounded scope and a named failure class. Define the contract between every pair of agents before writing the agents themselves — the schema is the interface. Put a hard time bound on every step. Add a verifier that reads the real result, never the performer's self-report, and that runs as a distinct step you cannot skip. Finally, instrument the outcome, not the heartbeat, so a stale or partial result raises an alarm rather than a clean summary. Built this way, a multi-agent system fails loudly and locally instead of quietly and globally — which is the whole point of running agentic AI in production at all. --- If you are moving agentic AI from a working demo to something you can trust in production, this is the discipline that gets you there. We are glad to think it through with your stack and constraints — start with [an AI working session](/book/), or explore [AI consulting](/ai-consulting/) to see how we take AI into verified production in your business. If your agentic pilot has already stalled on exactly this, that is what [AI Rescue](/ai-rescue/) is for. ============================================================================== # Single-agent vs multi-agent: when each architecture wins URL: https://newgenapps.com/single-vs-multi-agent/ ============================================================================== # Single-agent vs multi-agent: when each architecture is the right call **In short:** A single well-scoped agent is the correct architecture for a bounded, low-stakes job — adding agents to a task that does not warrant them is overhead, not maturity. Multi-agent earns its keep only when the work spans roles that should not grade each other and when the cost of a confidently wrong answer is high enough to justify more components, more latency, and an orchestration layer. The right question is not "how many agents" but "do the parts of this job need to be separated so they can check each other?" --- We have been making early architecture calls since 2009, when we built an augmented-reality demo on the G1 — the first Android phone — before the category had a name. The recording of that demo is still public (vimeo.com/618305787). The pattern established then is the same discipline this page is built on: read the signal early, build on the raw stack, and ask how much architecture a job actually warrants before committing to complexity that outlasts the problem. The agent architecture decision in 2025 is structurally identical. Multi-agent systems are technically possible; they are widely discussed; Gartner tracked a 1,445 percent increase in client inquiries about them from Q1 2024 to Q2 2025 (Gartner, *Multiagent Systems in Enterprise AI*, December 18, 2025). That same inquiry surge is paired with a Gartner prediction that over 40 percent of agentic AI projects will be canceled by the end of 2027, driven by escalating costs, unclear business value, and inadequate risk controls (Gartner press release, June 25, 2025, analyst Anushree Verma). Both data points are true at once, and together they say the same thing: the question is not whether multi-agent is a real pattern but whether a specific job warrants it. --- ## What is the difference between a single-agent and a multi-agent system? A single-agent system is one model loop that performs the whole job — it gathers, decides, acts, and evaluates its own work in a single thread of context. A multi-agent system splits that job across specialized agents with bounded scope, coordinated by an orchestration layer, so that the agent doing the work is not the only thing grading it. The deciding question is not "how many agents" but whether the parts of the job need to be separated so they can check each other — and whether the work decomposes into parts that can run independently at all. Two distinctions worth being precise about: **Bounded scope is the real unit, not agent count.** A second agent that shares the first one's context and goals is not a separate failure domain — it is the same loop with added latency. Multi-agent only buys you anything when each agent owns a distinct slice of the work: a role, a tool surface, a failure class that the others do not touch. Adding agents to appear more sophisticated, without that separation, is the over-engineering this page is designed to prevent. **The orchestration layer is a cost, not a capability.** The moment you have more than one agent, you have to engineer coordination: routing, context-passing, conflict resolution, retry logic, time bounds, and a way to attribute failure to the right agent. Naming it as overhead — not headline feature — is what keeps an architecture honest. Orchestration is the mechanism that makes independent verification work; it is not the point in itself. For a fuller treatment of the [agentic AI category](/agentic-ai/) — including autonomous agents, human-in-the-loop patterns, and where multi-agent sits in the broader agentic stack — see the [agentic AI overview](/agentic-ai/). For the production engineering of multi-agent systems once you have decided on the architecture — bounded scope contracts, structured handoffs, and independent verifiers — see [how we build reliable multi-agent systems in production](/multi-agent-orchestration-production/). --- ## When does a single agent win? Single-agent is the right architecture — not a starter version to outgrow — when the job is bounded to one role and draws on one source, carries low stakes if it outputs incorrectly, and would add latency and operational surface area without a commensurate verification gain by splitting it up. A single agent is the correct choice when all of the following hold: - **One bounded job, one source, one tool surface.** Summarise this document. Classify this ticket. Answer from this corpus. There is nothing to coordinate; coordination would be pure cost. - **The work is one coupled thread of decisions.** When later steps depend on the full context of earlier ones, splitting that thread loses context and lets sub-agents make conflicting implicit decisions. Walden Yan at Cognition described the failure mode in June 2025: "the decision-making ends up being too dispersed and context isn't able to be shared thoroughly enough between the agents" (Cognition, *Don't Build Multi-Agents*, June 12, 2025). The single linear agent keeps the whole trace in one place. - **The cost of a wrong answer is low or the output goes to a human reviewer before any action is taken.** When a confident error is cheap to catch and cheap to absorb, the independent-verifier overhead does not pay for itself. - **There is no genuine parallelism to exploit.** Anthropic noted in their June 2025 engineering post that "most coding tasks involve fewer truly parallelizable tasks than research" (*How we built our multi-agent research system*, June 13, 2025). When sub-tasks are not actually independent, multi-agent adds coordination overhead with no latency benefit. The practitioner observation that follows: adding a second agent to a coupled, bounded job does not make it more reliable. It makes it more fragile and more expensive, because you have introduced a coordination problem where none existed. The default discipline, as Anthropic described it in their December 2024 framework, is "finding the simplest solution possible, and only increasing complexity when needed" (Anthropic, *Building Effective Agents*, December 19, 2024). Optimize the single agent first — better tools, better retrieval, better in-context structure — before reaching for a second one. Common workloads that are correctly single-agent: document summarisation from a known source, drafting with human review, classification against a defined schema, extraction from a single structured input. --- ## When does multi-agent win? Multi-agent earns its overhead when work spans roles that should not grade each other — gathering, deciding, acting, and checking are steps that benefit from separation — and when the cost of a confidently wrong answer is high enough to justify the added components and latency. Two conditions both need to hold: **The separation condition (the verification argument).** The work spans roles that should not grade each other: a researcher, a drafter, a verifier, an executor. No agent should both perform a step and be the sole judge of it. The payoff of that separation is legible failure — when something goes wrong, the orchestrator can attribute the failure to a specific agent and a specific step, rather than producing one confident, fluent, wrong answer with no visible seam. The irreversibility of the downstream action matters here: when a wrong answer triggers a filed record, a sent communication, or a triggered workflow, independent verification is no longer optional. **The decomposition condition (the parallelism argument).** The job has to break into parts that can run independently. Anthropic reported that its multi-agent research system outperformed a single-agent baseline on internal evaluations because research is breadth-first work — many independent sub-queries across many sources, with no shared write state for sub-agents to corrupt (*How we built our multi-agent research system*, Anthropic, June 13, 2025). The same architecture is a poor fit for a tightly-coupled planning task, where later steps depend on the full context of earlier ones. Workloads where multi-agent commonly earns its keep: research that requires querying many independent sources; long-horizon tasks that exceed a single context window; pipelines where one agent's output becomes another agent's input and independent verification of each handoff is required; tasks that interface with multiple distinct tool surfaces or permission domains. The broader production context: the difficulty of getting agentic systems into production makes the architecture decision consequential before any code is written. IDC's *Lenovo CIO Playbook 2025* (February 2025) found that approximately 88 percent of AI proofs of concept do not reach widescale deployment. Choosing the right architecture is part of why that gap exists — over-engineered systems that cannot be debugged, debugged systems that should not have been multi-agent in the first place. One important guard against a common error: multi-agent does not fix a retrieval failure. If a single agent is failing because the right passage never reaches it, adding agents multiplies that retrieval problem across more outputs. The architectural question and the retrieval quality question are separate. For the measurement layer underneath this — how you confirm an agent is verifying correctly, what a confident-but-wrong rate looks like in practice — see the [AI evaluation harness](/ai-evaluation-harness/) and the [guide to evaluating an AI agent](/how-to-evaluate-an-ai-agent/). --- ## What does multi-agent add — and what does it cost? Multi-agent adds independent verification and legible failure. It costs more components, higher latency, and an orchestration layer that must itself be engineered, monitored, and maintained. The ledger: **What it adds:** bounded failure domains (a mistake is attributable to one agent and contained); an independent check (a verifier that no working agent grades past); parallelism on genuinely independent sub-tasks; the ability to exceed a single context window. **What it costs:** more components to build and operate; higher latency because steps are sequential or semi-sequential; materially more tokens. Anthropic measured this directly: agents use roughly four times the tokens of a chat interaction; multi-agent systems roughly 15 times, with token usage explaining approximately 80 percent of the variance in research-task performance (*How we built our multi-agent research system*, Anthropic, June 13, 2025). At a roughly 15 times cost multiple, the task must justify the investment. **The payoff framed correctly:** what you are buying with multi-agent is legible failure. A multi-agent system that fails visibly and attributably is operationally superior to a single-agent system that fails silently, even if the absolute error rate is similar. The ability to see which step failed, attribute it, contain it, and correct it — without re-running the whole pipeline from scratch — is the production value. For the production engineering that delivers that legibility — bounded scope contracts, structured output handoffs, time bounds, and an independent verifier no agent grades past — see [multi-agent orchestration in production](/multi-agent-orchestration-production/). One caution when reading any benchmark: part of what a multi-agent system "wins" can be the extra computation it is given, not the coordination itself. A second agent spends more tokens and more wall-clock time, and on a task that does not decompose, that spend buys little. The case for multi-agent should not rest on coordination as a mechanism in the abstract. It rests on what the added compute buys on the specific job — independent verification and legible failure on work that genuinely breaks apart. --- ## How do you choose? The decision rule and the comparison table ### Three questions that determine your architecture **1. Does the job decompose into parts that can run independently?** If no — the work is one coupled thread of decisions that must share full context — stay single-agent. Splitting a coupled thread disperses context and invites the conflicting implicit decisions Cognition documents (June 12, 2025). If yes, continue. **2. Does the work span roles that should not grade each other, and is the cost of a confidently wrong answer rising?** If no — one bounded, low-stakes job where a human reviews the output before any action — single-agent is correct. It is not the less mature choice; it is the right choice. If yes, continue. **3. Is the value of the task high enough to justify the overhead?** More components, more latency, a roughly 15 times token multiple in the full multi-agent case, and an orchestration layer to build, monitor, and maintain. If yes — multi-agent earns its keep; engineer it for legible, attributable, contained failure. If no — a single well-scoped agent is the right call. The one-line version: **the question is never "how many agents" — it is "do the parts of this job need to be separated so they can check each other, and do they decompose enough to run apart at all." If not, one agent is the senior choice.** ### Single-agent vs multi-agent: the decision table | Dimension | Single-agent | Multi-agent | |---|---|---| | **Best for** | One bounded job, one source, one coupled thread of decisions with a clear end state | Work that decomposes into genuinely independent parts and spans roles that should not grade each other | | **Decomposition / parallelism** | Nothing to parallelize; coordination is pure overhead | Independent sub-tasks run in parallel; can exceed a single context window; breadth-first research across many sources | | **Failure isolation** | One bad input contaminates the whole chain; the error has no seam to inspect | Bounded scope — a failure is attributable to one agent and contained at that step | | **Verification** | Self-grades; the only check is the agent's own judgment | An independent verifier — agent or deterministic code — reviews the output of each upstream step; no agent grades its own work | | **Dominant failure mode** | Confident-but-wrong output with no internal check; one bad step propagates with no seam to catch it | Dispersed decisions and dropped context across agents — the coordination tax Cognition documents (Jun 12, 2025) | | **Cost / latency** | Lowest: one model loop, minimal infrastructure, lowest token spend | More components, higher latency, roughly 15× tokens in the heavy case (Anthropic, Jun 13, 2025), plus an orchestration layer to engineer and operate | | **Operational legibility** | One loop — hard to attribute which step failed | Each step is a discrete checkpoint; failures are visible, attributable, and containable | | **Orchestration overhead** | None | Real: prompt contracts, handoff schemas, retry logic, time bounds, observability tooling — all must be engineered | | **When it is the WRONG choice** | When verification is required that the agent cannot credibly provide for itself; when steps should not grade each other; when a wrong answer triggers an irreversible downstream action | When the task is bounded and low-stakes, or is one tightly-coupled thread that loses context and gains fragility when split; when latency is the binding constraint and the added cost cannot be justified | *This table supersedes the shorter comparison in [agentic AI systems](/agentic-ai/), which answers "what is the difference." This table answers "which do I choose, and when is each the wrong call."* Note on the dominant-failure-mode row: the practical asymmetry is between an error that is caught and one that is not. Between independent agents with no central check, a wrong sub-output can pass downstream and compound; under an orchestrator that inspects and can reject sub-outputs before they propagate, the same error is contained at the step where it occurred. That containment is the reason to engineer the orchestration layer rather than simply add it — and the judgment call sits on top: is the task worth building that layer at all? See [how we approach judgment-not-complexity decisions](/how-we-work/) for the method that informs this framework. --- ## Is your job single-agent or multi-agent? A second opinion before you build. The architecture decision is consequential before any code is written. Getting it wrong in either direction — over-building with multi-agent when one agent suffices, or under-building with single-agent when independent verification is required — costs time, budget, and trust in AI delivery. A single well-scoped agent is not the beginner version of a multi-agent system. It is the correct version for a bounded job, and choosing it on purpose is the senior move. You add agents to separate doing from checking and to run independent work in parallel; you do not add them to signal that your AI work is advanced. We are model-flexible by design — because the architecture decision should follow the workload, not the vendor. Production AI, proven, means committing to the simplest design the job warrants and then showing, through independent verification and evaluated reliability, that it works. If you are choosing an agent architecture and want a senior second opinion before you commit to complexity, that judgment call is what an [AI working session](/book/) is for. Or see how we design and deliver AI systems into production in [AI consulting](/ai-consulting/). --- ## Frequently asked questions **What is the difference between single-agent and multi-agent AI?** A single-agent system gives one agent the entire job; it reasons, acts, and checks its own output within one context. A multi-agent system splits the job across specialized agents, each with bounded scope, coordinated by an orchestration layer, so that no agent both performs a step and independently verifies it. The core distinction is structural separation between action and verification. **When should I use a single agent instead of multiple agents?** Use a single agent when the job is bounded to one role and one source, the work is one coupled thread of decisions that must share full context, a human reviews the output before any action is taken, and adding agents would increase complexity without adding a meaningful verification gain. Single-agent is not a less mature choice; it is the correct choice for a bounded, low-stakes task. **What does multi-agent AI actually add?** Multi-agent adds independent verification — a separate agent reviewing work that another agent produced — and legible failure: when something goes wrong, the orchestrator can attribute it to a specific step and a specific agent. The cost is more components, higher latency, and an orchestration layer that requires engineering and maintenance. Anthropic's own measurement found multi-agent systems use roughly 15 times the tokens of a chat interaction (June 2025). The task must justify that multiple. **How many agents does a production AI system need?** The right number is the minimum needed so that no agent is both performing a step and grading its own output on a consequential task. For many bounded business tasks, that is one. For tasks that span gather, decide, act, and check across high-stakes outputs with genuinely parallelizable sub-tasks, a small number of specialized agents with one orchestrator is a common pattern. More is not better; separation with a clear purpose is better. **What is agent orchestration?** Agent orchestration is the coordination layer in a multi-agent system. It routes tasks to the appropriate agent, enforces output contracts between agents, manages retries and time bounds, and surfaces failures to a human operator or a downstream system. Orchestration is the mechanism that makes independent verification work in practice — not a feature in itself, but the infrastructure that makes multi-agent systems operationally legible. It is a cost that multi-agent systems carry by definition; whether the task justifies that cost is the architecture question. ============================================================================== # RAG in production: retrieval reliability, not just retrieval URL: https://newgenapps.com/rag-in-production/ ============================================================================== # RAG in production: retrieval reliability, not just retrieval **In short:** In a demo, a RAG system is graded on the queries you tried. In production, it must retrieve well on queries you did not try, keep doing so as the underlying documents change, and decline to answer when the evidence is not there. The failure mode almost never originates in the language model — it originates in retrieval. Measuring retrieval quality and answer-groundedness continuously, and gating on those numbers, is the discipline that separates a RAG demo from a RAG product. --- NewGenApps ships AI into production and proves it works — instrumented, monitored, evaluated against ground truth, including systems answering to real users in regulated settings. That posture is applied here to retrieval. What follows reflects how retrieval reliability is actually held in production, not how it looks in a notebook. If you are newer to what RAG is, how it differs from fine-tuning, and where it fits in an AI architecture, start with [our overview of RAG, RAG vs fine-tuning, and common failure modes](/rag/). This page picks up where that one ends: at the point where a retrieval system meets real users. --- ## What changes when RAG goes to production? In a demo, retrieval quality is graded on the handful of queries the team tried; in production, retrieval quality is the variable that determines whether the system is trustworthy, and it degrades silently as the corpus changes underneath it. In a demo, the corpus is small and fixed, queries are hand-picked, and the verdict — "that looks right" — is reached by eye on a sample of perhaps five questions. The system passes because no one tried the questions that would have exposed its retrieval gaps. In production, the picture is different in every dimension. The query distribution is set by real users, not by the team that built the system. Documents are added, updated, and deprecated continuously. The correctness bar is held by users and, in regulated contexts, auditors. And crucially: the failures that were invisible in the demo now surface — not as retrieval failures, which would be diagnosable, but as complaints that "the AI is hallucinating." Most of those hallucination complaints are retrieval failures in a costume. If the passage that contains the answer never reaches the model, no model can answer correctly; it will instead produce a fluent, plausible, ungrounded guess. The production move is not to get a better model. It is to [measure retrieval quality and answer-groundedness continuously, and gate on them](/ai-evaluation-harness/) — the same evaluated-reliability posture that applies across the cluster, pointed at retrieval. --- ## Why do production RAG systems fail? Production RAG failures almost always trace back to retrieval — the right passage never reached the model — not to the model generating something false from good evidence. Here is the diagnostic taxonomy, ordered by how a practitioner should triage: **1. Retrieval recall failure, misread as hallucination.** When the right passage is absent from the retrieved set, the model has nothing to ground on and fills the gap. The fix is not the model; it is recall. A hallucination complaint should trigger a retrieval audit first, a model investigation second. **2. Chunking that splits the answer.** If a chunk boundary falls through the middle of the fact, no single retrieved chunk contains the whole answer, and similarity search returns half-answers. Chunk size and overlap determine whether an answer is retrievable in one piece at all. This is not a cosmetic preprocessing decision. (Barnett et al., *Seven Failure Points When Engineering a Retrieval Augmented Generation System*, IEEE/ACM CAIN 2024 / arXiv:2401.05856 — failure point "Not in Context," where the passage carrying the answer fails to survive into the consolidated context.) **3. No reranker.** First-pass retrieval returns passages ranked by approximate similarity, not by whether they actually answer the specific question. Without a reranker, the most-relevant evidence can sit outside the position the model uses best. Skipping rerank is one of the most common reasons a demo that "retrieves the right thing" degrades on the long tail. **4. Dense-only retrieval fails out-of-domain.** BM25 (lexical) retrieval remains a strong baseline and frequently outperforms dense retrievers on out-of-domain queries — in-domain evaluation is not a reliable predictor of out-of-domain performance (Thakur et al., *BEIR*, NeurIPS 2021 / arXiv:2104.08663). Hybrid retrieval — semantic plus keyword, then rerank — is the production default: keyword catches codes, proper nouns, and acronyms that embeddings blur; semantic catches the paraphrases keyword misses. **5. Position failure — the right chunk reached the model, and still wasn't used.** Performance on multi-document QA degrades by more than 20 percentage points when the relevant passage sits in the middle of a long context, a U-shaped curve that persists in long-context models (Liu et al., *Lost in the Middle*, TACL 2023 / arXiv:2307.03172). Adding more chunks to a long context window is not a substitute for precise retrieval and rerank: Databricks AI Research confirmed that most models' RAG performance declines past a context-length threshold (Leng et al., arXiv:2411.03538, November 2024). Retrieval hit-rate is not answer-faithfulness. **6. No abstention.** A system not instructed to say "I don't know" when retrieved evidence is absent will generate anyway, from thin or absent context. A confident wrong answer is the expensive failure in production; "the documents don't cover this" is often the more valuable behavior. Abstention is a designed capability — a retrieval-score threshold below which the system declines — not a default. **7. Citations that don't survive the click.** An answer can name a source that doesn't support the attributed claim. The discipline that catches this is a faithfulness metric: decompose the answer into atomic claims and check each against the retrieved evidence. RAG reduces hallucination; it does not eliminate it. A corpus of nearly 18,000 naturally-generated RAG responses, span-annotated at the word level, found that even when relevant context was retrieved and placed in front of the model, LLMs still presented unsupported or contradictory claims relative to the retrieved content (Niu et al., *RAGTruth*, ACL 2024 / arXiv:2401.00396). Faithfulness is a measured number, not a vibe. [Stale or ungoverned source data](/ai-data-integrity-contract/) is a retrieval-reliability failure before it is a model failure. **8. No eval harness, and ignored access control.** Without a standing harness, every change to the chunker, embedding model, top-k, reranker, or prompt is an unmeasured bet. And access control is a retrieval-reliability failure, not just a security concern: if the retriever surfaces documents a given user is not permitted to see, the system is reliably retrieving the wrong thing. The permission model must be enforced in the retriever — before ranking — not assumed by the language model. The synthesis: in production RAG, the failure is almost never the model. It is recall, chunking, position, faithfulness, abstention, drift, or access. Every one is measurable. "The AI is hallucinating" is the symptom; the diagnosis is in the retrieval pipeline. --- ## How do you measure RAG reliability? RAG reliability requires two measurement tracks — retrieval quality (did the right passages come back?) and answer-groundedness (is the response supported by what was actually retrieved?) — measured continuously against a held-out evaluation set, not once at launch. A RAG system has two failure surfaces with two different fixes: - If **retrieval recall is low**, the evidence never arrived. No generation improvement will help. Fix chunking, go hybrid, add rerank, raise top-k, fix index freshness. - If **retrieval recall is high but faithfulness is low**, the right evidence reached the model and the generation drifted from it. Fix the prompt, address position effects, or add a faithfulness gate. A better embedding model will not help. A single blended accuracy score hides which surface is broken. The production move is two dashboards, two gates. This is the [same measure-don't-vibe-check discipline applied to agent outputs](/how-to-evaluate-an-ai-agent/) and to AI systems more broadly via [the standing eval harness](/ai-evaluation-harness/) — here, pointed at retrieval. **Retrieval quality metrics:** - **Recall@K:** of the passages that contain the answer, how many appeared in the top K retrieved? - **Context precision:** of the top K retrieved, how many are actually relevant? Low precision means the context is noisy, which degrades answer quality even when recall is high. - **MRR (Mean Reciprocal Rank):** is the first relevant passage near the top of the retrieved set? - **Rerank quality:** does re-ranking the first-pass candidates move the most-relevant passage to the top position, measured against the evaluation set? (Recall@K, MRR, and rerank quality are standard IR metrics; context precision is one of RAGAS's four dimensions — faithfulness, answer relevancy, context precision, context recall — per Es et al., *RAGAS*, EACL 2024 / arXiv:2309.15217.) **Answer-groundedness (faithfulness):** - Does each claim in the generated response trace to a retrieved passage? - A system grounded by default generates only from what was retrieved, and declines when that retrieved evidence is insufficient. - Citation integrity: does the cited passage actually contain the claim attributed to it? Vectara's faithfulness work, benchmarking across leading models, confirms that LLMs still frequently introduce unsupported claims even when provided with relevant context (Tamber et al., *Benchmarking LLM Faithfulness in RAG with Evolving Leaderboards*, EMNLP Industry 2025 / arXiv:2505.04847). This is the reason faithfulness requires its own metric, measured independently of retrieval recall. Groundedness as an outcome signal — as distinct from a green pipeline — is the same distinction covered in [production AI liveness vs outcome](/production-ai-liveness-vs-outcome/). **How to measure RAG reliability: a working checklist** 1. **Build a held-out question set from real, production-shaped questions** — including the messy, the long-tail, and the questions no one tried in the demo — each paired with the passage(s) that should answer it. Freeze and version it. The set is the ruler; if it drifts with the system, the score is meaningless. 2. **Measure retrieval first, in isolation.** Recall@K, context precision, and MRR on the evaluation set. This is the number that catches the hallucination that is really a retrieval-recall failure. 3. **Measure answer faithfulness separately.** Decompose each answer into claims; score the fraction the retrieved evidence supports. Automate this check using a grounding-verification step. 4. **Measure abstention behavior.** On questions whose answer is not in the corpus, does the system decline rather than confabulate? Track the false-answer rate on out-of-corpus questions as its own metric. 5. **Set thresholds and gate on them in CI.** A number with no release gate is a dashboard, not a control. Every change to chunker, embedding model, top-k, reranker, or prompt re-runs retrieval recall, faithfulness, and abstention — and blocks the release on regression. This is [the eval-harness mechanics](/ai-evaluation-harness/) applied to retrieval. 6. **Re-validate as the corpus changes.** Documents are added, edited, deprecated; the retriever's behavior shifts. Re-sample the evaluation set from current traffic, re-check faithfulness against current sources, and re-freeze. [Stale or ungoverned source data](/ai-data-integrity-contract/) is a retrieval-reliability failure upstream of the model. --- ## How do you keep a RAG system reliable over time? The primary threat to a production RAG system is corpus drift — documents are updated, added, and deleted, and retrieval quality measured at launch no longer reflects current behavior — so the discipline is a standing eval harness, not a one-time green build. A knowledge base that was clean and well-chunked at launch accumulates stale, conflicting, and missing documents. Retrieval quality degrades without any change to the model or retrieval code. Barnett et al., studying RAG failure points across three domains, found that "validation of a RAG system is only feasible during operation" and that "robustness evolves rather than is designed in at the start" (*Seven Failure Points When Engineering a Retrieval Augmented Generation System*, IEEE/ACM CAIN 2024 / arXiv:2401.05856). The implication is that a launched RAG system requires ongoing calibration — not a one-time assessment. **Re-evaluation is triggered by:** - Bulk document updates or index refreshes - Shifts in the live query distribution (new user question types that the frozen set no longer represents) - Time-based schedule regardless of corpus changes **Grounded-by-default as a reliability property.** A system that abstains when evidence is absent degrades gracefully — it surfaces the gap for remediation rather than generating confidently from stale context. An abstention is a quality signal, not a failure. **Access control as a reliability concern.** If the permission model changes — users added, documents reclassified — the retrieval layer must reflect those changes. Evaluation should include permission-boundary test cases: can a user retrieve documents they are not entitled to? A retrieval layer that ignores entitlements will, reliably, surface the wrong thing for some users. **The signal being watched.** The metric is not "is the pipeline green" — that is liveness, not outcome. It is: are retrieval recall, context precision, and answer-groundedness above threshold on the current corpus against the current evaluation set? See [the standing eval-harness infrastructure](/ai-evaluation-harness/) and [production AI liveness vs outcome](/production-ai-liveness-vs-outcome/) for how this posture is applied across the delivery method described in [how we work](/how-we-work/). --- ## RAG in a demo vs RAG in production The gap between a RAG demo and a production RAG system is not model capability — it is the engineering disciplines that make retrieval measurably reliable at the scale and diversity of real use. This table is distinct from [the RAG vs fine-tuning vs long-context comparison on our overview page](/rag/). That table is about approach selection; this one is about production readiness. | Dimension | RAG in a demo | RAG in production | |---|---|---| | **Retrieval quality** | Graded on the handful of queries the team tried | Measured recall and context precision on a held-out evaluation set covering the real query distribution, including the long tail | | **Faithfulness** | "Looks grounded" — assessed by eye | Scored: each claim traced to a retrieved passage; citations that survive the click; grounding check automated | | **Abstention** | System answers every query, including those beyond the available evidence | System declines when retrieved evidence is absent or below a confidence threshold | | **Corpus drift** | Static at demo time; evaluation not revisited | Corpus changes trigger re-evaluation; reliability is a continuously watched signal | | **Access control** | Single user or open corpus; permissions not tested | Retrieval respects the permission model; evaluation includes permission-boundary cases | | **Reliability signal** | A green pipeline — it ran without error | Retrieval recall, context precision, and faithfulness scores above a defined threshold, measured on schedule | The last row is the page's positioning in miniature: "ran without error" is [liveness](/production-ai-liveness-vs-outcome/); "the retrieval and faithfulness numbers held" is outcome. --- ## The missing layer most pilots don't have If your "chat with your documents" pilot answers confidently but you cannot put a number on how often it is grounded, that is the missing layer. If a retrieval-based system has stalled before reaching production, that is a signal worth diagnosing before rebuilding. See how NewGenApps ships and proves RAG in [how we work](/how-we-work/), or if a pilot has already stalled, [see our AI rescue work](/ai-rescue/). [Book a 30-minute working session](/book/) to talk through what is missing in your specific setup. *NewGenApps — production AI, proven. Stay a step ahead, always.* ============================================================================== # Foundation Models vs Bespoke Training: When to Stop Retraining URL: https://newgenapps.com/foundation-models-vs-bespoke-training/ ============================================================================== # Foundation Models vs Bespoke Training: When to Stop Retraining **In short:** at the data sizes most real projects run on, a foundation model — used well, with prompting and retrieval — often beats a custom-trained model outright, especially for language and code tasks. Bespoke training still wins in a narrower band: when the target distribution sits far from anything in pretraining, when unit cost or latency at scale is dominated by raw model size, when the data is proprietary in a way no prompt can convey, or when a hard constraint (on-premises, no external inference, strict determinism) rules out a hosted model. And once a model is in production, retrain it on evidence of drift, not on a calendar — a fixed schedule is usually solving a problem that was never there. There is a reflex in applied machine learning, inherited from the decade before foundation models, that treats a custom-trained model as the serious answer and anything off-the-shelf as a placeholder. That reflex is now frequently wrong, and it is expensive. The question worth asking on most projects is no longer "what should we train?" but "what makes us think we need to train at all — and if we retrain on a schedule, what is that schedule actually tracking?" This piece sets out where bespoke training and fine-tuning still earn their keep, where foundation models quietly win, and how to set retraining cadence so it follows the decay of signal in your data rather than the rhythm of a cron job. The discipline behind it was forged operating a production, AI-orchestrated system at the hard end of reliability, then generalized for client delivery. ## When should you use a foundation model vs bespoke training? The decision is usually framed as a binary — foundation model or custom model — when it is really a question about where the marginal information lives. A pretrained foundation model arrives having already learned an enormous amount of structure about language, code, images, and the world. A task-trained model starts from far less and must recover that structure from your data alone. The relevant comparison is not "general versus specialized" in the abstract; it is whether the specialization you need is *already latent* in the foundation model and reachable through prompting, retrieval, or light adaptation — or whether it genuinely requires new weights. At the data sizes most organizations actually have — thousands to low millions of examples, not billions — the foundation model often holds the advantage, because the thing your bespoke model would have to learn from scratch is mostly the thing the foundation model already knows. Custom training pulls ahead in a narrower band than people assume: when the target distribution is far from anything in pretraining, when latency or unit cost at scale is dominated by model size, when the data is proprietary in a way no amount of prompting can convey, or when a hard constraint (on-premises, no external inference, strict determinism) rules out a hosted model. Outside that band, you are usually paying to rediscover general knowledge. | Dimension | Foundation model (prompted / RAG / lightly adapted) | Bespoke-trained model | |---|---|---| | Data needed | Works from zero to a few hundred examples; retrieval adds your proprietary context without retraining | A meaningful volume of labeled, representative data — usually more than teams first estimate | | Upfront cost | Lowest — a prompt and a retrieval pipeline can ship in days | Highest — data pipeline, training compute, evaluation harness | | Ongoing maintenance | Low, until the base model is swapped or deprecated | A standing cost: retraining cadence, drift monitoring, regression risk on every update | | Reversibility | High — a prompt change ships in an afternoon | Low — a training pipeline is a liability carried for the life of the system | | Where it tends to win | Language and code tasks at realistic (thousands to low-millions) data sizes, where the needed knowledge is already latent in pretraining | Target distribution far from pretraining, unit cost/latency dominated by model size, proprietary data no prompt can convey, or hard deployment constraints (on-prem, no external inference, strict determinism) | ## Why do teams keep retraining models that don't need it? The most common and least examined waste is not the initial build decision. It is the standing assumption that a model in production must be retrained continuously, or at least monthly, because that is what mature teams are seen to do. The behaviour gets copied without the conditions that justified it — a textbook cargo cult. Consider a non-proprietary, illustrative example: a content-classification model that routes incoming documents into categories. The team ships it, then sets up a weekly retraining pipeline because the data "keeps coming in." Each week the pipeline ingests the latest labelled examples, retrains, runs an offline accuracy check, and promotes the new model. For months, accuracy on the offline test set wobbles within noise — sometimes up a fraction, sometimes down — and the team reads the occasional uptick as vindication. What is actually happening is that the underlying category structure is stable; the documents this month look like the documents last month. The model is being rebuilt to learn nothing new. The cost is real and recurring: compute for every run, engineering time to babysit the pipeline, and a subtler tax — every retrain is a fresh opportunity to introduce a regression, a data-leak, or a silent distribution shift that the offline check fails to catch. A model that did not need to change has been handed a weekly chance to break. The mirror-image failure is just as damaging: a team retrains quarterly because the calendar says so, while the real-world distribution shifts weekly — a fraud pattern, a pricing regime, a user behaviour — and the model is stale for most of its life without anyone measuring it. In both cases the cadence is decoupled from the only thing that should govern it: how fast the signal in the data decays. ## When should you retrain a model — and how often? Retraining is a response to **drift** — the divergence between the distribution a model was trained on and the distribution it now serves. The correct cadence is therefore an empirical property of your problem, not a default. The discipline has three parts. **Measure drift before you act on it.** Instrument the live system to detect distribution shift directly — changes in input feature distributions, in the model's output mix, and, where ground truth eventually arrives, in realized accuracy against fresh labels. This is the difference between knowing your model is stale and assuming it. It also exposes the comfortable lie of the offline test set: a fixed historical test set cannot tell you the world has moved, because it has not moved with it. **Retrain on a trigger, not a timer.** Once drift is measured, the cadence becomes a consequence: retrain when drift crosses a threshold tied to a business-relevant change in outcome, not when a fixed interval elapses. Some models, on stable distributions, will go a long time untouched and that is the correct, frugal answer. Others will need frequent updates, and the same measurement tells you so honestly. Either way the schedule is *derived*, defensible, and auditable, rather than assumed. **Treat liveness and outcome as different signals.** A retraining pipeline that runs green every week is an infrastructure signal — the job executed — not a business one. What matters is whether the model in production is producing correct outcomes on current data. Alarm on the outcome, not on the ping. A pipeline can succeed flawlessly while shipping a model that is quietly worse, and a model can be performing perfectly while its retraining job has silently stopped. Conflating the two is how teams end up confidently wrong. ## When is bespoke training still worth it? None of this is free, and a senior reader will want the costs named. Drift instrumentation is itself engineering work; it adds monitoring surface and demands fresh ground truth that some problems supply only with a lag, or never. A trigger-based cadence can feel less predictable to a planning function than "we retrain every month," and it requires agreeing thresholds up front — a negotiation, not a default. And there are genuine cases where a simple periodic retrain is the pragmatic choice: when drift monitoring would cost more than the occasional unnecessary run, or when regulatory expectations demand a documented, regular refresh regardless of measured need. The point is not to ban scheduled retraining; it is to make the schedule a decision someone can defend, with evidence, rather than a ritual no one questions. The same honesty applies to the build decision. Fine-tuning a foundation model sits between the two poles and is often the right middle: you keep the pretrained knowledge and adapt the margins, at a fraction of the data and cost of training from scratch. But fine-tuning carries its own standing cost — every base-model upgrade potentially invalidates your adaptation, and you inherit a maintenance commitment that prompting and retrieval do not impose. Before committing weights, it is worth exhausting the cheaper levers: a stronger prompt, better retrieval over your proprietary data, a tool the model can call. They are reversible in an afternoon; a training pipeline is a liability you carry for the life of the system. ## How do you decide, in practice? A workable sequence for any team facing this choice: 1. **Establish the foundation-model baseline first.** Before scoping any training, measure how far a capable foundation model gets with prompting and retrieval over your own data. This number reframes everything that follows; surprisingly often it is good enough, and the build decision dissolves. 2. **Identify the genuine gap, if any.** Where the baseline falls short, characterize *why* — distribution distance, latency, unit cost at scale, a hard deployment constraint. Only a gap with a named mechanism justifies new weights. 3. **Prefer the lightest adaptation that closes the gap.** Prompt and retrieval before fine-tuning; fine-tuning before training from scratch. Each step up is a larger and more permanent cost. 4. **Instrument drift before you schedule anything.** Build the measurement before the retraining pipeline, so cadence is derived from signal decay rather than guessed. 5. **Promote on evidence.** Move a new or retrained model forward only when it clears explicit, agreed criteria on fresh data and is independently verified — the builder of a change should not be the sole judge of whether it works. The thread running through all of this is that AI infrastructure cost is the sum of decisions, most of which are made by default and never revisited. The teams that spend well are not the ones with the largest training budgets; they are the ones who can say, for every model in production, exactly why it exists, why it retrains on the cadence it does, and what evidence would change that. On model choice we stay deliberately flexible — an orchestra conductor, not a single instrument. Our own depth sits one layer up, in how we build: **Claude Code and a library of custom skills**, the agent-driven workflow that lets a small senior team ship and verify production systems fast, running on AWS — Bedrock, SageMaker, GPU compute, Kiro — accelerated by NVIDIA, which is what keeps the model layer itself replaceable. The instrument matters less than the discipline of knowing when not to play it. If you are carrying a retraining pipeline you have never questioned, or weighing a custom build against a foundation model and want a clear-eyed read on which is the cheaper path to the same outcome, a focused [AI working session](/book/) is a good place to pressure-test it. For longer engagements where the cadence and cost questions recur across many models, our [AI consulting](/ai-consulting/) brings the same proof-based discipline to your stack. ============================================================================== # The Timestamped-Foresight Record: More Than Fifteen Years of Dated Technology Calls | NewGenApps URL: https://newgenapps.com/the-timestamped-foresight-record/ ============================================================================== # The Timestamped-Foresight Record: More Than Fifteen Years of Dated Technology Calls NewGenApps (New Generation Applications Pvt. Ltd., incorporated in 2009) holds a timestamped record of moving on each generation of platform technology before the category had an agreed name: cloud infrastructure adopted in 2007–08 while AWS was still in beta, before Elastic Block Store launched; iPhone-OS development built the moment Apple disclosed its frameworks; an augmented-reality demonstration on the first Android handset in 2009. Each call is anchored to a retrievable, dated artifact — a post in the firm's public blog (spanning 2008 to the present) or a documented firm action. The record is presented here in full, as a first-party dataset, because the judgment that read those signals early is the judgment now applied to a harder problem: moving AI out of the pilot stage and into production that can be proven to work. The pattern is the proof. ## In short NewGenApps has a public record, dated from October 2008, of moving on platform technologies before each became a recognized category. The firm's blog opened in October 2008 naming mobile and cloud together as the coming structural shift; it bet on iPhone and Android over BlackBerry and Symbian in writing by April 2009, before those incumbents declined; it built and demonstrated augmented reality on the G1 — the first Android phone — in 2009; and it tracked big data and machine learning in its public archive as those categories matured. The same posture — read the durable signal before consensus forms, learn the raw stack on the day it ships, put something real into production early — is what the firm now applies to production AI: not pilots that look good on a slide, but working systems with verified outcomes. Production AI, proven. ## Why a dated record of technology calls is worth examining A firm's record of *when* it moved — not just that it moved — is the only verifiable proxy for judgment quality. Retrospective claims of early adoption are easy to make; timestamped public artifacts are not. The difference is between an assertion ("we saw this coming") and a receipt. An assertion can be written today and backdated in the telling. A receipt — a dated blog post, a recorded demonstration, a documented infrastructure decision — carries a timestamp that cannot be moved. That distinction is the whole point of presenting this as a dataset rather than a story: every node is checkable against the public record by anyone who wants to check it. The firm's public blog archive, spanning 2008 to the present, is that record. It is a continuous, retrievable signal going back to the founding, across each platform generation — and it is a primary source NewGenApps holds by virtue of having published, in real time, since October 2008. This section rests on the logic of that argument alone, not on any external statistic. ## What the record proves — the analytic thesis A record of moving early on technology shifts is not a collection of lucky bets. It is timestamped evidence of a repeatable judgment: reading a weak signal before the category has a name, learning the raw stack on the day it ships, and putting something real into production before consensus forms. The firm's public record carries that pattern across several shifts — mobile platforms in 2008, cloud (AWS adopted in 2007–08 while in beta, before EBS launched), augmented reality (a demonstration on the G1, the first Android phone, in 2009), and the later big-data and machine-learning waves — each documented at the time, not claimed in hindsight. The same judgment that picked iPhone and Android over BlackBerry and Symbian before the outcome was obvious is the judgment a buyer needs now to tell which AI capability is production-ready and which is a demo. The wave is different; the discipline is the same. This framing survives a skeptical read for one reason: the claim is not "we are always right about technology," which is unfalsifiable and reads as marketing. The claim is the *mechanism* — early signal-reading, raw-stack learning, ship-before-consensus — and each instance is timestamped so it cannot be retrofitted. The qualifier strengthens it rather than weakening it: not every early call pays off, and this record makes no such claim. The receipts that survive are the ones with a public date. That is why a dated archive is better evidence than a memory. ## The timestamped record, 2008–2017 The table below is the dataset. Each row is a documented call — a platform NewGenApps moved on early, anchored to a dated artifact. The Source column states whether the receipt is a dated blog post (which proves the firm was publicly writing about a technology on that date) or a firm-action record (which proves the firm *did* something, not merely wrote about it). The two are kept distinct, not merged. | Year | The call (platform / technology) | What NewGenApps did (the receipt) | Source / artifact | How early (context at the time) | |------|----------------------------------|-----------------------------------|-------------------|---------------------------------| | 2007–08 | Cloud infrastructure (AWS) | Adopted AWS while it was in beta, before Elastic Block Store launched; ran the firm's own site on AWS by 2009 | Firm record | AWS EC2 entered public beta in August 2006 and exited beta in October 2008; EBS was new in 2008. Most enterprise IT was still on-premise. | | Oct 2008 | Mobile + cloud as a structural thesis | First blog post ("The journey has begun…", 5 Oct 2008) named iPhone, Android Marketplace, cloud and mobile together as a structural shift — the founding premise of the company | Dated blog post: `The-journey-has-begun` (2008-10-05) | The iPhone App Store had opened three months earlier (July 2008); the Android G1 had not yet shipped. "Cloud computing" was a nascent term. | | Oct 2008 | Android platform bet, pre-G1-launch | Described the firm as "one of the pioneers on the Android platform development" thirteen days before the first Android phone shipped | Dated blog post: `Android-Emulator` (2008-10-09) | Android SDK 1.0 had been released 23 Sept 2008; the HTC G1 launched 22 Oct 2008. The post predates the device. | | 2008 → Apr 2009 | iPhone over BlackBerry / Symbian | Bet on iPhone-OS and against the incumbents; built capability the moment Apple disclosed Objective-C, Xcode and Cocoa; stated iPhone-first, Android-second, BlackBerry/Symbian-as-legacy in writing | Dated blog posts: `iPhone-Survey` (2008-10-08), `Why-is-iPhone-such-a-great-development-platform` (2009-04-13); firm record for the framework-disclosure timing | BlackBerry and Symbian were the incumbents in 2009; the post predates Nokia's "burning platform" memo (Feb 2011) and BlackBerry's decline. | | 2009 | Augmented reality on mobile | Built and demonstrated an AR application on the G1 — the first Android phone — using the Android NDK | Firm record + recorded demonstration (`vimeo.com/618305787`, "AR Demo by NewGenApps Team in 2009 Using the G1… + the Android NDK") | The G1 launched Oct 2008; the Android NDK was first released June 2009. No consumer AR category existed; ARKit/ARCore were eight years away. | | Oct 2009 | Mobile + cloud as the next architecture | Publicly articulated mobile+cloud as the architecture of the next era | Dated blog post: `Eric-Schmidt-on-the-magical-potential-of-Mobile-Cloud` (2009-10-29) | "Mobile-first" was coined in 2009 but still fringe; "cloud-native" would not enter common use until 2013 and after. | | 2016 → 2017 | Big data, then machine learning | Sustained public coverage of big-data and machine-learning practice for business as those categories matured | Dated blog posts: `how-is-big-data-impacting-internet-marketing` (2016-02-12); `machine-learning-in-retail` (2017-02-27) | Early relative to mainstream *production* adoption of these stacks, not relative to the terms' invention — framed here as continuity, not first-mover. | | 2008–present | Continuous public record | A sustained blog archive, retrievable via the live site and archived originals, spanning each platform generation above | Firm record (public blog archive) | No further claim is needed — the archive is the fact. | A note on degrees of earliness, because it matters to the credibility of the dataset. The "before the category had a name" claim is strongest for the 2008–2009 mobile, cloud and Android nodes, where the dated archive is robust. The big-data (2016) and machine-learning (2017) nodes are later and blog-anchored: they are participation receipts that show the firm kept reading the next wave, not first-mover claims. Distinguishing the two — rather than flattening everything into "we were first" — is itself part of the discipline the record is meant to demonstrate. ## What a pattern of early calls reveals about judgment Repeated early adoption across unrelated technology generations is not coincidence. It reflects a consistent practice of reading primary signals — developer disclosures, infrastructure betas, capability demonstrations — rather than waiting for analyst consensus. The structural similarity across the calls is the point. In each case the signal came from the technology itself: Apple's framework disclosure, AWS beta access, a working AR demonstration on pre-production hardware. None came from a market-category report. Reading the durable signal under the loud one — that the iPhone's developer frameworks, not its hardware, were the real platform bet — is a specific, repeatable skill, and it is what the timestamps document. Applied to the present wave, that posture changes the question. The useful question is not "is AI mature enough?" It is "which specific capability is production-ready now, and what is the verification method?" That is the discipline behind an [evaluation harness](/ai-evaluation-harness/) and a [deploy-and-verify](/deploy-and-verify-ai/) approach: judging a system against production criteria — failure modes, data pipelines, monitoring, latency under real load — before it ships, not after. The constraint stands here too: an early call does not mean every bet pays off, and this record claims no such thing. What it demonstrates is a consistent orientation toward the pre-consensus signal, applied across more than fifteen years and multiple platform generations, with receipts that carry dates. ## Where the current AI wave stands — what the primary research shows The following is external context, not NewGenApps data. It connects the historical pattern to why the firm's judgment matters now. Each figure is attributed to its named primary, with its date, as published. The pattern across the timeline above is that each platform transition carried a gap between early experimentation and production-scale adoption. The 2025 AI evidence describes the same gap, and it is wide: - **MIT NANDA, *The GenAI Divide* (Aug 2025):** approximately 95% of GenAI pilots show no measurable P&L impact. On build-versus-buy, organisations reach production with a delivery partner about twice as often as they build internally alone — roughly 67% partner versus 33% internal. - **IDC, *Lenovo CIO Playbook 2025* (Feb 2025):** 88% of proof-of-concept deployments do not reach widescale production — only about 4 of every 33 POCs make it through. - **S&P Global, *Voice of the Enterprise* (Oct 2025):** 42% of organisations surveyed had abandoned most of their AI initiatives, up from 17% a year earlier. - **Informatica, *CDO Insights 2025* (Jan 2025, n=600):** data quality and access is named an obstacle by 43% of respondents — tied with technology infrastructure as the most-cited barrier, not the sole leading cause. Read together, these say something the timeline already implies: the variables separating pilots that stall from systems that reach production are evaluation discipline, data readiness and delivery experience — not the AI technology itself. That is [why AI pilots stall before production](/why-ai-pilots-fail/), and it is [the question of how you know an AI system works in production](/how-do-you-know-an-ai-system-works-in-production/) that most pilot programmes never structure themselves to answer. ## The same judgment, applied to today's AI The bridge from the historical record to the present is not "we are old." It is that production AI is a judgment problem before it is a model problem — and judgment under uncertainty is what the timestamped record demonstrates. Three places where reading the signal early separates a production system from a demo: **Reliability is not capability, and they are different numbers.** The production question for an [agentic AI](/agentic-ai/) system is not whether it can do the task but whether it does the task every time. "The agent succeeded" is a single-attempt claim; production runs on the probability it succeeds across every attempt, and the two diverge sharply, because end-to-end reliability is the product of per-step reliabilities, not the average — ten steps at 95% each is about 60% end-to-end. The τ-bench benchmark (Yao et al., Sierra, June 2024) introduced the multi-attempt measure to expose this gap. Spotting that "works once in a demo" predicts little about "works on the thousandth run" is the same early-signal skill as spotting that the iPhone's frameworks, not its hardware, were the real platform bet. **The product of multi-agent design is legible failure, and orchestration is the mechanism, not the headline.** Adding agents is not free. Anthropic's published engineering account of building a multi-agent research system (June 2025) is explicit that a multi-agent architecture consumes far more tokens than a single agent, is harder to evaluate and debug because behaviour is emergent, and is worth it only when the task's value justifies the cost. Knowing *when not* to reach for the more elaborate architecture is the same discipline as the firm's earlier calls: pick the bet that fits how the technology actually works. [Multi-agent orchestration](/multi-agent-orchestration-production/) is the means to see, attribute and contain a failure — a mechanism, not a benefit in itself. **The signal most teams miss is that retrieval fails silently.** Grounding a model in your data with [retrieval-augmented generation](/rag/) reduces ungrounded answers but does not eliminate them, and the failures are invisible to a demo. Even when the right document is in the context window, models use it better at the beginning or end than in the middle — the "lost in the middle" effect (Liu et al., TACL 2024) means "we retrieved the right chunk" is not "the model used it." And a grounded-looking answer can still be unfaithful to its source; the discipline that catches it is a faithfulness metric that decomposes the answer into claims and checks each against the retrieved evidence (Es et al., RAGAS, EACL 2024). A fluent, plausible, wrong answer that no error log catches is exactly the kind of weak signal the record shows the firm reads early. ## Frequently asked questions ### What is the NewGenApps timestamped-foresight record? The timestamped-foresight record is NewGenApps' first-party, dated archive of technology calls made before each platform generation reached consensus adoption — cloud infrastructure adopted in 2007–08 while AWS was in beta, before EBS launched; mobile-first development built when Apple disclosed its iPhone-OS frameworks; augmented reality demonstrated on the first Android handset in 2009; and big-data and machine-learning practice tracked in the firm's public blog as those categories matured. The record is drawn from retrievable, dated public artifacts: blog posts, a recorded demonstration, and documented firm decisions. NewGenApps (newgenapps.com) is operated by New Generation Applications Pvt. Ltd., incorporated in 2009; the public blog archive spans from October 2008 to the present. ### Why does a track record of early technology adoption matter for AI consulting? A verified history of reading platform signals early — anchored to dated, retrievable artifacts rather than retrospective claims — is evidence of the practice that separates firms that wait for category consensus from firms that operate at the signal layer. For [AI consulting](/ai-consulting/) specifically, the same practice determines whether a firm evaluates an AI system against production criteria from the start — failure modes, data pipelines, monitoring, reliability under real load — or treats a working demo as the deliverable. The external research suggests the former is what most AI pilot programmes are missing. ### How does NewGenApps approach production AI deployment differently from a typical pilot? The firm starts from the assumption that a pilot which cannot be verified against production criteria is not evidence of production readiness — it is evidence of a controlled environment. Every engagement is structured to answer one question before build, not after: does this system work under real operating conditions, with real data, at the reliability level the business actually needs, and can that be demonstrated with evidence that holds up to scrutiny? The firm runs production AI at the demanding end of reliability, enforcing a data-integrity contract (no synthetic data) and proof-based deployment — verified from source to live outcome, not "it merged." For detail on the method, see [how we work](/how-we-work/) and [AI rescue](/ai-rescue/). ## What the record says about the next call The pattern across more than fifteen years is consistent: the calls that held up were grounded in working demonstrations and primary technical signals, not category claims. The AR demonstration on the G1 in 2009 was not a trend report — it was a working system on pre-production hardware. The AWS adoption in 2007–08 was not a cloud-strategy document — it was an infrastructure decision made while the category was still unnamed. The firm applies the same posture to production AI. The question is not whether AI is significant in aggregate, but whether a specific system, with specific data, at a specific organisation, works — and can be demonstrated to work, with verification evidence that survives scrutiny. That is a harder question than "does the demo look good?" It is also the question that separates the systems that reach production from the 88% of proofs-of-concept that do not (IDC, *Lenovo CIO Playbook 2025*, Feb 2025). The same judgment that read those earlier waves early is what NewGenApps now applies to getting AI into production and proving it works. If that is the problem you are trying to solve, [how we work](/how-we-work/) and [AI consulting](/ai-consulting/) are the right starting points. NewGenApps. Stay a step ahead, always. ============================================================================== # AI Consulting Companies: The 2026 Landscape | NewGenApps URL: https://newgenapps.com/ai-consulting-companies/ ============================================================================== # AI Consulting Companies: The 2026 Landscape *Last updated: July 2026. A category map of the AI consulting market — the four kinds of firm, what each actually sells, and which category is structurally built to reach production rather than stop at a pilot.* ## What are the different types of AI consulting company? "AI consulting companies" is a single search term for four very different businesses. A strategy advisory sells a decision; a systems integrator sells scaled delivery; a senior-led boutique sells a working system built by the people who scoped it; a product-AI vendor sells a platform configured to your context. They price differently, staff differently, and — the point of this page — reach production at very different rates. **In short.** "AI consulting companies" are not one category but four — strategy advisories, systems integrators, senior-led boutiques, and product-AI vendors — that sell different things and reach production at different rates. Roughly 95% of enterprise GenAI pilots show no measurable P&L impact (MIT Project NANDA, 2025); which category you hire changes where you land against that number. The four categories, defined on their own terms: - **Strategy / management-consulting advisories.** McKinsey, BCG, Bain, and the advisory arms of the Big Four (Deloitte, PwC, EY, KPMG). They sell strategy, opportunity assessment, operating-model design, and organizational change. Their core deliverable is a decision and a roadmap; the build is frequently handed to a separate delivery organization. - **Systems integrators (SIs) and IT-services firms.** Accenture, Capgemini, IBM Consulting, Cognizant, TCS, Infosys, Wipro. They sell large-scale implementation and managed delivery through a leverage pyramid — a senior partner leads the relationship, a broad team executes under structured methodology. - **Senior-led AI boutiques.** Smaller specialist practices where the practitioners who assess the problem and design the architecture are the same people who build, deploy, and verify the system. Narrow scope, deep specialization, a short chain from decision to code. - **Product-AI shops and platform vendors.** Companies selling a productized AI platform or tool, plus the configuration services to fit it to you. Their deliverable is a product; the consulting is the implementation wrapper around it. The lines blur in practice — an SI has a strategy arm, a boutique will advise, a product vendor will consult — but the categories capture where each firm's economics and delivery model actually sit. For the underlying job all four claim to do, see [what an AI consulting firm actually does](/what-does-an-ai-consulting-firm-do/). ## Which type of AI consulting company actually reaches production? The category that reaches production is the one whose delivery model keeps senior judgment on the work from scope through verified production — and defines "done" as a measured outcome, not a launch. That is a structural property, not a brand. The failure record is now well documented, and it is a delivery record, not a model-quality one. MIT Project NANDA found that **roughly 95% of enterprise GenAI pilots show no measurable P&L impact**, after $30–40B of enterprise spend (MIT Project NANDA, *The GenAI Divide: State of AI in Business 2025*; Fortune coverage, 2025-08-18). S&P Global Market Intelligence found the share of companies abandoning most of their AI initiatives rose to **42%, up from 17% the prior year** (S&P Global Market Intelligence, *Voice of the Enterprise: AI & ML*, 2025). And the incumbents' own research agrees: McKinsey reported that only about **39% of organizations see any EBIT impact from generative AI**, with roughly one-third scaling it enterprise-wide (McKinsey, *The State of AI*, March 2025); BCG found **74% of companies had yet to show tangible value** from AI, and only 4% generating significant value across functions (BCG, *Where's the Value in AI?*, October 2024). None of those numbers is about weak models. They describe the gap between a convincing pilot and a system that runs on real data, under load, with a verified outcome — and each category sits differently across that gap: - **Strategy advisories** are optimized for the decision, not the deployment. Their production risk is that the engagement ends at the roadmap; the build lands with a different team, and the constraints discovered in code never reach the people who set the design. - **Systems integrators** genuinely reach production at scale, but through a pyramid that hands delivery from the seniors who sold the work to a more junior bench, and are usually contracted on launch milestones rather than a business outcome. That is precisely the "liveness mistaken for outcome" failure — a live, adopted system with no measured impact. See [liveness vs outcome](/production-ai-liveness-vs-outcome/). - **Senior-led boutiques** are built production-first — the same seniors carry the work through evaluation and verification — but do not scale to hundred-person, multi-region programs. - **Product-AI vendors** reach production fast for the slice their product already covers, and stall at the custom last mile: the integration, the data-integrity boundary, and the evaluation their platform does not do for your specific task. The mechanisms behind each stall are catalogued in [the production-AI failure taxonomy](/production-ai-failure-taxonomy/), and read as symptoms in [why AI pilots fail](/why-ai-pilots-fail/). ## The four categories of AI consulting company, compared The table maps structural differences, not a ranking. Dated public figures are cited where they exist; the "production-reach risk" column is the one that predicts whether you land in the 5% that shows impact. | Category | What they sell | Delivery model | Structurally strong at | Production-reach risk | Best-fit job | |---|---|---|---|---|---| | **Strategy advisory** | A decision, a roadmap, an operating model | Senior partners; analysis-led; build usually handed off | Board-level framing, portfolio prioritization, change management | Ends at the deck; ~39% of firms see any GenAI EBIT impact (McKinsey, March 2025) | You need a defensible enterprise AI strategy and executive alignment before building | | **Systems integrator / IT services** | Large-scale implementation and managed delivery | Leverage pyramid; senior sale, broad junior delivery; milestone-based | Multi-region rollout, legacy integration, sheer delivery capacity | Handoff from senior to junior; contracted on launch, not outcome; liveness ≠ outcome | You need a program staffed across geographies and integrated into heavy legacy estates | | **Senior-led boutique** | A working, verified system built by the people who scoped it | Small senior team, end-to-end; short scope-to-code chain | Reaching production on a defined system, with independent verification | Does not scale to hundred-person, multi-region programs | You need one real system in production, proven, on a short timeline | | **Product-AI / platform vendor** | A productized platform plus configuration | Product team plus implementation services | Fast time-to-value where the product already fits the use case | Stalls on the custom last mile — integration, data integrity, task-specific evaluation | Your problem matches a mature product and you accept its boundaries | The comparison of the two categories most often confused — the boutique and the Big Four — is treated in full in [boutique vs Big-4 AI consulting](/boutique-vs-big-4-ai-consulting/). This page is the wider map those two sit inside. ## What do "top AI consulting companies" lists actually measure? Most "top AI consulting companies" roundups rank on inputs a buyer cannot eat — headcount, revenue, brand recognition, breadth of service — not on the one output that matters, which is whether the firm's systems reach verified production. A list can be perfectly accurate about size and still tell you nothing about your odds against the 95%. A ranking worth trusting scores firms on the criteria that actually predict delivery. If you are reading or building a "top" or "best" list, weight it on these instead of logos: - **Production-proof.** A stated PoC-to-production conversion rate, as a number, and a live system the firm can describe. Data quality — the most-cited cause of stalled pilots at 43% of chief data officers (Informatica, *CDO Insights 2025*, January 2025, n=600) — is a delivery discipline, not a model feature, so ask what happens when a data source is stale. - **Senior delivery.** The people who scope the work are the people who build it. In a leverage pyramid, the seniors you meet in the pitch are not the team you get. - **Independent verification.** Someone other than the builder confirms the system runs on real data — an evaluation harness, a read-only check, an outcome instrumented rather than a health check pinged. - **Outcome, not launch.** "Done" defined as a measured business result, not a green status light or an adopted tool. Those criteria are the vetting checklist in full at [how to choose an AI partner](/how-to-choose-an-ai-partner/). The reason they beat a headcount ranking is that the failures are structural: Gartner projects that **over 40% of agentic AI projects will be canceled by the end of 2027**, citing cost, unclear business value, and inadequate risk controls (Gartner, 2025-06-25) — none of which a firm's size prevents. ## How do you match an AI consulting company to your job? Start from the job, not the category. The right firm is a function of what you actually need done, and the honest matches are clean: a strategy advisory when the deliverable is a decision and executive alignment; a systems integrator when the program is genuinely enterprise-wide and multi-region; a product vendor when your problem fits a mature product's boundaries; a senior-led boutique when you need one real system in production, verified, on a short timeline. Two cautions worth stating plainly. First, do not hire a category for a job it is not built for — a boutique cannot staff a hundred-person multi-continent rollout, and a strategy deck is not a running system. Second, the build-versus-buy question sits underneath the category choice: the most reliable order is often partner, then transfer, then in-house, so you cross the production gap on real software rather than from a standing start. That decision is treated in [build an in-house AI team or hire a partner](/build-vs-buy-ai-team/). Whichever category fits, the tests do not change: production-proof, senior delivery, independent verification. Apply them to any firm on your shortlist — including ours. For how those map to an engagement, see [AI consulting](/ai-consulting/). ## Frequently asked questions **What are the different types of AI consulting companies?** There are four practical categories. Strategy advisories (McKinsey, BCG, Bain, the Big Four advisory arms) sell decisions and roadmaps. Systems integrators (Accenture, Capgemini, IBM, TCS, Infosys) sell large-scale implementation through a leverage pyramid. Senior-led boutiques sell a working system built by the same seniors who scoped it. Product-AI vendors sell a platform plus configuration. They differ in what they sell, how they staff, and how reliably they reach production. **Which type of AI consulting company is most likely to reach production?** The category whose delivery model keeps senior judgment on the work from scope through verified production, and defines success as a measured outcome rather than a launch. Roughly 95% of enterprise GenAI pilots show no measurable P&L impact (MIT Project NANDA, 2025); senior-led delivery with independent verification is the structural answer to that gap, and it can exist inside a boutique or a well-run larger firm — but it is rarer in a milestone-contracted pyramid or a strategy engagement that ends at the deck. **What should a "top AI consulting companies" list actually measure?** Not headcount, revenue, or brand — those are inputs. A trustworthy ranking weights production-proof (a stated PoC-to-production conversion rate and a describable live system), senior delivery (the people who scope also build), independent verification (a check by someone other than the builder), and outcome over launch. Size does not prevent failure: Gartner projects over 40% of agentic AI projects will be canceled by the end of 2027 (Gartner, 2025-06-25). **Is a boutique AI consulting company better than a Big-4 firm?** Neither is categorically better; they fit different jobs. A large firm has genuine structural advantages in multi-region, cross-practice transformation programs. A senior-led boutique has the advantage when the deliverable is one system in production, verified, on a short timeline. The full two-way comparison is at [boutique vs Big-4 AI consulting](/boutique-vs-big-4-ai-consulting/). **How is an AI consulting company different from an AI product vendor?** A product vendor sells a pre-built platform configured to your context and reaches production fast where the product already fits. An AI consulting company diagnoses the problem first — including whether a product, a custom build, or no AI at all is the right answer — and takes accountability for a working system. The product route stalls at the custom last mile: integration, data integrity, and task-specific evaluation the platform does not do for you. --- If you need one AI system in production — built by the seniors who scoped it, with an independent check that it performs as designed — that is the delivery model NewGenApps runs. See [AI consulting](/ai-consulting/), [how to choose an AI partner](/how-to-choose-an-ai-partner/), or [book a 30-minute working session](/book/). *Sources: MIT Project NANDA, "The GenAI Divide: State of AI in Business 2025" (2025; Fortune, 2025-08-18); S&P Global Market Intelligence, "Voice of the Enterprise: AI & ML" (2025); McKinsey QuantumBlack, "The State of AI" (March 2025); BCG, "Where's the Value in AI?" (October 2024); Gartner press release (2025-06-25); Informatica, "CDO Insights 2025" (January 2025, n=600). Figures are attributed to their original sources and are not NewGenApps measurements.* ============================================================================== # AI Evaluation: Methods, Metrics, and Why Production Evals Differ from Benchmarks | NewGenApps URL: https://newgenapps.com/ai-evaluation/ ============================================================================== # AI Evaluation: Methods, Metrics, and Why Production Evals Differ from Benchmarks *Last updated: July 2026. A definitional reference on AI evaluation — what it is, the methods and metrics that matter, and why a system that passes a public benchmark can still fail in production.* ## What is AI evaluation? AI evaluation is the practice of measuring whether an AI system produces correct, reliable output against a defined standard — before release and after. A benchmark asks "how does this model rank." A production evaluation asks a narrower, harder question: "does this system do *our* task, on *our* data, well enough to ship, and has it regressed since last time?" The second question is the one that predicts production behavior. **In short.** Most teams evaluate the wrong thing. They read a public leaderboard, see a strong score, and treat that as evidence the system will work — but a benchmark measures a model in the abstract, while production measures a system doing a specific job on live data. The two can disagree sharply. Rigorous AI evaluation means building a held-out, task-representative test the model has never seen, scoring every release against it with graders you trust, and gating the release on the result. This page defines the discipline, its methods and metrics, and how to build an evaluation that actually predicts production. ## Why does a passing benchmark score not mean an AI system works? A benchmark score is only useful if it correlates with the behavior you care about, and contamination and saturation routinely break that correlation. The most direct public evidence is GSM1k: Scale AI built a fresh test in the exact style of the widely used GSM8K math benchmark, and some models dropped by up to roughly 13 points on the new set — with the size of the drop correlated to a measured memorization signal for the original GSM8K problems ("A Careful Examination of LLM Performance on Grade School Arithmetic," arXiv 2405.00332, 2024). Identical model weights, same style of problem, and the score falls double digits once the questions are ones the model could not have memorized. A leaderboard tells you how a model does on problems that may be in its training data; it does not tell you how your system does on your task, on your data, under your conditions. That gap is why a system can look ready on a benchmark and be nowhere near ready in production — and why the go/no-go decision has to rest on an evaluation you built, not a score you read. ## What are the types of AI evaluation? The evaluation you need depends on when it runs and what it compares against. Four distinctions matter, and most teams get the tooling before they get the discipline. - **Offline vs online.** Offline evaluation runs before release on a frozen case set — the release gate. Online evaluation (production outcome monitoring) watches the live system on real, unchosen tasks — the detection layer. Both are necessary; neither substitutes for the other. - **Benchmark vs task-representative.** A public benchmark measures general capability, useful for model selection. A task-representative eval, built from your own data and labels, measures whether the system does *your* job. Only the second gates a release. - **Automated graders vs human review.** Automated graders — exact-match, rubric, or model-as-judge — give speed and repeatability; human review gives ground truth on cases graders cannot yet score. Production evaluation uses both, and calibrates the grader against human labels before trusting it. - **Output-only vs trajectory-aware.** Checking only the final answer misses *how* the system got there. Two runs can reach the same correct answer — one by a valid path on fresh data, one by luck on stale data — and only trajectory-aware evaluation tells them apart. See [how to evaluate an AI agent](/how-to-evaluate-an-ai-agent/). The discipline lags the tooling, and the numbers show it. In LangChain's *State of Agent Engineering 2025* survey (1,340 respondents, fieldwork November 18 – December 2, 2025), roughly 89% of teams had implemented observability for their agents, but only 52.4% ran offline evaluations and 37.3% ran online ones — and quality was the single most-cited barrier to production at 32%. Teams instrument what happened; far fewer measure whether it was correct. ### Benchmark evaluation vs production evaluation | Dimension | Benchmark evaluation | Production evaluation | |---|---|---| | **Question it answers** | How does this model rank against others? | Does this system do our task well enough to ship, and has it regressed? | | **Data** | Public, shared, potentially in training data | Private, task-representative, held out, unseen by the model | | **What it scores** | General capability | The specific job on real inputs and edge cases | | **When it runs** | Model selection, one-off | Every release (offline gate) + continuously (online monitoring) | | **Failure it catches** | Weak base model | Contamination, regression, drift, brittle orchestration, silent wrong answers | | **What it predicts** | Leaderboard position | Production behavior | | **Risk if you rely on it alone** | High — the score may be memorized or saturated | Low — it is built to mirror production | The two are complementary, not interchangeable. Use benchmarks to pick a starting model; use a production evaluation to decide whether to ship. ## What should you measure in an AI evaluation? Measure task correctness, regression against a fixed baseline, and the failure modes that stay silent — not a single accuracy headline. The metrics that separate a production-ready system from a demo-ready one are the ones that surface quiet failure: - **Task accuracy on a held-out set** — scored against your labels, not a public answer key, and reported as "X on the held-out set versus Y last release" so every change is measured against a fixed bar. - **Regression rate** — how many previously-passing cases the new version breaks. A higher average score that quietly regresses ten known-good cases is often a worse release, not a better one. - **The confident-but-wrong rate** — how often the system produces a fluent, plausible answer that is factually or operationally incorrect. A system that fails loudly is fixable; one that is fluently, confidently wrong is the production risk that matters. - **Trajectory correctness** (for agents and pipelines) — the tools called, the order, and the freshness of the data used, not just the final output. - **Cost and latency under load** — a correct system that is too slow or too expensive to run is not production-ready either. For models used as the reasoning core — Claude among them — a strong held-out set is also where a careful model-as-judge harness earns its keep, because the judge itself has to be calibrated against human labels before its scores can gate anything. The instrument that ties these metrics together is covered in depth in [the AI evaluation harness](/ai-evaluation-harness/). ## How do you build a production AI evaluation? Building an evaluation that predicts production is a five-step discipline, not a one-off test. Each step exists to close a specific way benchmarks mislead. 1. **Freeze a task-representative dataset the model has not seen.** Assemble real inputs — including the malformed, adversarial, and edge cases production will throw — and lock it. This is the private holdout that a contaminated public benchmark cannot substitute for. 2. **Define graders and calibrate them against human labels.** Decide how each output is scored — exact-match, rubric, or model-as-judge — then check the automated grader against a human-labeled sample before you trust it to gate anything. 3. **Establish a baseline and score every release against it.** Record the current version's score so each change is measured against a fixed bar. Track regressions case-by-case, not just the average. 4. **Evaluate the whole flow, not single turns.** For multi-step systems, score the full trajectory and add termination, timeout, and fault-injection tests so orchestration faults surface before production, not after. 5. **Wire the result into the release gate, then monitor online.** Make the offline score a go/no-go condition on shipping, and pair it with production outcome monitoring so drift and silent failure are detected live. See [deploy and verify AI](/deploy-and-verify-ai/). Run in this order, the evaluation stops being a score you read and becomes a number you can defend. ## Why is evaluation the blocker between an AI pilot and production? Because the pilot was judged on a curated happy path and production is not curated — and the measurements that would have caught the gap were never built. The failure record is consistent across independent studies. MIT Project NANDA found roughly 95% of enterprise generative-AI pilots show no measurable P&L impact, after $30–40B of enterprise spend (MIT Project NANDA, *The GenAI Divide: State of AI in Business 2025*; Fortune coverage, 2025-08-18). S&P Global Market Intelligence found the share of companies abandoning most of their AI initiatives rose to 42%, up from 17% the prior year (S&P Global, *Voice of the Enterprise: AI & ML*, 2025). Gartner projects that over 40% of agentic AI projects will be canceled by the end of 2027, citing cost, unclear business value, and inadequate risk controls (Gartner, 2025-06-25). And the most-cited concrete obstacle is upstream of the model entirely: across 600 chief data officers, 43% named data quality, completeness, and readiness as the top barrier keeping generative-AI initiatives from the finish line (Informatica, *CDO Insights 2025*, January 2025, n=600). None of these is a model-quality problem. They are measurement-and-delivery problems: success defined as launch instead of outcome, benchmarks trusted instead of held-out tests, and no gate to catch the regression. Evaluation closes each gap. For the full map of how these failures manifest, see [the production-AI failure taxonomy](/production-ai-failure-taxonomy/); for the distinction underneath the 95%, see [production AI: liveness vs outcome](/production-ai-liveness-vs-outcome/) and [how do you know an AI system works in production](/how-do-you-know-an-ai-system-works-in-production/). ## Frequently asked questions **What is AI evaluation?** AI evaluation is measuring whether an AI system produces correct, reliable output against a defined standard, before release and after. In practice it means building a held-out, task-representative test the model has not seen, scoring every release against a fixed baseline with graders you trust, and gating the release on the result — rather than relying on a public benchmark or a human impression. **Why is a public benchmark not enough to decide whether to ship?** Because a benchmark can be contaminated or saturated. Scale AI's GSM1k showed some models dropping by up to roughly 13 points on a fresh test in the same style as a popular benchmark, with the drop correlated to a memorization signal (arXiv 2405.00332, 2024). A public score measures how a model does on problems that may be in its training data; a production evaluation measures how your system does on your task, on data it has never seen. **What is the difference between offline and online AI evaluation?** Offline evaluation runs before release on a frozen case set and acts as the go/no-go gate on shipping. Online evaluation runs on the live system against real, unchosen tasks and acts as the detection layer for drift and silent failure. Both are necessary; a green offline score does not substitute for live monitoring, and monitoring does not substitute for a release gate. **What metrics matter most in AI evaluation?** Task accuracy on a held-out set, the regression rate against a fixed baseline, and the confident-but-wrong rate — how often the system is fluently, plausibly incorrect. For agents, add trajectory correctness (the tools and data path, not just the answer). Cost and latency under load complete the picture, because a correct-but-unaffordable system is not production-ready. --- If you need an evaluation that actually predicts production — a held-out test built from your data, graders calibrated against human labels, and a release gate that turns "we think it works" into a number you can defend — that is the verification layer NewGenApps builds. See [the AI evaluation harness](/ai-evaluation-harness/), [AI consulting](/ai-consulting/), or [book a 30-minute working session](/book/). *Sources: Scale AI, "A Careful Examination of LLM Performance on Grade School Arithmetic," arXiv 2405.00332 (2024); LangChain, "State of Agent Engineering 2025" (n=1,340; fieldwork 2025-11-18 to 2025-12-02); MIT Project NANDA, "The GenAI Divide: State of AI in Business 2025" (2025; Fortune, 2025-08-18); S&P Global Market Intelligence, "Voice of the Enterprise: AI & ML" (2025); Gartner press release (2025-06-25); Informatica, "CDO Insights 2025" (January 2025, n=600). Figures are attributed to their original sources and are not NewGenApps measurements.* ============================================================================== # AI Production Readiness: The Pre-Launch Go/No-Go Checklist | NewGenApps URL: https://newgenapps.com/ai-production-readiness/ ============================================================================== # AI Production Readiness: The Pre-Launch Go/No-Go Checklist *Last updated: July 2026. The affirmative inverse of the [production-AI failure taxonomy](/production-ai-failure-taxonomy/) — the pass criteria you verify before you ship, stated as a numbered go/no-go gate with a green-flag/red-flag table.* ## What is an AI production readiness checklist? An AI production readiness checklist is the set of pass criteria you verify before shipping — the gate that turns "the demo worked" into a defensible go/no-go decision. Where a failure taxonomy names the ways systems break after launch, a readiness checklist names the checks that catch each break first. **In short.** Teams ship on the wrong signal — the demo landed, the service is up — and join the 88% of AI proofs-of-concept that never reach widescale deployment (IDC, *Lenovo CIO Playbook 2025*, February 2025). A readiness checklist replaces optimism with a gate: seven pass criteria, each the inverse of a known failure mode, verifiable before launch. Any "no" is a no-go. ## Why does "it works in the demo" not mean it's production-ready? Because the demo was a curated happy path and production is not curated — and the record shows the gap is wide and structural. MIT Project NANDA found that roughly 95% of enterprise generative-AI pilots show no measurable P&L impact, after $30–40B of enterprise spend (MIT Project NANDA, *The GenAI Divide: State of AI in Business 2025*; Fortune coverage, 2025-08-18). S&P Global Market Intelligence found the share of companies abandoning most of their AI initiatives rose to 42%, up from 17% the prior year (S&P Global, *Voice of the Enterprise: AI & ML*, 2025). Gartner projects that over 40% of agentic AI projects will be canceled by the end of 2027, citing cost, unclear business value, and inadequate risk controls (Gartner, 2025-06-25). None of those numbers is a model-quality problem. They are readiness problems: a system judged ready on liveness, usage, or a benchmark score instead of on the checks that actually predict production behavior. The discipline also lags the tooling that would enforce it — in LangChain's *State of Agent Engineering 2025* survey (n=1,340; fieldwork 2025-11-18 to 2025-12-02), roughly 89% of teams had implemented observability for their agents, but only 52.4% ran offline evaluations. Teams instrument what happened; far fewer gate on whether it was correct. The checklist below is that gate, and it is deliberately the mirror image of the [failure taxonomy](/production-ai-failure-taxonomy/): each mode there becomes a pass criterion here. ## What are the go/no-go criteria before shipping an AI system? Run these seven gates before launch. Each is phrased as a pass criterion with a concrete tell for failure, so the decision rests on evidence you can produce, not a feeling in the room. A single "no" is a no-go until it is closed. 1. **Outcome gate — is success a business KPI with a baseline?** GO if someone can state, before launch, the outcome metric the system exists to move and its current baseline, with a holdout or control to attribute the change. NO-GO if success is defined as launch, usage, adoption, or uptime. A live system that no one can tie to a pre-registered outcome is exactly the shape of the 95% — see [production AI: liveness vs outcome](/production-ai-liveness-vs-outcome/) and [how do you know an AI system works in production](/how-do-you-know-an-ai-system-works-in-production/). 2. **Evaluation gate — does a private holdout confirm the score?** GO if the go/no-go rests on a held-out, task-representative evaluation built from your own data that the model has never seen, scored against a fixed baseline. NO-GO if it rests on a public benchmark. Scale AI's GSM1k showed some models dropping by up to roughly 13 points on a fresh test in the style of a popular benchmark, with the drop correlated to a measured memorization signal (arXiv 2405.00332, 2024) — a public score can be memorized; your holdout cannot. See [AI evaluation](/ai-evaluation/) and [the AI evaluation harness](/ai-evaluation-harness/). 3. **Data-integrity gate — is there a contract and a measured baseline?** GO if a data contract governs the boundary (schema, freshness, null-rate) and you have computed a divergence measure — Population Stability Index or a Kolmogorov–Smirnov statistic — between training data and a recent production-representative holdout, with the divergence inside agreed bounds. NO-GO if the system was validated on a hand-cleaned demo extract with no contract. Data quality, completeness, and readiness is the single most-cited barrier keeping GenAI initiatives from the finish line — 43% of 600 chief data officers named it first (Informatica, *CDO Insights 2025*, January 2025, n=600). See [the AI data-integrity contract](/ai-data-integrity-contract/). 4. **Trajectory gate — is the whole flow evaluated, not just single turns?** GO (for agents and multi-step pipelines) if you evaluate the full trace and have explicit termination, timeout, and fault-injection tests on tool and agent calls. NO-GO if you only score the final answer. The MAST taxonomy of multi-agent failures found that system-design and coordination issues account for 44.2% of failures — including being unaware of termination conditions (12.4%) — not weak models (arXiv 2503.13657, 2025). See [how to evaluate an AI agent](/how-to-evaluate-an-ai-agent/) and [single vs multi-agent](/single-vs-multi-agent/). 5. **Oversight gate — is the human actually deciding?** GO if the reviewer override or rejection rate is above zero in staging — evidence the human in the loop exercises real control. NO-GO if the rate is at or near zero, which means the reviewer is confirming, not deciding, and the loop has already collapsed before launch. Sample-audit approvals and rotate reviewers to keep the signal honest. 6. **Containment gate — can you detect and revert a bad release?** GO if you can roll back within a defined window, a kill switch exists, and alerting fires on the outcome metric — not just on uptime. NO-GO if the only alarm is a liveness probe, because a wrong answer is shaped exactly like a right one and will not trip it. This is where the offline gate hands off to live monitoring; see [deploy and verify AI](/deploy-and-verify-ai/). 7. **Cost-and-latency gate — does it hold under production load?** GO if correctness, cost, and latency are measured together under realistic load and sit inside budget and SLA. NO-GO if the system is correct in isolation but untested at volume — a correct answer that is too slow or too expensive to serve is not production-ready either. ## Green flags vs red flags: is your AI ready for production? The fastest way to run the gate is to score what evidence you can actually produce for each criterion. A green flag is a specific, inspectable artifact; a red flag is an adjective standing in for one. | Readiness gate | Green flag — GO (ship) | Red flag — NO-GO (hold) | |---|---|---| | Outcome | A pre-registered business KPI with a baseline and a holdout or control | Success defined as launch, usage, adoption, or uptime | | Evaluation | A private, task-representative holdout the model has never seen, scored vs a baseline | The decision rests on a public benchmark or a demo impression | | Data integrity | A data contract plus a measured distribution baseline (PSI/KS) within bounds | A hand-cleaned demo extract; no answer for stale or malformed data | | Trajectory | Full-trace evaluation with termination, timeout, and fault-injection tests | Only the final answer is scored; single-turn testing only | | Oversight | Reviewer override rate above zero in staging | Override rate at or near zero — a rubber-stamp loop | | Containment | Rollback path, kill switch, and alerting on the outcome metric | The only alarm is a liveness or uptime probe | | Cost & latency | Correctness, cost, and latency measured together under production load | Correct in isolation, untested at volume | ## Which failure does each readiness gate prevent? Every gate exists because a specific, documented failure mode exists. Reading the checklist against the taxonomy makes the logic explicit — pass the gate on the left, and you have pre-empted the failure on the right before it can reach production. | Readiness gate (this page) | Failure mode it pre-empts ([taxonomy](/production-ai-failure-taxonomy/)) | |---|---| | Outcome | Liveness mistaken for outcome | | Evaluation | Evaluation gaps (contaminated or unrepresentative eval) | | Data integrity | Data-integrity drift | | Trajectory | Orchestration brittleness | | Oversight | Human-in-the-loop collapse | The remaining two gates — containment and cost-and-latency — are readiness checks the failure taxonomy treats as operational rather than definitional, but they are where an otherwise-ready system still stalls at go-live. For the fuller picture of what changes between the pilot and the production system, see [pilot vs production AI: what actually changes](/pilot-vs-production-ai-what-changes/) and [why AI pilots fail](/why-ai-pilots-fail/). ## How do you run the go/no-go review? Treat it as a single decision meeting with a written record, not a rolling opinion. Collect the evidence for each of the seven gates in advance — the holdout score against baseline, the divergence measurement, the trace-evaluation results, the staging override rate, the rollback drill, the load-test numbers — and put each gate to a binary GO or NO-GO on the artifact, not the assurance. Where the artifact does not exist, the gate is a no-go by default; "we're confident it's fine" is the answer the 95% gave. Record who signed off against what evidence, so the decision is defensible afterward and re-runnable on the next release. This is the same standard we hold our own work to: a separate, read-only check confirms the result on the real system before anything is called done, under a data-integrity contract that forbids synthetic or silently-stale inputs. The method is model-flexible — the reasoning core, whether a frontier model such as Claude or another, is replaceable; the gate is not. If a pilot has already stalled without passing these checks, the recovery path is the same list run in reverse as a diagnosis — see [AI Rescue](/ai-rescue/) or [how to choose an AI partner](/how-to-choose-an-ai-partner/). ## Frequently asked questions **Is my AI ready for production?** It is ready when it passes all seven gates on evidence, not assurance: a pre-registered outcome KPI with a baseline; a private held-out evaluation the model has never seen; a data-integrity contract with a measured distribution baseline; full-trace evaluation with termination and fault-injection tests; a staging reviewer override rate above zero; a rollback path with outcome-level alerting; and correctness, cost, and latency measured under load. A single missing artifact is a no-go until it is produced. **What is the difference between an AI production readiness checklist and a failure taxonomy?** A failure taxonomy is diagnostic and backward-looking — it names how systems break after launch. A readiness checklist is preventive and forward-looking — it names the pass criteria you verify before launch. They are inverses of the same map: each failure mode in [the production-AI failure taxonomy](/production-ai-failure-taxonomy/) corresponds to one go/no-go gate here, so passing the gate pre-empts the failure. **Why is uptime not enough to declare an AI system production-ready?** Because uptime confirms the process is running, not that the work is correct. In production AI a wrong answer is shaped exactly like a right one — it returns a clean value with no error, no stack trace, no non-200 — so a liveness probe will pass while the system delivers stale or incorrect output. Readiness requires alerting on the business outcome, not the ping. See [production AI: liveness vs outcome](/production-ai-liveness-vs-outcome/). **What is the single most common reason an AI system fails its readiness check?** The data. Data quality, completeness, and readiness was the most-cited barrier keeping generative-AI initiatives from the finish line, named first by 43% of 600 chief data officers (Informatica, *CDO Insights 2025*, January 2025). A system validated on a hand-cleaned demo extract, with no contract governing what the production feed is allowed to deliver, fails the data-integrity gate before any other. --- *If you need a go/no-go you can defend — the seven gates run on your system, each with the evidence behind it rather than an assurance in front of it — that is the verification layer NewGenApps builds. See [deploy and verify AI](/deploy-and-verify-ai/), [AI consulting](/ai-consulting/), or [book a 30-minute working session](/book/).* *Sources: IDC, "Lenovo CIO Playbook 2025: It's Time for AI-nomics" (report February 2025; CIO.com coverage 2025-03-25); MIT Project NANDA, "The GenAI Divide: State of AI in Business 2025" (2025; Fortune, 2025-08-18); S&P Global Market Intelligence, "Voice of the Enterprise: AI & ML" (2025); Gartner press release (2025-06-25); LangChain, "State of Agent Engineering 2025" (n=1,340; fieldwork 2025-11-18 to 2025-12-02); Scale AI, "A Careful Examination of LLM Performance on Grade School Arithmetic," arXiv 2405.00332 (2024); "Why Do Multi-Agent LLM Systems Fail?", arXiv 2503.13657 (2025); Informatica, "CDO Insights 2025" (January 2025, n=600). Figures are attributed to their original sources and are not NewGenApps measurements.* ============================================================================== # What Should an AI Proof of Concept Prove? | NewGenApps URL: https://newgenapps.com/ai-proof-of-concept/ ============================================================================== # AI Proof of Concept: What It Should Prove Before You Build *Last updated: September 2026. External figures are attributed and dated in the text. The sample-size table is illustrative arithmetic, not measured results.* **In short:** An AI proof of concept (POC) is a time-boxed build that answers one question on your own data: does this approach clear a quality, cost and latency bar that was agreed in writing before the work started? It does not prove the system is ready for production, and it should be allowed to end in "do not build." BCG's January 2025 AI Radar survey of 1,803 executives found 60% of companies failing to define and monitor any financial KPIs for AI value creation. In our reading, a POC without a written bar repeats that gap in miniature. Our POC in 3 Weeks, from $15,000, fixes the bar in Week 1 and ends in a build, refine or do-not-build verdict. ## What is an AI proof of concept? An AI proof of concept is a scoped, time-boxed build that tests whether one specific AI approach meets a pre-agreed bar on real data, at a cost and speed the business can accept. Its output is a decision, not a product. The four terms below are often used loosely; these are our working definitions. ## POC, prototype, pilot and production compared | | Question it answers | Data | Who uses it | Success bar | Typical output | |---|---|---|---|---|---| | Prototype | Can we show the idea? | Sample data | An internal audience | It looks convincing | A demo | | Proof of concept | On our real data, does it clear the bar we set, at an acceptable cost and latency? | A representative real-data extract | The evaluators | Thresholds agreed in writing before the build | An evaluation report and a build, refine or do-not-build verdict | | Pilot | Does it work for real users in a real workflow? | Live data, limited scope | A small group of real users | A business outcome for that group | Evidence for a wider rollout | | Production | Does it keep working, and can we prove it? | Live data at full scale | All intended users | Outcome monitoring with a named owner | A running, verified system | ## What should an AI proof of concept prove? It should prove four things, each against a threshold set in advance: quality on your real data, cost per task at production volume, latency at a stated percentile, and reliability on edge cases and on the "I don't know" path. Quality alone is not enough. The *Holistic Evaluation of Language Models* study (Liang et al., arXiv:2211.09110, 2022) measured seven metrics across sixteen core scenarios instead of accuracy alone, and reported that before it, models had on average been evaluated on just 17.9% of those scenarios. Two conditions come before the four measures. The task has to be technically feasible: Andrew Ng's *AI Transformation Playbook* (Landing AI, first released in 2018) advises having experienced AI engineers check a project's feasibility before kickoff. And the objective has to be clearly defined and measurable, creating business value that someone can point to. A POC built on a task nobody can score is a demo with a longer schedule. ## Why write the pass/fail bar down before the build starts? Because a bar set after the result is a rationalization. In its January 2025 AI Radar survey of 1,803 C-level executives across 19 markets and 12 industries, BCG found that 60% of companies were failing to define and monitor any financial KPIs for AI value creation (BCG, 2025-01-15). That survey is about AI programs, not proofs of concept. Our reading is that a POC which begins without a written bar is the same gap in miniature: nothing exists to grade the result against, so it is graded on impression. In NewGenApps' POC in 3 Weeks, the thresholds for quality, cost, latency and reliability are signed off at the end of Week 1, together with the frozen test set. They change afterward only through change control, with the effect on the timeline stated first. That closes the door on a POC being quietly graded as a win whatever the result. Gartner predicted in July 2024 that at least 30% of generative AI projects would be abandoned after proof of concept by the end of 2025, naming poor data quality, inadequate risk controls, escalating costs and unclear business value. That is a forecast, not a measured rate. Whatever the true rate turns out to be, the choice is when you find out: early, at a fixed scope, or late. ## How big does the test set need to be? Big enough that the bar can be decided by it. A pass rate measured on a small set comes with a wide range of plausible true values, and if your threshold sits inside that range, the set cannot settle the question. ## Sample size and the width of the answer Illustrative arithmetic, not measured results: the 95% Wilson interval for an observed 90% pass rate. | Test cases | Observed passes | 95% interval for the true pass rate | |---|---|---| | 30 | 27 | 74.4% to 96.5% | | 50 | 45 | 78.6% to 95.7% | | 100 | 90 | 82.6% to 94.5% | | 200 | 180 | 85.1% to 93.4% | | 500 | 450 | 87.1% to 92.3% | Read the second row this way: with 50 cases, a system that scored 90% could truly sit near 79% or near 96%, so a bar of "at least 90%" cannot be decided from that set. With 200 cases the lower end is about 85%, and with 500 about 87%. The Wilson interval is one of the two that Brown, Cai and DasGupta recommend for small samples (*Statistical Science*, 2001); the simple normal-approximation interval is erratic at small sizes such as the first row. Two things make real uncertainty wider than this table: cases that are not independent, such as many rows drawn from one document or one customer, and errors in the labels themselves. Treat the table as the narrowest plausible range. In practice, set the size of the test set in Week 1 from the tolerance the decision can bear, draw it from real data including the awkward cases, and freeze it. ## How do you scope an AI proof of concept? Seven steps, in order: 1. **Pick one task.** Narrow enough to evaluate unambiguously, such as classifying an inbound document or extracting fields from a contract. 2. **Name the decision the output informs.** Who uses it, in what workflow, and what changes as a result. This sets whether the bar is "good enough to act on directly" or "good enough to speed up a human who stays in the loop." 3. **Write the thresholds.** A target and a minimum for quality, a ceiling for cost per task and per month, a maximum p95 latency, and a maximum rate of confident-but-wrong answers. 4. **Split your real data.** Freeze a test set from it, awkward cases included and sized against step 3. The frozen set is never used to steer changes. Keep the rest as a development slice. 5. **Build the evaluation harness with the system, and iterate on the development slice.** Every meaningful change is scored on the slice, so progress is measured rather than asserted. The frozen test set is not touched. 6. **Score the frozen test set exactly once,** on a frozen build, with someone other than the builder running it. 7. **Decide by the rule you wrote in step 3.** Build, refine or do not build. ## What does a 3-week AI POC look like? The POC in 3 Weeks compresses define, build and verify into 21 days, on the client's own data. ## The three weeks at a glance | Week | Focus | Gate to advance | |---|---|---| | 1. Scope and design | Fix the use case and the input and output contract; sign off the thresholds; take delivery of the data extract; freeze the test set and set aside a development slice; design the approach | Signed-off metrics, a frozen test set, a development slice, and a reviewed design with its risks logged | | 2. Build with the harness | Build on real data; build the evaluation harness in parallel from day one; score every meaningful change on the development slice, never on the frozen set | A working system producing scored results on the development slice, trending against the thresholds | | 3. Evaluate and decide | Freeze the build; score the frozen test set exactly once, run independently; stress the edge cases and the "I don't know" path; write the report and the recommendation | An independently verified result on the running system, and a recommendation the budget owner can act on | You receive a working proof of concept on your real data, an evaluation report, the evaluation harness, the codebase and an architecture document, and an honest recommendation. When the verdict is build or refine, a costed production roadmap is included. The full terms are on the [engagements page](/engagements/#poc-in-3-weeks). ## What does a POC not prove? A pass means "worth building," not "ready to run." Outside a POC's scope: production hardening (availability, scaling, full security review); production integrations with live systems, write-back and access control at scale; data engineering at scale, because the POC uses a representative extract; more than one use case; interface work beyond what the evaluation needs; model fine-tuning unless the agreed metrics cannot be met without it; and change management. Those belong to a [production build](/engagements/#ai-production-build), and the questions to ask before shipping are in the [AI production readiness checklist](/ai-production-readiness/). A system can also pass every check and still deliver nothing, which is the point of [liveness versus outcome](/production-ai-liveness-vs-outcome/). ## When is "do not build" the right answer? When the evidence says so: the approach does not clear its minimum bar on real data; the cost or latency economics fail at production volume; or the reliability profile is unacceptable for the decision the output informs. In those cases a reasoned "do not build," ideally with a different approach worth testing, is a successful outcome of the POC. Finding that out in three weeks at a fixed scope is cheaper than finding it out six months into a production program. ## What does an AI POC cost? The POC in 3 Weeks starts from $15,000 and runs for 3 weeks (21 days). The fee is set against a scoped engagement after a free 30-minute call. What drives an AI engagement's price, and how to compare quotes, is in [how much AI consulting costs](/how-much-does-ai-consulting-cost/); the full ladder is on the [engagements page](/engagements/). ## Related reading and sources *To see what your own bar and test set would look like, [book a 30-minute working session](/book/). Related reading: the [evaluation harness](/ai-evaluation-harness/), [how you know an AI system works in production](/how-do-you-know-an-ai-system-works-in-production/), [deploy and verify AI](/deploy-and-verify-ai/), the [POC-to-production rate](/poc-to-production-rate/) to ask a vendor for, and [AI Rescue](/ai-rescue/) for a pilot that has already stalled.* *Sources: BCG, ["One Third of Companies Plan to Spend More than $25 Million On AI in 2025 Amid Widespread Optimism for Autonomous Agents"](https://www.bcg.com/press/15january2025-ai-optimism-autonomous-agents) (AI Radar 2025, n=1,803; press release, 2025-01-15); Landing AI, ["AI Transformation Playbook"](https://landing.ai/wp-content/uploads/2020/05/LandingAI_Transformation_Playbook_11-19.pdf) (Andrew Ng; first released December 2018); Gartner, ["Gartner Predicts 30% of Generative AI Projects Will Be Abandoned After Proof of Concept By End of 2025"](https://www.gartner.com/en/newsroom/press-releases/2024-07-29-gartner-predicts-30-percent-of-generative-ai-projects-will-be-abandoned-after-proof-of-concept-by-end-of-2025) (press release, July 2024); P. Liang et al., ["Holistic Evaluation of Language Models,"](https://arxiv.org/abs/2211.09110) arXiv:2211.09110 (2022-11-16); L. D. Brown, T. T. Cai and A. DasGupta, ["Interval Estimation for a Binomial Proportion,"](https://doi.org/10.1214/ss/1009213286) Statistical Science 16(2): 101-133 (May 2001), DOI 10.1214/ss/1009213286; NewGenApps, POC in 3 Weeks scope and the [engagements page](/engagements/#poc-in-3-weeks) (2026-09-29). Sample-size table computed by NewGenApps with the Wilson score formula, 2026-09-29.* ============================================================================== # How to Prioritize AI Use Cases Before You Fund a Build | NewGenApps URL: https://newgenapps.com/ai-use-case-prioritization/ ============================================================================== # How to Prioritize AI Use Cases Before You Fund a Build *Last updated: October 2026. External figures are attributed and dated in the text.* **In short:** To prioritize AI use cases, write each candidate as one specific use case, score it from 1 to 5 on value, effort and risk, and give its data a separate red, amber or green verdict. Fund first the candidate that ranks well on all four, and record why the others wait. Data gets its own verdict because readiness belongs to a use case, not to a company. Gartner predicts that through 2026 organizations will abandon 60% of AI projects unsupported by AI-ready data (a forecast, 2025-02-26). NewGenApps offers the ranking as AI in a Day, from $2,500 in one working day, and the ranking plus a data verdict as AI Compass, from $5,000, typically 2 to 6 weeks. ## What does it mean to prioritize AI use cases? Prioritizing AI use cases means ranking candidate uses of AI against each other, on the same scales, so budget goes to the one most likely to pay back and prove out, and the rest are deferred or declined on the record. The output is a ranked shortlist, one recommended first move and a "not yet / not this" list. It comes before a [proof of concept](/ai-proof-of-concept/) (POC): prioritization chooses the candidate, and the POC tests it. A use case is a task, not a technology. "AI for customer service" is a category; "draft first replies to routine support emails from the knowledge base, with an agent approving every send" is a use case. Rank only the second kind. For why a strategy should end in a working build, see [AI strategy](/ai-strategy/). ## Why does data readiness belong in the ranking? Because an idea can be sound and unbuildable at the same time. In a Q&A published on 2025-02-26, Gartner predicted that "through 2026, organizations will abandon 60% of AI projects unsupported by AI-ready data." That is a forecast, not a measured rate. The same page says 63% of organizations either do not have, or are unsure whether they have, the right data management practices for AI. Gartner's page description ties that figure to a third-quarter 2024 survey of 248 data management leaders; the body text names a July 2024 survey of 1,203 and gives no sample for the figure. We cite the 63% without a sample size and read it as directional. A survey conducted by Harvard Business Review Analytic Services with Cloudera, a data-platform vendor, points the same way. Of more than 230 HBR audience members involved in their organizations' AI data decisions, surveyed in October 2025, only 7% said their data was completely ready for AI adoption (press release, 2026-03-05). The sample is small and drawn from one publication's audience: a signal, not a rate. Gartner's definition matters most for ranking. AI-ready data must be representative of the use case, including its patterns, errors and outliers, and the first of Gartner's five steps is to align data to AI use cases. Readiness belongs to a use case on a system: the same customer export can be ready for sorting inbound email and unready for forecasting churn. So readiness gets its own verdict beside the scores. What a live system must then enforce about its data is covered by the [data-integrity contract](/ai-data-integrity-contract/). ## How do you score a use case on value, effort and risk? Score each candidate from 1 to 5 on three scales and write one sentence of reasoning beside every score. For value, 5 is best. For effort and risk, 5 is worst. Value asks how large and certain the benefit is against a named baseline; effort, what it takes to reach production; risk, how likely a wrong or missing output is and what one costs. NIST's AI Risk Management Framework defines risk as the combination of an event's probability and the magnitude of its consequences (NIST AI 100-1, January 2023), and says the acceptable level is "highly contextual and application and use-case specific." Tolerance is yours to set, within any rules your sector already imposes. ## Scoring anchors for value, effort and risk Our working anchors, so two reviewers land in the same place: | Scale | Question it answers | 1 | 3 | 5 | |---|---|---|---|---| | Value (5 = best) | Size and certainty of the benefit | Marginal or speculative | Meaningful, plausible case | Large, tied to a named line item with a defensible baseline | | Effort (5 = heaviest) | Work to reach production | Weeks, self-contained | About a quarter, manageable dependencies | Many months, heavy dependencies, new data or integrations | | Risk (5 = highest) | Likelihood and cost of a wrong output | Low and reversible; a person reviews each output | Moderate; manageable with controls | High: customer-facing, regulated or hard to reverse | The scores are ordinal judgments, not measurements: a 4 is not twice a 2. If you combine them into one number, show the three scores beside it and treat the total as a rough guide. The decision is argued over the components. Score each candidate separately before reconciling, because the person who proposed an idea should not be its only scorer. ## How do you check whether the data is ready? Ask four plain questions about each system the use case depends on, answer them with evidence, and give the system a red, amber or green verdict. - **Freshness.** How old is the data when the use case uses it, how old can it be before the output is wrong, and would anyone notice if the feed stopped? - **Ownership.** Who owns it, who can approve its use, and who fixes it when it is wrong? - **Access.** Can the team reach it, how long does approval take, and do privacy, residency or contract terms limit its use? - **Quality.** Is it complete, consistent and representative of the cases the use case will meet, including the awkward ones, and labeled where the use case needs labels? What "the data" means shifts with the use case. For drafting or search over documents, it is the documents: who maintains them, whether they contradict each other, who may see each one. For prediction, it is labeled history: enough examples of each outcome, and labels you trust. Evidence means a sample pulled, a person named, a timestamp seen. ## Red, amber and green verdicts | Verdict | Meaning | Effect on the ranking | |---|---|---| | Green | Reachable, owned, fresh enough and good enough to start on now | Eligible to be the first build | | Amber | Usable with named fixes, each with an owner and a rough effort | Eligible once the fixes are scheduled; their effort counts in the effort score | | Red | A blocker: no access, no owner, or not usable for this task | Moves down the order; the fix becomes its own work, or the candidate waits | ## How do you turn the scores into a ranking? Run the same seven steps for every candidate and keep the working visible. 1. **Write each candidate as a use case.** One sentence naming the task, who does it today, the decision it informs and the system it touches. 2. **Screen out the wrong tool.** Set aside anything a written rule, a report or a process change would handle better. 3. **Name the value.** The line item it should move, today's baseline and the person who owns that number. A benefit with no baseline cannot score 5. 4. **Score value, effort and risk,** each 1 to 5 with a sentence of reasoning, separately by at least two people, then reconcile. 5. **Ask the four data questions** for each system and record a red, amber or green verdict with its evidence. 6. **Rank.** High value with low effort, low risk and green or amber data goes first. High value with high effort is a staged bet, usually proven by a POC first. Low value with high effort is declined, in writing. 7. **Name the first move and write the defer list.** Say what the first build must prove, and give each deferred candidate a reason and a condition for revisiting it. ## When is AI the wrong tool for a candidate? When a simpler tool does the job, or when success cannot be shown. Five signs: - **A rule can be written down.** If an expert can state the logic and the inputs are structured, ordinary software is usually cheaper to build, test and audit than a model. - **Nobody can score the output.** With no agreed way to judge an output, whether against known answers or a reviewer's written rubric, you cannot show the system works; see the [evaluation harness](/ai-evaluation-harness/). - **Volume is too low.** A task done a few times a month is unlikely to repay the build and run cost. - **A wrong answer is costly and unreviewed.** Either design the review in, or do not automate the step. - **The data is out of reach.** That is a red verdict, not a modeling problem, and a better model will not fix it. Each is a result, not a failure: a shortlist can include candidates that need no AI. ## What do AI in a Day and AI Compass deliver? Both rank candidates with a senior practitioner, and neither builds anything. AI in a Day scores value, effort and risk from what the people in the room know. It does not connect to your systems, so it gives no data verdict. AI in a Day delivers the ranked shortlist, a one-page "where to start" and a "not yet / not this" list in one working day. AI Compass goes deeper for leaders who need a costed roadmap and a data-readiness verdict before funding a build: initiatives are scored on value, feasibility, time to value and risk with their ROI assumptions exposed, and each system is rated red, amber or green. ## AI in a Day and AI Compass compared | | [AI in a Day](/engagements/#ai-in-a-day) | [AI Compass](/engagements/#ai-compass) | |---|---|---| | Good fit when | Several ideas compete, an earlier effort stalled, or a board wants a plan | You need a costed, ranked roadmap and a data verdict before funding a build | | You get | Ranked shortlist, one-page "where to start," "not yet / not this" list | Prioritized, costed roadmap; red, amber or green data-readiness scorecard; recommended first POC; executive readout | | Duration | One working day | Typically 2 to 6 weeks | | Starting price | from $2,500 | from $5,000, credited toward a subsequent build | ## Which should you start with? - **Several ideas and no shared way to choose:** the free 30-minute call, or [AI in a Day](/engagements/#ai-in-a-day). - **A short list already, and a funding decision to make:** [AI Compass](/engagements/#ai-compass), whose fee is credited toward a subsequent build. - **One well-defined use case, with requirements and data access agreed:** go straight to a [POC in 3 Weeks](/engagements/#poc-in-3-weeks); how to read one is in [what a proof of concept should prove](/ai-proof-of-concept/). - **A pilot that stalled before production:** [AI Rescue](/ai-rescue/). The full decision guide is in [which AI engagement you need](/which-ai-engagement-do-you-need/), and the price ladder is in [how much AI consulting costs](/how-much-does-ai-consulting-cost/). ## What happens after the ranking? The first move is funded as the smallest valuable thing the evidence supports, usually a proof of concept with its pass/fail bar written first, then a [production build](/what-does-an-ai-production-build-include/). Decide at the same time how payback will be measured against a baseline taken before launch; [measuring AI ROI in production](/measure-ai-roi/) covers the method. Deferred candidates return when their condition is met: the data fixed, the review designed in, the volume proven. Re-score when the data or the business changes. ## Related reading and sources *To rank your own candidates with a senior practitioner, [book a 30-minute working session](/book/) or see the [engagements](/engagements/). Related reading: [why AI pilots fail](/why-ai-pilots-fail/), the [AI production readiness checklist](/ai-production-readiness/) and [AI consulting](/ai-consulting/).* *Sources: Gartner, ["Lack of AI-Ready Data Puts AI Projects at Risk"](https://www.gartner.com/en/newsroom/press-releases/2025-02-26-lack-of-ai-ready-data-puts-ai-projects-at-risk) (Q&A with Roxane Edjlali, 2025-02-26); Cloudera and Harvard Business Review Analytic Services, ["Only 7% of Enterprises Say Their Data Is Completely Ready for AI, According to New Report from Cloudera and Harvard Business Review Analytic Services"](https://www.cloudera.com/about/news-and-blogs/press-releases/2026-03-05-only-7-percent-of-enterprises-say-their-data-is-completely-ready-for-ai-according-to-new-report-from-cloudera-and-harvard-business-review-analytic-services-reveals.html) (press release, 2026-03-05; report *Taming the Complexity of AI Data Readiness*); NIST, ["Artificial Intelligence Risk Management Framework (AI RMF 1.0)"](https://doi.org/10.6028/NIST.AI.100-1), NIST AI 100-1 (January 2023); NewGenApps, [engagements page](/engagements/) (published 2026-09-29).* ============================================================================== # Embedded AI Team: Retainer, Project or Hire? | NewGenApps URL: https://newgenapps.com/embedded-ai-team/ ============================================================================== # Embedded AI Team, Project or Hire: What an AI Retainer Includes *Last updated: October 2026. External figures are attributed and dated in the text.* **In short:** An embedded AI team, or AI team as a service, is a small senior group that works inside your tools and meetings on a monthly fee and carries AI work from strategy to production and operation. "Fractional AI team" is used both for this and for part-time senior advice, so ask which you are buying. A fixed project buys one deliverable, a hire buys one person, and a retainer fits continuous work whose priorities will move. NewGenApps' Your AI Team starts from $10,000 per month for the smallest pod and runs month to month after an initial term of typically three months. Someone other than the builder verifies work before it is promoted; the exit is included in the fee. ## What is an embedded AI team? An embedded AI team is a small group of senior AI practitioners that works inside your organization's tools, stand-ups and working hours, on a recurring monthly fee, and carries AI work from strategy through build, deployment and operation. The market also says "AI team as a service" and "fractional AI team" for this idea, and not consistently: "fractional" is often used for a part-time senior adviser, which is a different product. In this article an embedded team does the hands-on work and is accountable for what runs in production. It also differs from staff augmentation, where you direct individual contractors to fill seats. In a retainer you decide what is built next, the team decides how, and someone other than the builder shows that it works. ## Retainer, project and hire compared This is our working comparison, a judgment rather than a measurement. We looked for an independent study comparing outcomes of retainers, fixed projects and in-house hires and did not find one; the comparisons we found were published by firms that sell one of the three. The nearest evidence compares building with partnering in general: see [build vs buy for an AI team](/build-vs-buy-ai-team/). | | Fixed project | Embedded retainer | In-house hire | |---|---|---|---| | What you buy | A written deliverable at a set scope and price | A standing team's capacity on a monthly fee | One person's time and judgment, on an open-ended basis | | Who sets priorities | Fixed at signing; changes go through change control | You, through a backlog re-planned every sprint | You | | Who carries the risk | The vendor carries overrun for the written scope | Shared: you pay for capacity, the vendor answers for how it is used | You: recruiting, ramp-up and key-person risk | | Best for | A decision or build whose scope is knowable up front | Continuous AI work whose priorities will move | AI that is core to the product, with steady work | | Watch for | Improvement after handover needs a new scope | A vague backlog, or no named owner on your side | Time to fill senior roles; knowledge leaves with the person | | How it ends | Delivery and handover | Notice, then a structured hand-over | Notice under the employment contract | ## What does an AI retainer include? Your AI Team, the NewGenApps retainer, includes the following (headline terms published on the [engagements page](/engagements/#your-ai-team) on 2026-09-29; detail from the engagement charter): - A small senior pod working in your tools, stand-ups and overlapping hours, led by an Engagement Lead, with an architect, builders and verification sized to the tier. - A rolling backlog, re-planned every sprint. - A monthly strategic review that reads the roadmap alongside outcome metrics. - Independent verification: work is checked by someone other than the person who built it before it is promoted. - Responsiveness tiers for incidents, set by severity and applied within the pod's working hours. - Continuous knowledge transfer and a structured, no-lock-in exit at no extra charge. Not included, unless added through change control: general application development unrelated to AI; round-the-clock on-call beyond those tiers; standalone data-platform builds or large migrations; regulatory certification or legal sign-off; staff backfill. Third-party model, license and cloud costs are billed at cost or paid by you directly. Terms are confirmed in the engagement order. ## How do the pod tiers differ? Pods come in indicative tiers, sized to how much AI work is running at once. Every tier has an Engagement Lead accountable for outcomes and works to the same verification discipline; the tiers differ in team size and in how much of the lead's, architect's and verifier's time the pod carries. The tier is confirmed at scoping and can be adjusted at the monthly review. ## Pod tiers at a glance Indicative shapes from the engagement charter, not a quote. | Pod shape | Indicative composition | Best for | |---|---|---| | Focused pod | Part-time Engagement Lead and senior builder capacity, with architect and verification on call | A capability in steady production: maintenance plus measured improvement | | Balanced pod | Part-time Lead, architect, builders and verification | Live capabilities being built and operated at the same time | | Multi-stream pod | Dedicated Lead, architect, several builders and dedicated verification | A portfolio of AI capabilities across teams, with parallel build and operate streams | ## Who decides what, and how is work proven before it ships? You decide what is built next, the Engagement Lead decides how, and a verifier who did not build a change decides whether it is done. The routine is a daily stand-up in your tools, sprint planning, a sprint demo of working software against the live system, a retrospective, and a monthly strategic review. "Independent" has one specific meaning here: the person who verifies a change is not the person who built it, and judges it against acceptance criteria agreed in advance. It does not mean an outside audit. The verifier sits in the same pod and is paid from the same monthly fee. Formal independent verification and validation of software is a stricter standard. NASA's program, citing IEEE, describes independence in three parts: technical, managerial and financial (NASA, IV&V Overview, page updated 2024-07-23). If a regulator or your board needs that standard, commission a separate reviewer alongside the retainer. The monthly review is where the Engagement Lead and your senior stakeholders step back from the sprint and read outcome metrics next to the roadmap: capabilities running in production, the business metric each exists to move against its baseline, and how much of what runs your own team can operate. Baselines are agreed early in the engagement. For how deployments are verified, see [deploy and verify AI](/deploy-and-verify-ai/); for why outcomes rather than uptime are monitored, see [liveness versus outcome](/production-ai-liveness-vs-outcome/). ## Decision rights at a glance | Decision | Held by | |---|---| | What to build next | Your product owner, advised by the Engagement Lead | | How it is built: architecture, tooling, model choice | The Engagement Lead, consulting your architects | | Whether a change is done and ready to promote | The verifier, against criteria agreed in advance, never the builder | | Whether to promote from pilot to limited to full rollout | Your product owner, on the verifier's evidence | | A change of scope or price | Your sponsor and the Engagement Lead, through change control | ## Can you leave, and what do you keep? Yes, on notice. There is an initial term of typically three months, after which Your AI Team runs month to month, and a structured exit is included in the fee. The work lives in your repositories, documentation and chat from the first day; there is no NewGenApps-only environment holding it and no tooling that only we can operate. Knowledge transfer is continuous, not an exit event. On notice, a structured hand-over covers a final knowledge-transfer pass, runbook walkthroughs, access transition and a documented sign-off that your team can operate the system. It is not charged as an extra. The test applies to any vendor: at any point, could your team operate what is in production with the documentation and access it already has? Ask who owns the repositories, what only the vendor can operate, and what leaving costs. ## When is a fixed project the better choice? When the outcome can be written down: one use case, a clear bar, a known end date. A fixed-scope project puts overrun risk on the vendor for the scope written down, and it ends with a handover. A retainer is the wrong way to buy a single deliverable, because you pay for standing capacity you may not use. A project is often the right first step. A [proof of concept](/ai-proof-of-concept/) shows whether an unvalidated use case is worth building on your data, an [AI production build](/what-does-an-ai-production-build-include/) takes a validated one into production, and [AI Rescue](/ai-rescue/) triages a stalled pilot with a real off-ramp after week one. A retainer assumes a continuing queue of AI work worth running; in our view that queue is usually visible only once a first system is live. ## When is a hire the better choice? When AI is core to what you sell, the workload is steady, and you can hire and keep senior people. A permanent hire tends to hold your context and shape your culture in a way a vendor pod cannot. The economics, and the evidence on building versus partnering, are on [build vs buy for an AI team](/build-vs-buy-ai-team/); this page does not repeat them. Hiring is also constrained by the labor market. The World Economic Forum's Future of Jobs Report 2025 (January 2025) surveyed over 1,000 employers representing more than 14 million workers across 55 economies. In it, AI and big data top the list of fastest-growing skills for 2025-2030, and 63% of employers named skill gaps in the labor market as a major barrier to business transformation. In the same survey chart, 27% listed inability to attract talent to their own firm. These are employer expectations across all skills, not measurements of AI hiring time. Read them as context, not as a reason to avoid hiring. The routes are not exclusive: a pod's continuous knowledge transfer lets a later in-house team take over what is running, the partner, transfer, then build sequence described on the build-vs-buy page. ## How do you decide between a retainer, a project and a hire? Ask five questions, in this order. 1. **Can the outcome be written as one deliverable with an end date?** If yes, buy a fixed project. 2. **Is there a continuing queue of AI work whose priorities will change as you learn?** If yes, a retainer fits. If not, you would pay for idle capacity. 3. **Is AI core to what you sell, with steady work and a realistic path to hiring and keeping senior people?** If yes, plan to hire, perhaps after a project or a retainer has taken the first system into production. 4. **Can you name a product owner and a sponsor with time for sprint decisions, and give the team access to data and systems?** If not, a retainer will wait on you whatever the contract says. 5. **Can you leave?** Ask for the exit terms in writing: who owns the repositories, what only the vendor can operate, and what leaving costs. None of the three fits if the best candidate does not need AI. A rule, a report or a process change sometimes does the job, and a team that only ever answers "build it" is not advising you. If you are unsure where you stand, start with [which AI engagement you need](/which-ai-engagement-do-you-need/) or [how to choose an AI partner](/how-to-choose-an-ai-partner/). ## What does an AI retainer cost? Your AI Team starts from $10,000 per month for the smallest pod. Larger pods cost more, and the tier is confirmed at scoping. The price, in US dollars, was published on the [engagements page](/engagements/#your-ai-team) on 2026-09-29. The fee is, indicatively, a fixed monthly amount per pod tier, covering the pod's capacity and the governance described above, with no per-ticket or hourly billing. Model usage, licenses and cloud infrastructure are separate: billed at cost or paid by you directly. To compare quotes, see [how much AI consulting costs](/how-much-does-ai-consulting-cost/). ## Related reading and sources *To see what a pod would look like for your workload, read the [Your AI Team terms](/engagements/#your-ai-team) and [book a free 30-minute working session](/book/). Related reading: [what an AI proof of concept should prove](/ai-proof-of-concept/), [what an AI production build includes](/what-does-an-ai-production-build-include/), [which AI engagement you need](/which-ai-engagement-do-you-need/), [how we work](/how-we-work/), [the data-integrity contract](/ai-data-integrity-contract/) and [AI consulting](/ai-consulting/).* *Sources: World Economic Forum, [The Future of Jobs Report 2025](https://www3.weforum.org/docs/WEF_Future_of_Jobs_Report_2025.pdf) (January 2025; "Key findings" and section 4.1, "Barriers to transformation"); NASA, [IV&V Overview](https://www.nasa.gov/ivv-overview) (page last updated 2024-07-23); NewGenApps, Your AI Team terms on the [engagements page](/engagements/#your-ai-team) (published 2026-09-29) and the client-facing Your AI Team engagement charter.* ============================================================================== # Frequently Asked Questions — NewGenApps URL: https://newgenapps.com/faq/ ============================================================================== # Frequently Asked Questions Straight answers about who we are, how we work, and how to start. Each answer below is written to stand on its own. The official NewGenApps website is **newgenapps.com**. ## What is NewGenApps? NewGenApps (New Generation Applications Pvt Ltd) is an AI delivery partner, incorporated in 2009, that orchestrates AI into a business — strategy, build, and the unglamorous production engineering — and takes enterprises from AI strategy to production systems. We are AI-led and model-flexible: our engineering depth is agentic delivery — we build with Claude Code and a library of custom skills, the agent-driven workflow that lets a small senior team ship and verify production AI fast — while staying fluent across the stack. We do the building, not just the slide deck — moving teams from idea to a working system in weeks, not quarters. ## What makes NewGenApps different? The scarce skill is no longer building a model, an agent, or a chatbot — those are individual instruments. The harder, more valuable work is orchestrating AI *into* how a business actually runs: the right models and agents, the data, the integration, the evaluation, and the operational discipline, sequenced against real processes and systems. That is the conductor's job, and more than fifteen years delivering across enterprise, mid-market, SME and startup is what it takes to do it well. We go deep where depth matters — our engineering depth is agentic delivery with Claude Code and custom skills — but we are not bound to one vendor or one trick. ## Who founded NewGenApps and where is it located? NewGenApps was founded by Anurag Rastogi, who remains its CEO, and was incorporated in 2009. The company is headquartered in Lucknow, India and New York, USA, and serves clients across the US, Europe, and India. ## How long has NewGenApps worked on AI and emerging technology? NewGenApps wrote about machine learning and AI as early as 2016 and built an augmented-reality demo on the first Android phone in 2009 — part of a documented track record, dated from October 2008, of calling technology megatrends early, including mobile, cloud, big data, and IoT. We ran our own website on AWS in 2009, having adopted it in beta in 2007–08 before EBS had launched. That publicly dated record, running from October 2008, is original primary-source proof, not a 2024-vintage rebrand. ## What services does NewGenApps offer? NewGenApps offers a small set of clear entry points: a 30-minute AI working session (no deck, no pitch), "AI in a Day," an AI Compass opportunity audit, a scoped proof of concept, an AI production build, and an embedded AI team retainer. You can start small with a working session and scale up to a full embedded pod as the work proves out. ## What are your engagement and pricing models? We work in fixed-scope, fixed-outcome engagements (for example a POC in 3 weeks or a scoped production build) and in a monthly embedded-AI-team retainer for ongoing delivery. Pricing depends on scope and is set per engagement after a working session, so you know the deliverable and the price before any commitment — book a 30-minute session at /contact/ for a specific quote. Starting prices are on the [engagements and pricing](/engagements/) page, and how to read any quote is in [how much AI consulting costs](/how-much-does-ai-consulting-cost/). ## How fast can NewGenApps ship an AI proof of concept? NewGenApps builds scoped AI POCs in weeks, not quarters — our flagship engagement takes a defined use case from strategy to a clickable, demonstrable proof of concept in about three weeks. Tighter scope ships faster; an "AI in a Day" sprint produces a working artifact in a single day. ## What is the typical timeline from first call to production? Most clients start with a 30-minute working session, move to a scoped POC in roughly three weeks, and then advance proven POCs to a production build over the following weeks. The exact path depends on the use case, but the operating principle is the same throughout: ship something real early, then harden it — capability earns its way to full rollout through measurable gates, not a calendar. ## How does NewGenApps handle data security and confidentiality? We work under NDA, scope data access to only what an engagement requires, and build on enterprise-grade AI platforms — including Claude Code and AWS (Amazon Bedrock, SageMaker), where API data is not used to train models by default. Security and confidentiality terms are agreed in writing before any data is shared — raise your specific requirements in the first working session. ## Is NewGenApps an official Anthropic partner? Our engineering depth is agentic delivery — we build with Claude Code and a library of custom skills, the agent-driven workflow that lets a small senior team ship and verify production AI fast. Beyond that, we stay model-flexible by design: AWS, NVIDIA-accelerated compute, and Anthropic's models are among the tools we currently reach for on a given engagement, chosen for fit to the problem rather than a fixed stack, so clients are never locked into one vendor. We are actively building toward formal Anthropic partner credentials. To be precise and honest: we do not currently claim official Anthropic partner status, and we will say so plainly until that designation is formally in place. ## Is newgenapps.io the same company as NewGenApps? No. The official NewGenApps website is **newgenapps.com**. The domain newgenapps.io is not affiliated with, endorsed by, or operated by NewGenApps (New Generation Applications Pvt Ltd). If you are looking for us, you are in the right place at newgenapps.com. ## Is NewGenApps the same company as Newgen Software? No. NewGenApps is New Generation Applications Pvt Ltd (**newgenapps.com**), an independent senior AI delivery firm headquartered in Lucknow, India and New York, USA. Newgen Software Technologies Ltd (**newgensoft.com**) is a separate, unaffiliated company — a large NSE/BSE-listed enterprise software vendor based in New Delhi that builds ECM, BPM, and CCM low-code platforms, including its NewgenONE product. The two companies share a similar-sounding name but have no shared ownership, leadership, or operations. We are also unaffiliated with newgenapps.io (see above). If you're looking for the AI delivery firm, you're in the right place at newgenapps.com. ## What industries and clients does NewGenApps work with? NewGenApps works with enterprises and founders across sectors — selling primarily to CIOs, CTOs, CDOs, Heads of Data, and technical founders who need AI that actually ships. Because our delivery is model-flexible and method-driven rather than industry-specific, we adapt to the data and workflows of whatever domain a client operates in. ## Who at the client side typically engages NewGenApps? Our buyers are usually the people accountable for AI outcomes: CIOs, CTOs, CDOs, Heads of Data, and founders. We are built for leaders who are done talking about AI and want a senior team that can take a real use case to production. ## How do I get started with NewGenApps? Start by booking a free 30-minute AI working session — no deck, no pitch, just a working conversation about where AI actually pays off for you. Reach out through /contact/ on newgenapps.com, and we will scope a concrete next step, whether that is an AI Compass audit, an AI in a Day sprint, or a 3-week POC. ============================================================================== # How Much Does AI Consulting Cost? | NewGenApps URL: https://newgenapps.com/how-much-does-ai-consulting-cost/ ============================================================================== # How Much Does AI Consulting Cost? *Last updated: September 2026. Prices are the starting prices listed on the [engagements page](/engagements/). Every external figure is attributed and dated in the text.* **In short:** AI consulting has no standard price, because the phrase covers stages that differ in scope by an order of magnitude. Ask which stage you are buying. NewGenApps publishes starting prices for six: AI in a Day from $2,500, AI Compass from $5,000, POC in 3 Weeks from $15,000, AI Rescue from $25,000, AI Production Build from $40,000, and an embedded team from $10,000 a month. Gartner has estimated that CIOs who do not understand how their generative AI costs scale could make a 500% to 1,000% error in their cost calculations (Gartner, 2024-10-21; also reported by CIO Dive), so buy one stage at a time, each with its own exit. ## How much does AI consulting cost? There is no market-standard price to quote. In preparing this page we looked for an independent, non-vendor benchmark of AI consulting fees and found none. The fee guides we found were published by firms that sell the service, and we have not repeated their figures. The nearest external evidence comes from the buyer's side. According to Gartner, as reported by CIO Dive from its 2024 IT Symposium keynote, organizations that deployed AI spent between $300,000 and $2.9 million in 2023 on the proof-of-concept phase alone. That is total organizational spend on the phase, not consulting fees, but it is still a spread of nearly ten to one for the same named phase. The likeliest explanation is scope rather than brand: which data, how many systems, what has to be proven, and who verifies it. So compare what two prices buy, not the two prices. ## The price ladder: six engagements, six starting prices Prices are in US dollars and shown as "from" figures, because the fee is set against a scoped engagement after a free 30-minute call. Source: the [engagements page](/engagements/). | Engagement | The question it answers | Duration | Starting price | |---|---|---|---| | [AI in a Day](/engagements/#ai-in-a-day) | Where does AI pay off for us, and what do we do first? | One working day | from $2,500 | | [AI Compass](/engagements/#ai-compass) | What is the costed, ranked roadmap, and is our data ready? | Typically 2 to 6 weeks | from $5,000, credited toward the build | | [POC in 3 Weeks](/engagements/#poc-in-3-weeks) | On our real data, does this clear the quality bar at an acceptable cost and latency? | 3 weeks | from $15,000 | | [AI Rescue](/ai-rescue/) | Can this stalled pilot be recovered, re-scoped or retired? | About 90 days | from $25,000, credited toward the build | | [AI Production Build](/engagements/#ai-production-build) | Can this validated use case run in production, independently verified? | Phased, milestone-gated | from $40,000 | | [Your AI Team](/engagements/#your-ai-team) | Can we have a durable senior AI capability without hiring it ourselves? | Month to month after an initial term of typically three months | from $10,000 per month for the smallest pod | The 30-minute working session costs nothing: [book it here](/book/). ## How are AI consulting engagements priced? Quotes usually take one of four forms, and they differ mainly in who carries the risk of an overrun. A fixed-scope fee gives the buyer a known price for a written deliverable and puts overrun risk on the vendor for that scope only. Time and materials bills the hours worked, which puts the risk on the buyer. A monthly retainer buys a standing team's capacity. An outcome-linked fee ties part of the price to a measured result, and is only as good as the baseline agreed before the work starts. NewGenApps' six published engagements are five fixed-scope or phased stages and one monthly retainer. The production build is priced in phases, with milestone gates, because its scope is easiest to bound once a proof has shown what has to be built. ## Pricing structures compared Our working definitions, from the buyer's side of the table: | Structure | You pay for | Who carries an overrun | Fits best | Watch for | |---|---|---|---|---| | Fixed scope, per stage | A written deliverable at a set price | The vendor, for the scope written down | Decisions and proofs: an audit, a POC, a triage | Anything outside the scope is a change request; read the change-control clause | | Time and materials | Hours worked at a rate | The buyer | Exploratory work with no defined end | A budget that is a forecast, not a commitment, and no built-in exit | | Monthly retainer | A standing team's capacity | Shared: you pay for capacity, the vendor answers for productivity | Continuous delivery and operation | A vague backlog; ask for a monthly review and a defined exit | | Outcome-linked fee | Part of the price tied to a measured result | Shared, by formula | Work with a measurable baseline | No agreed baseline, so the fee formula becomes the argument | ## Why do AI project budgets slip? Gartner's public statements point at the same place from three angles. In July 2024 it predicted that at least 30% of generative AI projects would be abandoned after proof of concept by the end of 2025, due to poor data quality, inadequate risk controls, escalating costs or unclear business value. That is a forecast, not a measured rate. Rita Sallam of Gartner said "costs aren't as predictable as other technologies." In October 2024 Gartner estimated that CIOs who do not understand how their generative AI costs scale could make a 500% to 1,000% error in their cost calculations (Gartner, 2024-10-21; also reported by CIO Dive). And in a March 2026 report Gartner predicted that through 2028 at least 50% of GenAI projects will overrun their budgeted costs, because of poor architectural choices and a lack of operational know-how (reported by Campus Technology, 2026-06-22). Notice what the last prediction blames: how the system is designed and run, not the model. That is the reason to price the stages separately. A proof runs for weeks at a fixed scope. A build runs longer and is phased. Running the system is a third cost, recurring and driven by usage. The FinOps Foundation's *State of FinOps 2026* found that 98% of the 1,192 FinOps practitioners it surveyed now manage AI spend, up from 31% two years earlier (2026-02-19). Ask every quote for the build fee and the run cost as separate lines. These are analyst predictions and press-reported figures, not audited measurements. Read them as direction, not as a forecast for your project. ## What drives the price of an AI engagement? - **Scope breadth.** One use case or several, one system or many. Each added use case is its own proof. - **Data readiness.** Clean, accessible, labeled data is quick to use. Data that must be found, cleaned or licensed before anything can be tested is a frequent source of delay. - **Integration depth.** Reading from a system is usually cheaper than writing back to it, and a system with no clean interface costs more than either. - **Verification.** Independent checking of the running system adds time and a second senior person. A quote that leaves it out is not the same product. - **Risk and compliance context.** Access controls, audit evidence and review steps add work wherever the output feeds a regulated decision. - **Run environment.** Expected usage, latency targets and hosting decide the recurring cost. - **Team and duration.** Seniority, calendar time and, in a retainer, the size of the pod. ## What is and is not included in the price? - **Credits.** The AI Compass and AI Rescue fees are credited toward a subsequent build, on the terms in the engagement letter. - **Third-party costs.** On the AI Production Build and Your AI Team, model usage, third-party licenses and cloud infrastructure are separate: billed at cost, or paid directly by you. - **Verification.** Every engagement that builds something ends with independent verification on the running system. The person who builds it does not sign it off. - **Proof.** Under NDA we do not name clients. Past work is described by sector on the [Work](/work/) page, and the method is on [How we work](/how-we-work/). ## How do you compare two AI consulting quotes? Put both quotes through the same seven checks before you compare the totals. 1. **Name the stage.** Is each quote for a decision, a build or an operating team? Do not compare a proof with a production build. 2. **Get the deliverables in writing.** A working system, an evaluation report, the code, an architecture document: each named. 3. **Ask for the acceptance criteria.** What result makes each deliverable done, and who decides? 4. **Ask what happens on "do not build".** A proof that can only end in "build" is a sales step, not a test. A reasoned no should count as a successful outcome. 5. **Split build from run.** Ask for the recurring cost at your expected volume, not the pilot's. 6. **Ask who verifies.** The person who builds the system should not be the only person who signs it off. 7. **Read the exclusions and the change clause.** What is out of scope, and what does a change cost? The same checks apply when you [choose an AI partner](/how-to-choose-an-ai-partner/) more broadly, and to any [build-versus-partner](/build-vs-buy-ai-team/) decision. ## Is the first conversation free? Yes. The 30-minute working session costs nothing, with no deck and no pitch. You leave with a view of where AI pays off and what a first proof would take. [Book it here](/book/). ## Are the starting prices fixed quotes? No. They are floors, not quotes. The fee is set against a scoped engagement after the call, so you see the deliverable and the price before you commit. Larger builds and larger pods cost more, and the tier is confirmed at scoping. ## Where should you start, and what does the first step cost? - **Several ideas competing, no clear first move:** the free call, or [AI in a Day](/engagements/#ai-in-a-day) from $2,500. - **One use case and a budget owner who needs a go or no-go on your own data:** [POC in 3 Weeks](/engagements/#poc-in-3-weeks) from $15,000. How to read one is in [what an AI proof of concept should prove](/ai-proof-of-concept/); the rate to ask a vendor for is in [what a good POC-to-production rate looks like](/poc-to-production-rate/). - **A pilot that stalled before production:** [AI Rescue](/ai-rescue/) from $25,000, with a real off-ramp after week one. - **A validated use case that now has to run in production:** [AI Production Build](/engagements/#ai-production-build) from $40,000, phased. - **How to measure whether it paid off:** [measuring AI ROI in production](/measure-ai-roi/). ## Related reading and sources *If you would rather see the price against your own scope, [book a 30-minute working session](/book/) or read the full [engagements and pricing](/engagements/) page. For the wider picture of what a firm like ours does, see [AI consulting](/ai-consulting/).* *Sources: NewGenApps, [engagements page](/engagements/); Gartner, ["Gartner Predicts 30% of Generative AI Projects Will Be Abandoned After Proof of Concept By End of 2025"](https://www.gartner.com/en/newsroom/press-releases/2024-07-29-gartner-predicts-30-percent-of-generative-ai-projects-will-be-abandoned-after-proof-of-concept-by-end-of-2025) (press release, July 2024); Gartner, ["Gartner Identifies Four Emerging Challenges to Delivering Value from AI Safely and at Scale"](https://www.gartner.com/en/newsroom/press-releases/2024-10-21-gartner-identifies-four-emerging-challenges-to-delivering-value-from-ai-safely-and-at-scale) (press release, 2024-10-21); L. Wilkinson, ["Gartner sounds alarm on AI cost, data challenges,"](https://www.ciodive.com/news/gartner-symposium-keynote-AI/730486/) CIO Dive, 2024-10-21; ["Gartner: Half of Gen AI Projects Could Exceed Budget by 2028,"](https://campustechnology.com/articles/2026/06/22/gartner-half-of-gen-ai-projects-could-exceed-budget-by-2028.aspx) Campus Technology, 2026-06-22 (citing Gartner, "10 Best Practices for Optimizing Generative and Agentic AI Costs," March 2026); FinOps Foundation, [State of FinOps 2026](https://www.prnewswire.com/news-releases/state-of-finops-survey-ai-value-and-skills-top-priorities-as-finops-matures-across-technology-value-98-manage-ai-90-saas-64-licensing-48-data-center-302691410.html) (press release, 2026-02-19).* ============================================================================== # How to Rescue a Failing AI Project: A Week-One Triage | NewGenApps URL: https://newgenapps.com/how-to-rescue-a-failing-ai-project/ ============================================================================== # How Do You Rescue a Failing AI Project? A Week-One Triage *Last updated: October 2026. External figures are attributed and dated in the text.* **In short:** Do not start by fixing a failing AI project, and do not start by cancelling it. Start with a one-week triage that ends in a written verdict: recover, re-scope or retire. RAND's August 2024 report, drawing on 65 interviews with experienced AI practitioners, cites estimates that more than 80% of AI projects fail and names five root causes. In our reading, four are about how a project was framed, fed and run rather than about the model. NewGenApps' AI Rescue runs that triage in week one, with a real off-ramp, inside a fixed-scope engagement of about 90 days that starts from $25,000. ## Should you rescue a failing AI project or cancel it? Neither yet. Both are outcomes of a decision, and the decision needs a short, evidence-based look at what exists. A triage returns one of three verdicts. **Recover:** the problem is still worth solving and a costed path to production exists. **Re-scope:** the problem is real, but the target as set cannot be reached or measured, so it is narrowed. **Retire:** the evidence says the project should not be carried further, and says why. A reasoned "retire" is a successful result. It costs the triage, not a second failed relaunch. If you are not yet sure the project has stalled, the [five-signal self-check](/has-your-ai-pilot-stalled-5-signal-diagnosis/) is quicker, and [why AI pilots fail](/why-ai-pilots-fail/) covers the upstream causes. ## How often do AI projects fail, and why? RAND's August 2024 report, *The Root Causes of Failure for Artificial Intelligence Projects and How They Can Succeed*, starts from an estimate rather than a finding: more than 80% of AI projects fail, twice the rate of failure for information technology projects that do not involve AI (RAND, RR-A2680-1, 2024-08-13). RAND introduces it with "by some estimates": a figure it cites, not a rate it measured. Its notes trace the figure to a July 2022 Fortune article and a November-December 2023 Harvard Business Review article. RAND's own contribution is a list of causes. Its authors interviewed 65 data scientists and engineers with at least five years of experience building AI/ML models, 50 in industry and 15 in academia, between August and December 2023. The industry interviews produced five leading root causes: - Stakeholders misunderstand or miscommunicate the problem AI is meant to solve. - The organization lacks the data to train an effective model. - The organization focuses on the latest technology rather than on real problems for its intended users. - Infrastructure to manage data and deploy finished models is inadequate. - The technology is applied to problems that are too difficult for AI. Three limits apply. "Failure" meant a project the organization perceived as one, so this is testimony, not a measured rate. Most interviewees were non-managerial engineers, which RAND says may skew the results toward leadership failures. And the scope was machine-learning projects: those that simply prompted a pretrained language model were excluded, so a pilot built that way sits outside what RAND studied. Use the list as questions to put to your project, not as a diagnosis. ## What does a week-one triage involve? In AI Rescue the first week is a structured forensic of the existing project. It produces a root-cause diagnosis, a verdict and a register of the dependencies a recovery would rest on, in six parts: 1. **Take delivery of what exists.** The code, data access, any evaluation artifacts, the integration points, the run history and the account of how the project stalled. 2. **Hear from the people involved.** At least one person who worked on or commissioned the project has to be available that week to recount what happened. 3. **Test each finding against the record.** A finding enters the diagnosis only if it traces to evidence in the code, the data or the run history, not to impression. 4. **Map findings to root causes.** Each goes under one or more of four headings: data, evaluation, integration and ownership. 5. **Define the production target and cost the path.** For a project worth continuing, state what "working in production" would mean and what reaching it would take. 6. **Write the verdict.** Recover, re-scope or retire, with the reasoning and the conditions that would change it. Your sponsor is needed that week for the verdict itself. A re-scope needs the narrowed target agreed before further work starts; a retire closes the engagement with the written rationale. ## What questions frame the verdict? Six questions help frame it, one for each way a project fails: the wrong problem, missing data, technology ahead of the user, missing infrastructure, a problem too hard for AI, and no owner. Each answer points toward recovering the project or retiring it. The table follows RAND's five causes, plus ownership. The verdict itself rests on the diagnosis under the four headings (data, evaluation, integration and ownership) and on the three questions in the next section. ## The week-one questions, mapped to the verdict | Reason a project fails | Question the triage asks | Points toward recover or re-scope | Points toward retire | |---|---|---|---| | The wrong problem, or no clear success | Who defined the problem, and would they sign today's definition of success? | The problem is real and a named owner will sign a measurable outcome | No one can say what outcome the project exists to change, or a few written rules would do the job | | Missing data | Does data like what production will supply exist, and may it be used? | The gaps can be closed from data you already hold | The data does not hold what the task needs and cannot be gathered in time | | Technology ahead of the user | Was the approach chosen for the user's problem or for the tool? | A simpler or more established approach would meet the need | Remove the technology and no user need remains | | Missing infrastructure | What stands between this model and a live process? | The gap is engineering work of a known size | The path to production needs investment no one will fund | | A problem too hard | At the bar the decision needs, has anything come close on real inputs? | Results so far on real inputs are near the bar, or there are none yet and nothing in the record rules the approach out | Results are far from the bar, and the gap lies in the approach, not the build | | Ownership and commitment | Who owns the outcome, and is the commitment funded beyond the pilot? | An owner exists or can be named this week | No one with budget will own the outcome | The table is our reading of RAND's causes, not RAND's framework. RAND's industry findings add that 84% of interviewees cited at least one leadership-driven cause, such as setting the wrong problem or underestimating the time needed, as a primary reason projects fail. If the data row is where your project is stuck, the standard we hold any data path to is on the [data-integrity contract](/ai-data-integrity-contract/) page. ## How is the recover, re-scope or retire call made? The call reduces to three questions, each answered in writing: is the problem still worth solving, does the evidence so far give a credible reason to expect the approach can reach the required bar on real inputs (the bar itself is agreed and measured in the weeks that follow), and is the path from here to production known and fundable? Yes to all three is recover. A real problem recoverable only with a narrowed task or an agreed data fix is re-scope. A problem not worth solving, an approach the record shows cannot reach the bar, or a path no one will fund is retire. ## The three verdicts at a glance | Verdict | What it says | What you receive | What happens next | |---|---|---|---| | Recover | A costed path to production exists | A costed, sequenced recovery plan and a defined production target | Recovery continues through stabilize-and-verify and relaunch; the fix is checked by someone other than its builder | | Re-scope | Recoverable only with a narrowed task or a data fix that has to be agreed | The same plan, for a narrowed target | The narrowed target is agreed in writing before any further work | | Retire | The evidence says not to carry this further, and why | A written rationale and the conditions that would change the verdict | The engagement closes; no further weeks are committed | The verdict states what evidence would reverse it, so a sponsor can see what would have to change. A reasoned "retire" meets the acceptance criterion for the verdict in full: it counts as delivered, not as a failed engagement. ## When is retiring the project the right call? When the evidence shows that the project cannot clear the bar it was meant to clear, was never needed, or has no funded way to production. One of RAND's industry interviewees describes the "never needed" case: teams told to apply machine learning to data whose main patterns a few if-then rules could have captured. Such projects might succeed in a narrow sense but were never necessary. Retiring the AI approach is not dropping the goal: the verdict can point to a rules-based or process fix. One case is usually a re-scope, not a retire. RAND notes that AI models are inherently probabilistic, and that leaders who expect certainty lose faith when a model falls short. In our judgment, if the decision needs a guarantee a probabilistic component cannot give, the fix is typically to let a person or a fixed rule decide and have the model assist. ## Who should make the call on a failing AI project? Ideally someone with no stake in the project's history. In a 1999 study, Mark Keil and Daniel Robey interviewed 42 information-systems auditors about troubled software projects. They report that no factor always turned a project around, but that in the majority of cases the move away from the failing course was triggered by actors such as senior managers, internal auditors or external consultants. Through de-escalation, they note, a troubled project may be turned around or sensibly abandoned. The study is exploratory, rests on auditors' judgments and predates modern AI: direction, not a rule. Our reading is that the people closest to a project are the least free to call it. Whoever makes the call, ask what they need from you in week one, what you will hold in writing at the end, and whether they earn more if the verdict is recover. [How to choose an AI partner](/how-to-choose-an-ai-partner/) covers the wider checks. ## What does AI Rescue cost, and what is the off-ramp? AI Rescue starts from $25,000, runs about 90 days at a fixed scope, and has a real off-ramp after week one (NewGenApps engagements page, 2026-09-29). That is the purpose of the first week: a retire verdict ends the engagement there, rather than committing the remaining weeks to a foregone conclusion. The fee is credited toward a subsequent production build, on the terms in the engagement letter. Ask any firm what you pay if the verdict is retire, and get it in writing before you sign. Prices across the ladder are in [how much AI consulting costs](/how-much-does-ai-consulting-cost/); full terms are on the [engagements page](/engagements/#ai-rescue). ## When is a rescue the wrong tool? When there is nothing yet to rescue, or the system already works. Match the situation to the rung: - **An idea not yet tested on real data:** a proof of concept comes first; see [what it should prove](/ai-proof-of-concept/). - **No clear view of where AI pays off:** start with [AI in a Day](/engagements/#ai-in-a-day) or [AI Compass](/engagements/#ai-compass). - **A validated use case that has to run in production:** see [what an AI production build includes](/what-does-an-ai-production-build-include/) and the checks to run first in [AI production readiness](/ai-production-readiness/). - **Not sure which applies:** [which AI engagement you need](/which-ai-engagement-do-you-need/) sorts it. ## Related reading and sources *If you have a project in this state, [book a 30-minute working session](/book/). Related reading: [how AI Rescue works](/ai-rescue/), the [taxonomy of production AI failures](/production-ai-failure-taxonomy/), and the [data-integrity contract](/ai-data-integrity-contract/).* *Sources: RAND, ["The Root Causes of Failure for Artificial Intelligence Projects and How They Can Succeed: Avoiding the Anti-Patterns of AI"](https://www.rand.org/pubs/research_reports/RRA2680-1.html), J. Ryseff, B. De Bruhl and S. J. Newberry, report RR-A2680-1 (published 2024-08-13; DOI 10.7249/RRA2680-1); M. Keil and D. Robey, ["Turning Around Troubled Software Projects: An Exploratory Study of the Deescalation of Commitment to Failing Courses of Action,"](https://jmis-web.org/articles/531) Journal of Management Information Systems 15(4): 63-87 (1999); NewGenApps, AI Rescue scope on the [engagements page](/engagements/#ai-rescue) (2026-09-29).* ============================================================================== # Kiro & Spec-Driven Development: AWS's Agentic IDE, Explained | NewGenApps URL: https://newgenapps.com/kiro-spec-driven-development/ ============================================================================== # Kiro & Spec-Driven Development: AWS's Agentic IDE, Explained **In short:** Kiro is AWS's agentic development environment (an IDE and CLI) built for *spec-driven development* — a workflow where you first turn a prompt into a structured, human-readable specification (requirements, design, tasks), and the agent writes code against that spec. The spec becomes the source of truth; the code is a build artifact. That inverts the prompt-to-code model of tools like Cursor, GitHub Copilot, and the Amazon Q Developer that Kiro replaces. It's a genuinely useful idea with real tradeoffs — worth understanding on its merits, not the hype. This is an explainer, not a pitch. We'll define the terms, describe how Kiro works, compare it fairly to the alternatives, and say plainly where it helps and where it's overkill. At the end, a short honest note on how NewGenApps works — because we build in this spec-first, agentic paradigm ourselves, with our own toolchain. > **Quick disambiguation.** This page is published by **NewGenApps** (New Generation Applications Pvt Ltd, newgenapps.com), a senior AI delivery firm. We are not affiliated with **Newgen Software Technologies Ltd** (newgensoft.com, the NewgenONE low-code/ECM platform) or with newgenapps.io. And we are not affiliated with AWS — this is an independent explainer. ## What is Kiro? Kiro is an AI coding tool from AWS that AWS itself describes as built "from the ground up" for spec-driven development. In AWS's words, it helps developers "turn prompts into executable specs, validate code correctness to find bugs unit tests miss, and build across large codebases with parallel agents." A few things are worth pinning down, because they shape everything else: - **It's an agentic environment, not just autocomplete.** Kiro ships as an IDE and a CLI. Alongside inline suggestions and chat, it runs agents that plan and implement changes across a codebase, and it supports MCP (Model Context Protocol) for connecting to external tools and data. - **It replaces Amazon Q Developer.** AWS has stated Kiro is the successor to Amazon Q Developer. Per AWS's own end-of-support announcement, new Q Developer signups are blocked as of May 15, 2026, and Q Developer IDE plugins and paid subscriptions reach end of support on April 30, 2027 — a roughly 12-month transition window. Kiro launched in preview in 2025 and rolled out internationally in 2026. - **It routes across multiple models.** Kiro isn't tied to a single model. Its documentation lists several — Claude Opus, Sonnet, and Haiku, plus open-weight options — and an "Auto" mode that "picks the best model for the task based on complexity, factoring in model quality, latency, and cost." As an AWS product it draws on Amazon Bedrock as the model layer, which in principle lets teams route to Bedrock-available models for cost, capability, or compliance reasons. The consistent through-line: Kiro is trying to put *structure* around agentic coding rather than let an agent free-associate its way through a codebase. ## What is spec-driven development? Spec-driven development is a workflow where a written specification — not a prompt, and not the code itself — is the primary artifact you edit and version. You describe *what* the system should do; an agent proposes *how*; and only after that agreement does code get generated. If you want a change, you change the spec and regenerate, rather than nudging the code directly. In Kiro, that spec is typically split into three linked parts: 1. **Requirements** — what the feature must do, written as structured, testable statements rather than a loose paragraph. Kiro uses **EARS** (Easy Approach to Requirements Syntax), a notation originally developed for safety-critical and aerospace systems, to keep each requirement unambiguous and checkable. That precision is the point: the agent codes against a specification, not an open-ended request. 2. **Design** — the system design and architectural decisions that satisfy those requirements. This is where the reasoning lives, documented explicitly so a teammate (or the agent, later) can see *why* something was built a certain way. 3. **Tasks** — a sequenced list of implementation steps derived from the design, which agents then work through. The underlying claim is straightforward: **agentic coding without formal structure tends to drift.** An agent given only a prompt can wander from the intended architecture, quietly miss edge cases, and produce code that demos well but buckles under production load. A spec is an attempt to separate *thinking* from *doing* — to force the hard decisions to the surface before thousands of lines get written on a shaky foundation. This isn't a new insight. It's the old wisdom that "the requirements are the hard part" applied to a world where writing the code is suddenly cheap. When code is expensive, you skimp on specs to save effort. When an agent can generate code in minutes, the spec is the only place your judgment still compounds. ## How does Kiro work? In practice, a Kiro workflow looks roughly like this: - **Prompt to spec.** You give Kiro a natural-language description of what you want. It generates the requirements/design/tasks spec — expanding your prompt into structured detail. (One frequently cited user reaction: Kiro wrote out user stories "like a product manager and capture[d] so many details that I didn't even need to mention.") - **Review and edit the spec.** You correct the requirements and design *before* code exists. This is the leverage point — mistakes caught here are cheap. - **Agents implement the tasks.** Kiro works through the task list, generating code across the codebase, with support for parallel agents on large projects. - **Correctness checking.** Beyond example-based unit tests, Kiro emphasizes property-based testing — asserting "rules that must hold across all inputs instead of checking a few examples" — to catch edge cases that a handful of hand-written tests miss. Two other pieces round it out: - **Agent Hooks** are event-driven automations that fire on triggers like file save or commit — for example, regenerating tests or updating docs automatically when a file changes. This is what lets some of the workflow run in the background rather than on every manual prompt. - **Steering files** give Kiro persistent, project-level context — your architecture, conventions, and constraints — so the agent's output stays consistent with how *your* codebase actually works, across sessions. Together, these are Kiro's answer to the two failure modes of naive agentic coding: drift from architecture (steering files + specs) and silent gaps in correctness (property-based tests + hooks). ## How is Kiro different from Cursor, Copilot, or Amazon Q? The cleanest way to see the difference is the direction of the workflow. Most popular AI coding tools are **code-first**: you prompt, the tool edits code, you iterate on the code. Kiro is **spec-first**: you prompt, the tool produces a spec, you iterate on the spec, and code is generated from it. That's a real philosophical split, and neither direction is universally "better" — they optimize for different things. | Dimension | Spec-driven (Kiro) | Prompt-to-code (Cursor, Copilot, Q Developer) | |---|---|---| | **Primary artifact** | The spec (requirements/design/tasks) | The code, edited directly | | **Where you catch mistakes** | Before code exists, in review of the spec | After code exists, by reading diffs | | **Best fit** | Larger features, multi-step changes, teams needing shared intent | Quick edits, exploration, "make this function do X" | | **Overhead** | Higher up front — you write and review a spec | Lower up front — you just prompt | | **Risk it targets** | Architectural drift, missed edge cases | Slow typing, boilerplate, context switching | | **Team coordination** | Spec is a shared, reviewable source of truth | Coordination happens in PRs and conversation | A fair reading: Cursor and Copilot are superb at *velocity in the small* — inline completion, fast local edits, staying in flow. Kiro is betting on *discipline in the large* — that for substantial features, the time spent nailing a spec pays back by preventing rework. Amazon Q Developer sat closer to the Cursor/Copilot end; Kiro is AWS deliberately moving toward the spec-first end. None of these is a lock-in-free utopia, and none is strictly superior. Cursor and Copilot also support agentic, multi-file workflows, and both are widely used and mature. Other credible tools in this space include Claude Code, Windsurf, Aider, and Zed's agent features. The honest takeaway is that spec-driven development is a *methodology* — one you can practice, to a degree, in several of these tools — and Kiro is the most opinionated productization of it so far. ## When does spec-driven development help — and when is it overkill? **It helps when:** - The change is **large or multi-step** — a new feature, a refactor across modules, anything where getting the architecture right matters more than typing speed. - **Multiple people need shared intent.** A spec is a reviewable artifact a team can agree on before code is written, which beats reconstructing intent from a diff. - **Edge cases and correctness are load-bearing** — regulated domains, data integrity, systems that break expensively in production. - You're working in a **large or unfamiliar codebase**, where drift from existing conventions is a real risk. **It's overkill when:** - The task is **small and local** — a one-line fix, a quick script, a tweak to one function. Writing a spec for that is ceremony. - You're **exploring or prototyping**, where the whole point is to poke at ideas fast and throw most of them away. - The requirements are **genuinely unknown** and can only be discovered by building. Sometimes the fastest way to learn what you want is to write throwaway code, not a spec. The mature stance is that spec-first and code-first are **modes, not tribes.** Good engineers already switch between them — a napkin sketch for a big feature, straight-to-code for a hotfix. Kiro's contribution is making the spec-first mode a first-class, tool-supported path rather than a discipline you have to impose by hand. ## Where we land At NewGenApps, we build in this spec-first, agentic paradigm — because we think the core idea is right: for anything that has to survive production, the judgment belongs in the specification, and the code is a build artifact downstream of it. That conviction predates Kiro; the tool is a well-executed expression of a direction the field has been moving for a while. (For how this shows up in our delivery, see [How we work](/how-we-work/).) Our own toolchain is Claude Code plus a library of internal skills — that's our delivery practice, and it's where our spec-first workflow actually runs. (More on that in [How we build](/claude-native-ai/).) But the deliberate point is **model- and vendor-flexibility**: nothing we build locks a client into one model or one cloud. Where a client already runs on AWS, Amazon Bedrock's routing across models is a reasonable place to land; where they don't, we don't push them there. We reach for the tools that fit the problem, and we're careful that none of them becomes a trapdoor. We're not an AWS partner, we hold no badge we haven't earned, and we don't sell any vendor's roadmap. What we sell is judgment and verified delivery — **production AI, proven.** Whatever generates the code, the part that matters is the same part Kiro is built around: getting the specification and the correctness right. That's what we check, with the same rigor we bring to any agentic system — see our [AI evaluation harness](/ai-evaluation-harness/) for how we prove a system actually works before it ships. If you're weighing spec-driven development for real work, the useful question isn't "Kiro or Cursor?" It's "where in my workflow does structure pay for itself, and where is it friction?" Answer that honestly and the tool choice mostly follows. ============================================================================== # How to Measure AI ROI in Production | NewGenApps URL: https://newgenapps.com/measure-ai-roi/ ============================================================================== # How to Measure AI ROI in Production *Last updated: July 2026. A finance-facing method for measuring the return on a production AI system — why a live system is not a returning one, what to instrument, and the counterfactual that turns "it's running" into a number a CFO can defend.* > **In short:** AI ROI is the business return a system produces net of its fully-loaded cost — not whether it is live, adopted, or accurate. A running system is not a returning one. Measuring ROI means fixing a baseline before launch, isolating the AI's contribution with a control or holdout, and counting every cost, not just the licence. Most AI programs are reported on the wrong axis. The status update says the system is deployed, usage is climbing, and accuracy is high — and none of those three facts is a return. This page is about the fourth axis, the one finance actually funds: did the system move a business number by more than it cost to run? It is the finance-facing companion to the engineering discipline of [liveness vs outcome](/production-ai-liveness-vs-outcome/) — one level up, where "the work got done correctly" still has to become "the work was worth doing." ## What is AI ROI in production, and why isn't a live system a returning one? AI ROI is the net business value a production system produces divided by its fully-loaded cost, over a defined period, and attributable to the system rather than to everything else that changed. Three questions get conflated, and only the third is ROI: - **Is it up?** — liveness. A process is running, the endpoint answers. Infrastructure, not value. - **Is it right?** — outcome correctness. The system produces fresh, valid results, which is the subject of [AI evaluation](/ai-evaluation/) and outcome monitoring. - **Is it worth it?** — ROI. The correct results changed a P&L line by more than the system's total cost. A system can clear the first two bars and fail the third completely: accurate, well-adopted, and still net-negative once you count the build, the tokens, the integration, and the humans reviewing its output. Liveness is not outcome, and outcome is not return. Confusing any of the three is how a program reports success for a year and shows nothing to finance at the end of it. ## Why does a live AI system so often show no return? Because the return was never instrumented — no baseline, no counterfactual, and a cost base counted at the licence line instead of the fully-loaded one. The failure record is now large and consistent, and it is a measurement failure, not a model one. **MIT Project NANDA found that roughly 95% of enterprise generative-AI pilots show no measurable P&L impact, after an estimated $30–40 billion in enterprise spend, with only about 5% reaching rapid revenue acceleration** (MIT Project NANDA, *The GenAI Divide: State of AI in Business 2025*; Fortune coverage, 2025-08-18). The incumbents' own surveys agree. McKinsey found only about 39% of organizations report any EBIT impact from generative AI at the enterprise level, and of those, most attribute less than 5% of EBIT to it (McKinsey, *The State of AI*, March 2025). BCG found 74% of companies had yet to show tangible value from AI, with only 4% generating significant value across functions (BCG, *Where's the Value in AI?*, October 2024). And the direction of travel is not improving on its own: S&P Global Market Intelligence found the share of firms abandoning most of their AI initiatives rose to 42%, up from 17% the prior year (S&P Global, *Voice of the Enterprise: AI & ML*, 2025). None of these is a statement about model accuracy. They describe systems that ran and were never measured against a baseline they could be proven to have beaten. The mechanism sits directly under the 95%; the [production-AI failure taxonomy](/production-ai-failure-taxonomy/) names it as *liveness mistaken for outcome*, and [why AI pilots fail](/why-ai-pilots-fail/) reads the same gap from the delivery side. ## Model metric vs business outcome: what should finance measure? Measure the business outcome the system was funded to move — not the model metric that is easy to export from a dashboard. A model metric describes the model; a business outcome describes the P&L. They are not interchangeable, and the model metric is often not even a reliable measure of the model: when Scale AI built a fresh test in the style of a popular benchmark, some models dropped by up to roughly 13 points once the questions could not have been memorized (arXiv 2405.00332, 2024). A number that moves with the test harness cannot anchor a return. | What teams report (the proxy) | Why it is not ROI | The business outcome to instrument instead | |---|---|---| | Model accuracy / F1 | Measures the model, not the money; can be contaminated or saturated | Error-driven cost avoided, or revenue from decisions the accuracy enabled | | Adoption / active users | Usage is a cost driver, not a benefit | Output per person-hour, or headcount-hours redeployed to higher-value work | | Latency / uptime | Infrastructure health, not value delivered | Cycle-time reduction that shortens a revenue or cash cycle | | Tokens or requests processed | A volume of spend, reported as if it were output | Unit economics: fully-loaded cost per successful business task | | "Went live" milestone | A timestamp, not an outcome | Baselined change in the target KPI, net of cost, measured against a control | The right-hand column is the only one a CFO can put in a business case. The left-hand column is what most programs actually track. ## How do you measure AI ROI in production? Treat ROI as an instrumented measurement, not a post-hoc slide. Six steps, in order: 1. **Name the outcome metric and its baseline before you build.** State the single business number the system exists to move — cost per claim, revenue per rep, hours per case — and record its current value. If no one can state the metric and its baseline before launch, the return is already unmeasurable. 2. **Choose a counterfactual.** Decide *before* deployment how you will separate the AI's effect from everything else: a randomized control group, a held-out cohort, or a clean before/after with a comparison line. Without one, any later number is a story, not a measurement. 3. **Instrument the outcome, not the proxy.** Wire the business metric to the live system so the target KPI is measured on real traffic — the same discipline as [proving the system works in production](/how-do-you-know-an-ai-system-works-in-production/), pointed at the money rather than the mechanics. 4. **Count the fully-loaded cost.** Total cost of ownership, not the model bill: build and integration, tokens and compute, monitoring, retraining, and — the line most often omitted — the human hours spent reviewing and correcting output. A system that shifts work from doing to checking has moved cost, not removed it. 5. **Attribute conservatively.** Subtract what would have happened anyway (market trends, seasonality, other initiatives), discount the ramp period, and report a range, not a point. An honest smaller number survives audit; an inflated one gets clawed back. 6. **Review on a finance cadence.** Re-measure at a set interval against the baseline and the counterfactual, because a return proven at launch decays — a green result is a timestamp, not a guarantee. This is the return-side of [deploy and verify](/deploy-and-verify-ai/). The two sides of the ledger those steps populate: | Benefit lines (measured, not asserted) | Cost lines (fully loaded) | |---|---| | Incremental revenue attributable to the system | Build and integration (one-time, amortized) | | Cost reduced or avoided vs. baseline | Tokens, inference, and compute | | Person-hours redeployed (valued at loaded rate) | Human-in-the-loop review and correction | | Cycle-time or cash-cycle improvement | Monitoring, evaluation, and retraining | | Risk or error-rate reduction (quantified) | Licences, platform, and maintenance | Where data quality is too poor to establish a baseline, the honest first finding is that the return is not yet measurable — which is itself a result, since data quality is the single most-cited barrier to enterprise GenAI, named by 43% of chief data officers (Informatica, *CDO Insights 2025*, January 2025, n=600). ## What should a CFO ask before approving the next AI budget? The vetting conversation is short, and it screens for measurement discipline rather than model sophistication: - **What is the baseline, and who recorded it before the build started?** No baseline, no ROI. - **What is the counterfactual — how will we know the AI caused the change, not the market?** - **What is the fully-loaded cost per successful task, including human review?** - **When success is claimed, who verifies it independently of the team that built it?** A partner who leads with a model and cannot answer these is selling liveness. One who leads with the measurement method is selling a return. That distinction is the core of [how to choose an AI partner](/how-to-choose-an-ai-partner/), and it applies whether you [build in-house or hire](/build-vs-buy-ai-team/) and whether you engage a [boutique or a Big-4 firm](/boutique-vs-big-4-ai-consulting/). The return, notably, does not come from which model you pick — Claude or otherwise — but from whether the outcome is instrumented and the cost is counted in full. ## Frequently asked questions **What is a good AI ROI?** There is no universal threshold; the meaningful test is whether the measured business return exceeds the fully-loaded cost against a defensible baseline and counterfactual. A modest, audit-proof return beats a large one built on unattributed before/after numbers. Report a range and the assumptions behind it, not a single headline multiple. **How is AI ROI different from a model metric like accuracy?** Accuracy describes the model; ROI describes the money. A system can be highly accurate and still net-negative once review labour and compute are counted, and a model metric can shift with the test harness itself — Scale AI measured drops of up to roughly 13 points on a fresh benchmark (arXiv 2405.00332, 2024). Instrument the business outcome, not the proxy. **Why do most AI pilots show no ROI?** Because the return was never instrumented. MIT Project NANDA found roughly 95% of enterprise GenAI pilots show no measurable P&L impact (2025; Fortune, 2025-08-18) — not because the models failed, but because success was defined as launch or usage, with no baseline, no counterfactual, and cost counted only at the licence line. See [why AI pilots fail](/why-ai-pilots-fail/). **How long before an AI system shows ROI?** It depends on scope and data readiness, and the honest answer is set by the review cadence, not a launch date. A narrow integration against clean data can show a measurable return in a quarter; anything requiring a data-quality fix first takes longer. Re-measure on a finance cadence rather than declaring victory at go-live. --- *If your AI is in production and finance still cannot see the return, the gap is almost always the measurement, not the model — a missing baseline, an absent counterfactual, or a cost base counted at the licence line. That instrumentation is what we build. See [how we work](/how-we-work/), [AI consulting](/ai-consulting/), or [book a 30-minute working session](/book/).* *Sources: MIT Project NANDA, "The GenAI Divide: State of AI in Business 2025" (2025; Fortune, 2025-08-18); McKinsey QuantumBlack, "The State of AI" (March 2025); BCG, "Where's the Value in AI?" (October 2024); S&P Global Market Intelligence, "Voice of the Enterprise: AI & ML" (2025); Informatica, "CDO Insights 2025" (January 2025, n=600); Scale AI, "A Careful Examination of LLM Performance on Grade School Arithmetic," arXiv 2405.00332 (2024). Figures are attributed to their original sources and are not NewGenApps measurements.* ============================================================================== # What Is a Good PoC-to-Production Rate for AI? | NewGenApps URL: https://newgenapps.com/poc-to-production-rate/ ============================================================================== # What Is a Good PoC-to-Production Rate for AI? *Last updated: July 2026. A definitional reference on the PoC-to-production rate — what the metric means, why nobody publishes a benchmark, and how to read any figure a vendor gives you against the documented base rate.* **In short:** a PoC-to-production rate is the share of AI proofs of concept a team or vendor moves into live production and keeps there, producing a measured outcome. No one publishes a benchmark, so judge any figure against the base rate: only about 12% of enterprise PoCs reach production (IDC, 2025) and roughly 5% show measured P&L impact (MIT Project NANDA, 2025). The [buyer checklist for choosing an AI partner](/how-to-choose-an-ai-partner/) names one number as the single most useful screening question: "What is your PoC-to-production conversion rate — as a number?" This page defines that metric precisely, because a number is only as good as the definition behind it, and benchmarks it against the published base rates so you can tell a strong answer from a meaningless one. ## What is a PoC-to-production rate for AI? A PoC-to-production rate is the fraction of AI proofs of concept that cross from a controlled prototype into a live production system that produces a measured, verified outcome. Stated as a formula: the numerator is PoCs that reached production *and* delivered against a pre-registered success metric; the denominator is all PoCs started in a defined window, including the ones quietly shelved. Three parameters decide what the number actually means, and a rate quoted without them is not interpretable: - **The bar.** "Reached production" (the system is deployed and live) is a much lower bar than "produced a measured business outcome." A rate can be honest at either bar — but only if it says which one. This is the [liveness-versus-outcome distinction](/production-ai-liveness-vs-outcome/): a system can pass every health check and still deliver nothing of value. - **The denominator.** A rate that counts only PoCs that shipped, silently dropping the ones that were killed, is not a conversion rate — it is a survivorship figure. The denominator must include abandonments. - **The window.** A PoC started last month has not had time to convert or fail. A credible rate is measured over a cohort old enough to have resolved — typically PoCs initiated 12 to 18 months earlier. Get those three straight and the number tells you something. Leave them vague and a high percentage is decoration. ## Is there a published benchmark for a good PoC-to-production rate? No. This is an open lane: as of mid-2026, no analyst house or vendor publishes a standard PoC-to-production conversion benchmark you can hold a firm against. What the research *does* give you is the base rate — the enterprise-wide floor that any credible vendor figure has to clear. The table below assembles it from the major dated studies, sorted by the bar each one measures. | Source (dated) | What it measured | The bar it sets | Base rate | |---|---|---|---| | IDC, *CIO Playbook 2025* (in partnership with Lenovo), February 2025 | PoCs reaching widescale deployment | Reaches production | ~12% — about 4 of every 33 PoCs (≈88% do not) | | S&P Global Market Intelligence / 451 Research, *Voice of the Enterprise: AI & ML*, October 2025 (n=1,006) | Firms abandoning most AI initiatives | Reaches production | 42% of firms abandoned most initiatives (up from 17%) | | Gartner, press release July 2024 | GenAI projects abandoned after proof of concept | Survives past PoC | ≥30% abandoned after PoC by end of 2025 | | Deloitte, *State of AI in the Enterprise 2026* (fielded Aug–Sep 2025, n=3,235) | Orgs moving ≥40% of AI experiments to production | Reaches production (portfolio) | 25% have; 54% expect to within 3–6 months | | MIT Project NANDA, *The GenAI Divide*, August 2025 | Pilots showing measurable P&L impact | Measured outcome | ~5% show measurable P&L impact (≈95% do not) | The pattern is the point. **Only about 4 of every 33 enterprise AI proofs of concept reach production — roughly 12% (IDC, *CIO Playbook 2025*, in partnership with Lenovo, February 2025) — and by MIT's stricter measure only about 5% show measurable P&L impact (MIT Project NANDA, *The GenAI Divide*, August 2025).** The floor at the reach-production bar and the floor at the outcome bar are different numbers, and both are low. A clear majority of enterprise PoCs still stall before either one — which is exactly why the mechanisms are mapped in the [production-AI failure taxonomy](/production-ai-failure-taxonomy/). ## So what counts as a good rate? A good rate clears the base rate by a wide margin, is stated at the outcome bar, and survives interrogation of its denominator, window, and selection. The headline percentage matters less than whether those four conditions hold — because all four are gameable, and a number that fails any of them can look excellent and mean nothing. The closest thing to a partner benchmark comes from the build-versus-buy evidence. MIT Project NANDA found that AI systems delivered through expert partnerships reached deployment roughly 67% of the time, against roughly 33% for internally built tools — partnerships shipped about twice as often, where "success" meant deployment beyond pilot with measurable KPIs tracked at six months (MIT Project NANDA, *The GenAI Divide*, August 2025). That gives a directional anchor: a senior-led delivery partner should be posting a rate far above the ~12% enterprise floor and in the neighborhood of that partner figure — but the anchor is a sanity check, not a target to hit on paper. We treat the build-versus-buy trade-off in full in [should you build an in-house AI team or hire a partner](/build-vs-buy-ai-team/). The trap is optimizing the number instead of the outcome: a firm that only accepts pre-qualified, low-risk PoCs can post a 90% conversion rate that measures its intake filter, not its delivery. A high rate is a starting question, not an answer. ## Why does the definition matter more than the number? Because the same headline percentage can describe a strong delivery record or a marketing artifact, and only the definition tells them apart. Two firms can both claim "70%" — one measured at the outcome bar across every PoC it started in an 18-month window, the other measured at the reach-production bar on a hand-picked subset over an undefined period. The first is evidence; the second is noise wearing the same number. This is why the metric has to be defined at the outcome bar to be worth anything. Reaching production is not the same as working: a deployed system can serve stale data behind a green health check, or pass a contaminated benchmark and fail on live inputs — the failure modes catalogued in the [failure taxonomy](/production-ai-failure-taxonomy/) and in [why AI pilots fail](/why-ai-pilots-fail/). A conversion rate counted at the "it's live" line rewards exactly the systems that liveness monitoring waves through. A conversion rate counted at "a measured outcome appeared, and someone other than the builder confirmed it" rewards the systems that actually pay off. That is the whole distinction between a [pilot and production](/pilot-vs-production-ai-what-changes/), reduced to a single metric. ## How do you interrogate a vendor's stated PoC-to-production rate? Ask six questions in the discovery call. Any firm that has genuinely crossed the gap can answer all six with specifics; a firm that cannot will pivot to logos or adjectives. 1. **Denominator.** Does the rate include PoCs that were killed or quietly shelved, or only the ones that shipped? If abandonments are excluded, it is not a conversion rate. 2. **Bar.** Does "converted" mean the system went live, or that it produced a measured business outcome? Ask for the outcome definition, in writing. 3. **Window.** PoCs started when? Insist on a cohort old enough to have resolved — a rate on last quarter's work is premature by construction. 4. **Selection.** Do you take on hard PoCs, or only pre-qualified ones? A high rate on cherry-picked engagements measures the intake filter, not delivery. 5. **Verification.** Who confirmed each outcome, and was it the team that built it? An independently verified outcome is worth more than a self-graded one — the reasoning is in [how to choose an AI partner](/how-to-choose-an-ai-partner/). 6. **One walk-through.** Ask for an anonymized account of one PoC that converted and one that did not, and why. The failures are more informative than the successes. If the answers are concrete, the number means something. If they are vague, the number is decoration — and the honest read across firm types, boutique or global, is the same test applied evenly ([boutique vs Big-4 AI consulting](/boutique-vs-big-4-ai-consulting/)). ## Frequently asked questions **What is a good PoC-to-production rate for AI?** A good rate clears the enterprise base rate — roughly 12% of PoCs reach production (IDC, *CIO Playbook 2025*, February 2025) and about 5% show measured P&L impact (MIT Project NANDA, August 2025) — by a wide margin, is stated at the outcome bar rather than the "it's live" bar, and holds up when you check its denominator, window, and selection. A specific number matters far less than whether those conditions are met. **What is the average PoC-to-production rate?** The enterprise average sits low. IDC found only about 4 of every 33 AI proofs of concept reach production — roughly 12% (IDC / Lenovo, *CIO Playbook 2025*, February 2025). S&P Global reported that 42% of firms abandoned most of their AI initiatives in 2025, up from 17% the year before (S&P Global Market Intelligence / 451 Research, October 2025). At the stricter outcome bar, MIT's ~95% no-measurable-impact figure implies about 5% (MIT Project NANDA, August 2025). **Why is there no industry benchmark for a good PoC-to-production rate?** Because the metric is not standardized: firms define "PoC," "production," and "success" differently, use different denominators, and measure over different windows, so no analyst has published a comparable cross-vendor rate. That is precisely why the base rates in the studies above are the right yardstick — and why any single vendor number has to be interrogated rather than accepted. **Does a high PoC-to-production rate mean a vendor is better?** Not on its own. A firm that only accepts low-risk, pre-qualified PoCs can post a very high rate that reflects its intake filter, not its delivery skill, and a firm that counts "deployed" instead of "delivered a measured outcome" can post a high rate on live-but-useless systems. A high number is a reason to ask the six interrogation questions, not a reason to stop asking. **How do I verify a vendor's PoC-to-production rate?** Shift from the number to the method. Ask who confirmed each outcome and whether it was the team that built the system; a separate, read-only check on the live system is worth more than a self-report. Then ask for one anonymized walk-through of a PoC that converted and one that did not. A firm that has crossed the gap can produce both; the [AI Rescue](/ai-rescue/) engagement exists for pilots that have not. --- If you are vetting a shortlist and want a straight read on whether a stalled pilot belongs in production — measured at the outcome bar, verified independently, not at the demo — put the question to us first. See [how to choose an AI partner](/how-to-choose-an-ai-partner/), [what an AI proof of concept should prove](/ai-proof-of-concept/), [AI consulting](/ai-consulting/), or [book a 30-minute working session](/book/). *Sources: IDC, "CIO Playbook 2025: It's Time for AI-nomics," in partnership with Lenovo (February 2025); S&P Global Market Intelligence / 451 Research, "Voice of the Enterprise: AI & Machine Learning, Use Cases 2025" (published October 2025, n=1,006); Gartner press release (July 2024); Deloitte, "State of AI in the Enterprise 2026" (fielded August–September 2025, n=3,235); MIT Project NANDA, "The GenAI Divide: State of AI in Business 2025" (August 2025; Fortune coverage, 2025-08-18). Figures are attributed to their original sources and are not NewGenApps measurements.* *NewGenApps — production AI, proven. Stay a step ahead, always.* ============================================================================== # The Production AI Failure Taxonomy: Why AI Projects Fail to Reach Production | NewGenApps URL: https://newgenapps.com/production-ai-failure-taxonomy/ ============================================================================== # The AI Pilot Failure Taxonomy — Why Most AI Projects Never Reach Production (and What the Minority Do Differently) Most enterprise AI pilots do not fail because the model is weak — they fail because the organization never bridges the gap between a working demo and a governed production system. The best-evidenced 2024–2025 studies converge on a single thesis: the scarce discipline that gets AI into production is not better modeling but trustworthy data, a real success metric, controlled access, and independent verification. This page assembles the verifiable failure statistics, organizes the root causes into a taxonomy, and distills what the surviving minority do differently — with every figure cited to its primary source and labeled for what it actually measures. > A note on honesty before the numbers: the headline "failure rates" below measure *different things* — zero P&L return, project failure, initiative abandonment, PoCs that never reached production — on different denominators. They are converging evidence of a large production gap, not one interchangeable number. We flag which are measurements and which are forecasts, and we debunk the most-repeated myth in this space at the end. ## How often do AI pilots fail? There is no single "AI failure rate." There is a spectrum of credible figures, each measuring a distinct thing. Here are the verified, load-bearing statistics: | Figure | What it actually measures | Type | Source (Year) | | --- | --- | --- | --- | | **95%** | Organizations investing in GenAI that saw **zero measurable P&L return**; only ~5% achieved rapid revenue acceleration. *Not* a technical pilot-failure rate. | Industry study (ROI) | MIT NANDA, *The GenAI Divide: State of AI in Business 2025* (2025) | | **at least 30%** | GenAI projects predicted to be **abandoned after proof-of-concept by end-2025** (poor data quality, weak risk controls, escalating cost, unclear value). | **Forecast** | Gartner (2024) | | **more than 80%** | AI projects that fail — "by some estimates," ~2x the rate of non-AI IT projects. RAND reports this figure from others' estimates; its own contribution is the five root causes. | Synthesis of estimates | RAND Corporation, RRA2680-1 (2024) | | **42%** (up from **17%**) | Businesses that **scrapped most of their AI initiatives** in 2025 vs 2024 — a ~2.5x year-over-year jump. The average org scrapped **46% of PoCs** before production. | Survey (measured) | S&P Global Market Intelligence (2025) | | **~86%** | GenAI **PoCs that never reached production**: orgs ran ~37 PoCs but only ~5 shipped. | Analyst estimate | IDC (2024) | | **88% adopt / 39% EBIT / ~6%** | 88% use AI in ≥1 function, yet only 39% report any enterprise EBIT impact (most <5% of EBIT); only ~6% are "high performers." | Survey (measured) | McKinsey / QuantumBlack, *The State of AI in 2025* (2025) | **How to read this spectrum:** Gartner's 30% is a *prediction*; S&P Global's 42% and IDC's ~86% are *measured* abandonment/production outcomes; RAND's >80% is a *synthesis of "some estimates,"* not RAND's own measurement; MIT's 95% is a non-peer-reviewed industry report measuring *value capture*, not technical success. Conflating them is the single most common error in competing write-ups. ## Why do AI pilots fail? The failure taxonomy Across RAND, MIT, BCG, McKinsey and Gartner, the failure is overwhelmingly **organizational and data-related, not model quality**. BCG's resourcing split makes the point bluntly: AI leaders put ~70% of effort into people and process, ~20% into technology and data, and only ~10% into algorithms (BCG, 2024) — the difficulty is almost never the algorithm. Eight recurring categories emerge: | # | Category | Core evidence | Source (Year) | | --- | --- | --- | --- | | 1 | **Data quality / integrity / availability** | 60% of AI projects abandoned through 2026 without AI-ready data; 63% of orgs lack or are unsure of proper AI data-management practices | Gartner (2025) | | 2 | **Evaluation & measurement** (can't prove value) | 95% "no measurable P&L return"; 74% "no tangible value" | MIT NANDA (2025); BCG (2024) | | 3 | **Integration & the production gap** | 46% of PoCs scrapped pre-production; ≥30% abandoned after PoC | S&P Global (2025); Gartner (2024) | | 4 | **Ownership & change management** | ~70% of AI challenges are people/process; workflow redesign most correlated with EBIT impact | BCG (2024); McKinsey (2025) | | 5 | **Governance / risk / compliance** | 44% of orgs hit ≥1 negative GenAI consequence (inaccuracy #1, then cybersecurity, explainability) | McKinsey (2024) | | 6 | **Use-case / scope selection** | RAND's #1 root cause: the problem is misunderstood/miscommunicated or the wrong metric is optimized | RAND (2024) | | 7 | **Talent / skills** | >90% of enterprises projected to face critical skills shortages by 2026 (~$5.5T potential loss) | IDC, via Workera (2024) | | 8 | **Cost / infrastructure** | Cost and "escalating costs" among top abandonment drivers; underinvestment in deployment infrastructure | S&P Global (2025); Gartner (2024); RAND (2024) | ### Is bad data really the top cause? Yes — it is the most-named technical root cause across otherwise unrelated studies. Gartner found organizations will abandon **60% of AI projects not supported by AI-ready data through 2026**, and that **63% of organizations either lack or are unsure whether they have the right data-management practices for AI** (Gartner, 2025). RAND independently names insufficient or poor data among its top root causes (RAND, 2024). Pilots run on curated demo data; production runs on the organization's real, fragmented, often ungoverned data — and that is where projects quietly die. ### Why do pilots that "worked" still get killed? Because they were never instrumented to prove value. The headline failures are, at root, **measurement failures**: MIT's 95% "no measurable P&L impact" and BCG's **74% with no tangible value** (BCG, 2024) describe pilots that could not demonstrate a business outcome, not pilots that crashed. RAND's leading root cause is upstream of this — a problem that was **misunderstood or miscommunicated**, so the system optimized the wrong metric (RAND, 2024). Without a pre-agreed, business-tied success criterion, a pilot has no definition of "pass." ### Is this a people problem or a technology problem? Predominantly people and process. BCG's **70/20/10** resourcing pattern shows only ~10% of the difficulty is the algorithm (BCG, 2024). McKinsey finds workflow redesign is among the strongest predictors of impact, yet only ~28% of organizations report CEO-level ownership of AI governance (McKinsey, 2025). Pilots stall when no one owns the operating-model change around the model. ### Does governance really stop projects? It does, especially in regulated and multi-tenant settings. McKinsey found **44% of organizations have already experienced at least one negative consequence from GenAI, with inaccuracy the most common, followed by cybersecurity and explainability** (McKinsey, 2024). Gartner lists "inadequate risk controls" among its four PoC-abandonment drivers (Gartner, 2024). Unmanaged accuracy, security, IP and access-control risk is a direct kill-switch. ## The production gap: where the last mile goes wrong The dominant way pilots "fail" is that the system never survives the last mile — and the forces that kill it in deployment are specifically about **data and verification**, not model capability. Four mechanisms recur: **1. Accuracy decays in production (drift).** A one-time pilot benchmark is a snapshot, not a guarantee. A *Scientific Reports* (Nature) study observed temporal ("aging") model-quality degradation in **91% of tested model–dataset pairs** across four industries (Vela et al., 2022). As production data drifts from training data, a passing pilot score erodes without continuous monitoring and re-validation. **2. Accuracy does not survive contact with production — even with RAG.** Stanford RegLab tested leading legal research tools explicitly marketed as reducing or eliminating hallucination and still found incorrect information on **17% (Lexis+ AI, Ask Practical Law AI) to 34% (Westlaw AI-Assisted Research)** of queries (Magesh et al., Stanford HAI, 2024). Retrieval augmentation reduces hallucination; it does not remove the need for an independent grounding/verification layer. **3. Access control and multi-tenant isolation are a distinct, under-counted failure surface.** Red Hat AI researchers formalized the "relevance–authorization gap": retrieval ranks documents by *semantic relevance*, not by *who is allowed to see them*. Without authorization gating, enterprise retrieval leaked cross-tenant data on **80.0% (client-side) and 62.2% (server-side)** of adversarial probes (Arceo & Narsing, Red Hat AI, 2026). Isolation has to be enforced *inside* the search, not by filtering results afterward — a failure class most "it works in the demo" pilots never test for. **4. Internal evaluation does not predict production reality — the verification gap.** In a 2026 enterprise survey, **50% of organizations shipped a feature that passed their own internal evals and then failed in front of customers** (24% more than once), and only **5% fully trust automated evaluation** (VentureBeat Research, 2026; n=157, directional). At industry level this is the builder-is-not-the-verifier principle: self-grading misses production failures. The data-readiness picture under all four is consistent: Cloudera found **~80% of enterprises say AI is held back by limited data access across environments, and only 18% say their data is fully governed** (Cloudera, 2026), while Precisely/Drexel found **64% of data leaders name data quality their top integrity challenge and 77% rate their own data "average or worse"** (Precisely & Drexel LeBow, 2026). **The industry-level pattern — stated generally.** The cluster that separates the surviving minority from the failing majority is **data integrity + governed access/isolation + independent verification** — precisely the RAG and multi-tenant seam where last-mile failures concentrate. This is not a proprietary insight; it falls straight out of the primary sources above. ## What the pilots that ship do differently The minority that reaches production is small — BCG found only **26% of companies have built the capabilities to move beyond proof-of-concept**, and just 4% are scaled "AI future-built" leaders (BCG, 2024). But what they do differently is consistent and learnable. A general, actionable de-risking framework: 1. **Pick a narrow, high-value, measurable use case.** Leaders deliberately do less — BCG found they pursue roughly **half as many** AI opportunities as less-advanced peers, concentrating on a few and scaling them. Start where value is measurable; aim at the core (leaders ultimately draw **62% of AI value from core processes**, not just back office) (BCG, 2024). 2. **Redesign the workflow around the model — don't bolt it on.** McKinsey finds high performers are nearly **3x** more likely to fundamentally redesign workflows, and that workflow redesign is among the strongest predictors of impact (McKinsey, 2025). MIT attributes most failures to brittle workflows that don't fit daily operations. 3. **Get the data ready: integrity, access control, governance.** With ~80% of enterprises blocked by data access and only 18% fully governed (Cloudera, 2026), reliable, access-controlled, well-governed data is the scarce production input — not a better model. 4. **Name an accountable senior owner and define the success metric up front.** McKinsey finds senior-leadership ownership is the single strongest predictor of success, and high performers are ~2x more likely to have defined ways to measure AI's impact (McKinsey, 2025). Agree the metric *before* the pilot, not after. 5. **Buy or partner where the capability is missing — and benchmark on outcomes.** MIT found externally sourced tools reached deployment at roughly **twice** the rate of internal builds (about two-thirds vs one-third), with the best buyers judging vendors on operational outcomes rather than model-leaderboard scores (MIT NANDA, 2025). 6. **Verify independently before you scale.** Because systems that demo well routinely break in production (and internal evals mispredict production behavior), an independent check — separate from the team that built it, run on real data — is the cheapest insurance against scaling a brittle pilot. The payoff for crossing the gap is real: BCG reports AI leaders posted roughly **50% higher revenue growth, 60% higher total shareholder return, and 40% higher return on invested capital** over three years than less-mature peers (BCG, 2024). ## Myth-check: "Gartner says 85% of AI projects fail" — no, it doesn't The most-repeated statistic in this field is a misquote. Gartner's actual 2018 statement was that **"through 2022, 85% of AI projects will deliver erroneous outcomes due to bias in data, algorithms or the teams responsible"** — a claim about *biased or erroneous outputs*, not about projects failing or being abandoned. If you see "85% of AI projects fail (Gartner)," it is not supported by the source. Use the figures in the table above instead, each for what it measures. ## The through-line Every credible source points at the same gating constraint: AI reaches production when an organization can trust the system's **data and its outputs** — through data integrity, controlled access and isolation, a real success metric, and independent verification — far more than when it adopts a better model. That is the discipline that moves a pilot across the chasm. It is also, candidly, the discipline NewGenApps was built around — *production AI, proven.* ## Methodology & sources This page synthesizes publicly reported findings from primary research organizations and peer-reviewed/preprint studies, dated 2022–2026. Where a primary source is paywalled or blocks automated access (Gartner, S&P Global, McKinsey), figures were verified against the primary report landing page or PDF where available, and corroborated by reputable outlets (Fortune, CIO Dive) cited as secondary confirmation. Statistics are labeled as **measurement**, **forecast**, or **estimate/synthesis**, and vendor/media surveys (S&P Global, Precisely, VentureBeat, Cloudera) are treated as directional. Two frequently cited figures — a Bain "~44% cite lack of in-house AI expertise" number and an Informatica "43% cite data readiness as the top obstacle" number — could not be verified to a primary source at publication and are therefore **omitted** from the figures above rather than presented as fact. **Primary and load-bearing sources** - MIT NANDA (Project NANDA / MIT Media Lab), *The GenAI Divide: State of AI in Business 2025* — https://fortune.com/2025/08/18/mit-report-95-percent-generative-ai-pilots-at-companies-failing-cfo/ (report PDF: https://www.artificialintelligence-news.com/wp-content/uploads/2025/08/ai\_report\_2025.pdf) - Gartner, *Predicts 30% of Generative AI Projects Will Be Abandoned After Proof of Concept by End of 2025* (29 Jul 2024) — https://www.gartner.com/en/newsroom/press-releases/2024-07-29-gartner-predicts-30-percent-of-generative-ai-projects-will-be-abandoned-after-proof-of-concept-by-end-of-2025 - Gartner, *Lack of AI-Ready Data Puts AI Projects at Risk* (26 Feb 2025) — https://www.gartner.com/en/newsroom/press-releases/2025-02-26-lack-of-ai-ready-data-puts-ai-projects-at-risk - RAND Corporation, *The Root Causes of Failure for Artificial Intelligence Projects and How They Can Succeed* (RRA2680-1, 2024) — https://www.rand.org/pubs/research\_reports/RRA2680-1.html - S&P Global Market Intelligence, 2025 AI survey (reported by CIO Dive) — https://www.ciodive.com/news/AI-project-fail-data-SPGlobal/742590/ - IDC (Meredith Whalen), reported via CIO.com (2024) — https://www.cio.com/article/3627447/idc-chief-research-officer-genai-from-experimentation-to-adoption.html - McKinsey & Company / QuantumBlack, *The State of AI in 2025* (5 Nov 2025) — https://www.mckinsey.com/capabilities/operations/our-insights/the-state-of-ai - McKinsey & Company, *The State of AI in Early 2024* — https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai-2024 - BCG, *Where's the Value in AI?* (Oct 2024) — https://www.bcg.com/publications/2024/wheres-value-in-ai - Vela et al., *Temporal quality degradation in AI models*, Scientific Reports (Nature), 12:11654 (2022) — https://www.nature.com/articles/s41598-022-15245-z - Magesh, Surani, Dahl, Suzgun, Manning & Ho, *Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools*, Stanford RegLab / HAI (2024) — https://hai.stanford.edu/news/ai-trial-legal-models-hallucinate-1-out-6-or-more-benchmarking-queries - Arceo & Narsing (Red Hat AI), *Securing the Agent: Vendor-Neutral, Multitenant Enterprise Retrieval and Tool Use*, arXiv (2026) — https://arxiv.org/html/2605.05287v1 - Precisely & Drexel University LeBow College, *State of Data Integrity and AI Readiness* (2026) — https://www.precisely.com/press-release/new-global-research-points-to-lack-of-data-quality-and-governance-as-major-obstacles-to-ai-readiness - Cloudera, *The Data Readiness Index* (2026) — https://www.cloudera.com/about/news-and-blogs/press-releases/2026-04-14-nearly-80-percent-of-enterprises-say-ai-is-held-back-by-data-access-challenges-cloudera-report-finds.html - VentureBeat Research, *The Agent Evaluation Gap* (2026; n=157, directional) — https://venturebeat.com/resources/the-agent-evaluation-gap-enterprise-ai-organizations-have-a-reality-alignment-problem-not-a-coverage-problem-and-most-are-shipping-to-production-anyway - IDC Analyst Brief, *Closing the Gap: Verifying AI Skills in the Enterprise* (via Workera, 2024) — https://www.workera.ai/guides-reports/the-5-5-trillion-skills-gap-what-idcs-new-report-reveals-about-ai-workforce-readiness *This taxonomy is the map; crossing the gap is the work. See [AI Rescue](/ai-rescue/) — a ~90-day path that takes a stalled pilot to verified production — or [why AI pilots fail](/why-ai-pilots-fail/) in plain terms.* ============================================================================== # What Does an AI Production Build Include? | NewGenApps URL: https://newgenapps.com/what-does-an-ai-production-build-include/ ============================================================================== # What Does an AI Production Build Include? Scope, Phases and Gates *Last updated: October 2026. External figures are attributed and dated in the text.* **In short:** An AI production build takes one validated use case and turns it into a system that runs on live data, is monitored on outcomes, is verified by someone other than its builders, and can be operated by your team. It covers design, data integration, the build, an evaluation harness, guardrails, observability, verified deployment, evidence-gated rollout and handover. We recommend turning each risk in the OWASP Top 10 for LLM Applications 2026 into a requirement with an acceptance test, agreed at design sign-off. NewGenApps' AI Production Build starts from $40,000, phased and milestone-gated. ## What is an AI production build? An AI production build is the stage after validation: making one proven approach run reliably on live data, inside your systems, with an owner and a way to show it still works. It is not a larger demo. Our statement of work assumes an earlier stage, such as a [proof of concept](/ai-proof-of-concept/), an opportunity audit or an experiment, has already shown the use case is worth building. The [production readiness checklist](/ai-production-readiness/) is the pre-launch go or no-go; this page covers the build. ## What does an AI production build include? It includes eight workstreams, from solution design and data integration to deployment, verification and handover. Each ends in a deliverable you can inspect, not a status report. ## The eight workstreams and their deliverables | Workstream | What it produces | |---|---| | Solution design | Architecture, data flows, model and agent topology, failure modes, decision log | | Data integration | Per-source freshness guarantees and defined stale or unavailable behavior, under the [data-integrity contract](/ai-data-integrity-contract/) | | Core build | The validated use case, implemented in your stack | | Evaluation harness | A reference set, regression checks and guardrail tests, built from day one, blocking promotion on a missed threshold ([how a harness works](/ai-evaluation-harness/)) | | Guardrails and safety | Input and output validation, policy controls, escalation and fallback paths, human review where required | | Observability | Tracing, outcome alerts, freshness alarms, cost and latency telemetry, an operator view | | Deploy and verify | A production deployment and an acceptance record ([deploy and verify AI](/deploy-and-verify-ai/)) | | Stabilize and hand over | Stabilization window, runbooks, ways of working, supervised operator dry-run | ## What are the phases of an AI production build, and what gates them? There are six phases, each with an exit gate that needs evidence. Gates can collapse only explicitly. 1. **Mobilize.** Access, environments, targets, reference set. Gate: access verified; targets and reference set signed off. 2. **Design.** Solution design, data-integrity contract specification, risk register. Gate: signed off by your accountable architect or sponsor. 3. **Build.** Core build, harness, guardrails, observability. Gate: every must-have requirement passes, the harness gates promotion, evidence is presented. 4. **Deploy and verify.** Promotion to production, signed off by a verifier independent of the builder. Gate: the acceptance record is signed and the pilot criteria are met. 5. **Operate.** Run under observation as rollout advances from pilot to limited to full. Gate: stability, cost envelope and outcome criteria hold at each stage. 6. **Hand over.** Runbooks, ways of working and knowledge transfer. Gate: your named operators complete the runbook procedures unaided. The rollout stages resemble canarying, which Google's *Site Reliability Workbook* defines as a partial, time-limited deployment of a change together with its evaluation, used to decide whether to proceed (Google, 2018). ## How does the OWASP Top 10 for LLM Applications turn into build requirements? We treat it as a source of acceptance tests, not a product checklist. The OWASP Gen AI Security Project released Version 2026 of its Top 10 for LLM Applications in August 2026, updating Version 2025 (November 2024). It names ten risk classes. We recommend turning each into a requirement with an acceptance test, agreed at design sign-off. Two OWASP statements shape the tests. It says no reliable prevention mechanism for prompt injection exists today, so defense is architectural rather than interceptive: tests measure what a successful injection can do, not only whether one gets through. It also says hidden context, such as a system prompt, should not be considered a secret or relied on solely as a security boundary. The controls that matter sit outside the model. For a model that acts with tools and memory, OWASP points to its separate Top 10 for Agentic Applications. ## OWASP Top 10 for LLM Applications 2026, mapped to build requirements The mapping is our reading and our recommendation, not a fixed deliverable list. Which requirements apply, and their tests, are agreed per use case at design sign-off. | OWASP risk (2026) | Build requirement | What must hold before sign-off | |---|---|---| | LLM01:2026 Prompt Injection | Guardrails on every model input and output, plus the LLM03 limits, so a successful injection can do little | Out-of-policy and hostile inputs in the harness are refused or escalated, and none gets an action past the permission limits. Passing is evidence, not immunity | | LLM02:2026 Sensitive Information Disclosure | Data classification, residency, retention and access rules, including whether the provider may train on your data | Sampled prompts, outputs and logs hold nothing outside the classification | | LLM03:2026 Excessive Agency | Least privilege: a short tool list, minimum permissions, actions checked in code outside the model, human approval for high-impact actions | A call outside the list or the permissions is refused by code outside the model, whatever the model decided | | LLM04:2026 Supply Chain | Every model, library and tool is approved and licensed by you, and a model change re-runs the harness | Production holds nothing off the approved list, and a swapped model passes the harness first | | LLM05:2026 Data and Model Poisoning | Retrieval, embedding and fine-tuning use approved sources only, each with an accountable owner, a declared origin and write access limited to that owner | An unlisted source is refused, and a person without write rights cannot change a source that feeds retrieval or fine-tuning. This bounds who can introduce data; it does not detect poisoned content inside an approved source | | LLM06:2026 Unbounded Consumption | A cost envelope per unit of work and per period; limits on input size, request rate, runtime and agent steps; an alert on breach, and a hard cap where a stop is acceptable | Overload, oversize-input and runaway-loop tests stay within the envelope, raise the alert and degrade or stop as specified | | LLM07:2026 Misinformation | A reference set and agreed correctness bar in the harness, human review for high-risk results, and a plain statement of the system's limits to users | The agreed correctness bar is met on the reference set before promotion, high-risk results go to human review, and users are told the system's limits | | LLM08:2026 Hidden Context Exposure | No credentials or access rules in hidden context (system prompt, injected policy text, tool schemas), and authorization enforced outside the model | All hidden context could be published without exposing a credential or weakening an access rule | | LLM09:2026 Vector and Embedding Weaknesses | Where retrieval is used, source-system access rights are enforced inside the query and tenants are kept apart ([RAG in production](/rag-in-production/)) | A user without access to a document cannot retrieve its content through a query | | LLM10:2026 Improper Output Handling | Model output is treated as untrusted and validated before it reaches a database, interface or other system | A deliberately malformed output is rejected by the receiving component | The 2026 edition moves Unbounded Consumption up four places, to sixth, and says spending caps should halt work rather than only alert. The entry is framed around attackers; we read the cost envelope as covering abuse and ordinary runaway usage, such as an agent that keeps calling a model. ## What non-functional requirements does a production build have to meet? In our view, four matter most because a prototype can defer them: auditability, cost guardrails, outcome monitoring and a defined failure behavior for every dependency. Each requirement comes from our statement of work; the acceptance tests show the form we recommend and are agreed at design sign-off. Numeric targets are agreed at design sign-off. Retries suit only operations that can be repeated without repeating their effect, so we recommend setting failure behavior per dependency and, where retries are involved, per operation. ## Four non-functionals as requirements and acceptance tests | Non-functional | Requirement | Acceptance test | |---|---|---| | Auditability | Each transaction is logged well enough to reconstruct what happened, what data was used and what was returned, for the agreed retention period | A person other than the builder picks a production transaction and reconstructs those three things from the records alone | | Cost guardrails | An agreed cost envelope per unit of work and per period, with cost telemetry and an alert on budget breach | Cost shows on the operator view, and the envelope holds under real load at the limited-rollout gate | | Outcome monitoring | Alerts fire on whether the work completed and the right result was delivered, not on whether a process is up | An injected outcome failure raises an alert, a liveness-only blip does not pass as success, and an injected stale input raises a freshness alarm | | Failure behavior per dependency | Every dependency (model endpoint, source system, integration) has a specified behavior on failure: retry, fall back, degrade, or fail closed with a notice. Never a confident wrong answer | Each dependency is made to fail in turn, and the specified behavior is what you observe, including an explicit unavailable state | ## Who verifies the build, and what counts as done? Someone other than the builder verifies it against the running system. Done means evidence from production, not a merged pull request or a demo. In the statement of work, the independent verifier, not the builder, is accountable for the acceptance record. It holds the four-layer proof (source, environment, running process, real outcome) with timestamps, plus your sign-off. See [deploy and verify AI](/deploy-and-verify-ai/) and [liveness versus outcome](/production-ai-liveness-vs-outcome/). ## What is not included in an AI production build? These are out of scope unless added through change control: - Further use cases and re-validating the use case. - Training or fine-tuning from scratch. Bespoke training enters only as a change, if evaluation proves it necessary. - Fixing data quality in your source systems. We flag problems; the fix is yours unless scoped separately. - New platform infrastructure, such as a cloud landing zone. - Organization-wide change management or large-scale end-user training. - Running the system beyond the stabilization window, offered under [Your AI Team](/engagements/#your-ai-team). - Regulatory certification or legal sign-off. We build to the compliance requirements you supply and produce the evidence. ## When is a production build the wrong next step? - **The use case is not validated.** Run a [proof of concept](/ai-proof-of-concept/) first. - **A pilot has stalled.** Triage first: [how to rescue a failing AI project](/how-to-rescue-a-failing-ai-project/) and [AI Rescue](/ai-rescue/). - **Several use cases compete.** [Prioritize them](/ai-use-case-prioritization/) first. - **The task is stable, rule-based and fully specified.** In our judgment, conventional automation is cheaper to build and easier to audit, and AI is the wrong tool. - **No one can grant access to real data or own the outcome.** The data-integrity criteria cannot be proven without real data. ## How long does an AI production build take, and what does it cost? The AI Production Build starts from $40,000 and is phased, with milestone gates (NewGenApps engagements page, 2026-09-29). We publish no duration; the calendar is agreed at scoping. The figure is a floor, not a quote; the fee is set at scoping, after a free 30-minute call. AI Compass and AI Rescue fees are credited toward a subsequent build. Model usage, third-party licenses and cloud infrastructure are separate, billed at cost or paid directly by you. [How much AI consulting costs](/how-much-does-ai-consulting-cost/) explains how to read any quote. ## Related reading and sources *To see how this applies to your use case, read the [AI Production Build terms](/engagements/#ai-production-build) or [book a 30-minute working session](/book/). [Which AI engagement do you need](/which-ai-engagement-do-you-need/) places you in three questions. Related reading: [what an AI consulting firm does](/what-does-an-ai-consulting-firm-do/), the [data-integrity contract](/ai-data-integrity-contract/), [measuring AI ROI](/measure-ai-roi/) and [how we work](/how-we-work/).* *Sources: OWASP Gen AI Security Project, ["OWASP Top 10 for LLM Applications 2026"](https://genai.owasp.org/resource/owasp-genai-llm-top-10-2026/) (Version 2026; resource page dated 2026-08-03, [PDF](https://genai.owasp.org/download/56857/) cover dated August 4th, 2026); OWASP Top 10 for LLM Applications project (now the Gen AI Security Project), ["OWASP Top 10 for LLM Applications 2025"](https://owasp.org/www-project-top-10-for-large-language-model-applications/assets/PDF/OWASP-Top-10-for-LLMs-v2025.pdf) (Version 2025, 2024-11-18); A. Warner and Š. Davidovič, ["Canarying Releases"](https://sre.google/workbook/canarying-releases/), chapter 16 of *The Site Reliability Workbook* (Google and O'Reilly, 2018); NewGenApps, [engagements page](/engagements/#ai-production-build) (2026-09-29) and AI Production Build scope, phases and acceptance criteria (checked 2026-10-13).* ============================================================================== # Which AI Engagement Do You Need? A Decision Guide | NewGenApps URL: https://newgenapps.com/which-ai-engagement-do-you-need/ ============================================================================== # Which AI Engagement Do You Need? A Decision Guide *Last updated: October 2026. External figures are attributed and dated in the text. This guide uses none; prices and durations are as listed on the [engagements page](/engagements/).* **In short:** Start at the rung that matches where you are. NewGenApps publishes six AI engagements, and three questions place you on them: is anything already built, can you state one use case in a sentence, and do you want a result or a standing team? No clear use case points to AI in a Day (from $2,500) or AI Compass (from $5,000). One use case points to POC in 3 Weeks (from $15,000). A stalled pilot points to AI Rescue (from $25,000), a validated use case to AI Production Build (from $40,000), and continuing work to Your AI Team (from $10,000 per month). The first step, a 30-minute call, is free. ## Where should a company start with AI? Start at the stage you have actually reached, not the one that sounds most ambitious. Each engagement produces one specific output: a ranked shortlist, a costed roadmap, a verdict on one use case, a recovered or retired pilot, a running system, or a standing team. Each output is the input to the next, so the useful question is which you already hold and which you are missing. Two mistakes follow from not asking that question: paying for an output you already hold, and funding a rung before its input exists, such as a production build with no written definition of what "working" means on your data. This is our judgement, not a measured result. Skipping rungs is legitimate when you hold the input: the published fit for the AI Production Build is a team whose use case is already validated (engagements page). If you are unsure what you hold, the [free 30-minute call](/book/) is the cheapest way to find out. ## What types of AI consulting engagement are there? Six, grouped by purpose. Two help you decide: AI in a Day, and AI Compass, the opportunity audit. One proves a single use case: POC in 3 Weeks. One recovers stalled work: AI Rescue. One puts a validated use case into production: AI Production Build. The sixth, Your AI Team, is a standing capability rather than a project. Five are scoped stages; the sixth is month to month after an initial term of typically three months. Price drivers are in [how much AI consulting costs](/how-much-does-ai-consulting-cost/). ## Decision table: situation, engagement and starting price | Your situation | Engagement | What you get | Starting price | |---|---|---|---| | Unsure where AI would pay off | Free 30-minute call ([book](/book/)) | A view of where AI pays off and where it does not | Free | | Several ideas compete, or a board wants a plan | [AI in a Day](/engagements/#ai-in-a-day) | Ranked shortlist (value, effort, risk); a "where to start" page; a "not yet / not this" list | from $2,500 per one-day session | | A sponsor needs a costed roadmap before funding | [AI Compass](/engagements/#ai-compass) | Costed roadmap; red, amber or green data-readiness scorecard; recommended first proof; executive readout | from $5,000, credited toward a subsequent build; typically 2 to 6 weeks | | One defined use case, and a go or no-go needed on your data | [POC in 3 Weeks](/engagements/#poc-in-3-weeks) | Working proof on real data; independent evaluation against a bar agreed in Week 1; build, refine or do-not-build call | from $15,000; 3 weeks | | A pilot that demoed well and then stalled | [AI Rescue](/engagements/#ai-rescue) | First-week triage ending in a recover, re-scope or retire verdict; remediation and handover if recoverable | from $25,000, credited toward a subsequent production build; about 90 days | | A validated use case that must run in production | [AI Production Build](/engagements/#ai-production-build) | Production build in your stack; outcome monitoring; independently verified deployment; runbooks | from $40,000, phased | | Continuing need for senior AI capability, without hiring a team | [Your AI Team](/engagements/#your-ai-team) | Small senior pod; rolling backlog; monthly strategic review; no-lock-in exit at no extra charge | from $10,000 per month for the smallest pod | Source: NewGenApps engagements page. Prices are in US dollars and are starting points. ## How do three questions place you on the ladder? Answer them in order. The first two place you on a rung; the third decides whether a standing team follows it. 1. **Is something already built?** If a pilot demoed well and then stalled, start with [AI Rescue](/engagements/#ai-rescue), whose first week ends in a recover, re-scope or retire verdict. If a validated use case now has to run in production, start with the [AI Production Build](/engagements/#ai-production-build). If nothing is built, go to question 2. 2. **Can you state one use case in a sentence, with a named owner, a decision it would change and data you can reach?** If so, the open question is whether it works on your data at a cost and speed you can accept: that is the [POC in 3 Weeks](/engagements/#poc-in-3-weeks). If not, choose by who must be convinced. [AI in a Day](/engagements/#ai-in-a-day) gives a ranked starting point in one working day; [AI Compass](/engagements/#ai-compass) gives a sponsor a costed roadmap and a data-readiness verdict to fund against. If you cannot name any use case, start with the free call. 3. **Do you want a result or a standing capability?** If the need is one outcome, the rung from question 1 or 2 is your answer. If the work will keep coming, and you do not want to hire and manage a team, [Your AI Team](/engagements/#your-ai-team) can follow any of them as a continuing capability (see [what an AI retainer includes](/embedded-ai-team/) and [build versus buy for an AI team](/build-vs-buy-ai-team/)). These questions are a heuristic built from the published fit of each engagement, not a scored instrument; a real situation can sit between two rungs. Deeper guides: [how to prioritize AI use cases](/ai-use-case-prioritization/), [what an AI proof of concept should prove](/ai-proof-of-concept/), [how to rescue a failing AI project](/how-to-rescue-a-failing-ai-project/) and the [5-signal pilot diagnosis](/has-your-ai-pilot-stalled-5-signal-diagnosis/). ## How do the six engagements fit together? As a ladder in which each rung's output feeds the next. AI in a Day produces a ranked shortlist and a first move. AI Compass turns it into a costed roadmap with a recommended first proof. The POC in 3 Weeks tests that proof on your real data and ends in a build, refine or do-not-build recommendation. The AI Production Build (see [what it includes](/what-does-an-ai-production-build-include/)) runs the validated case in production, and Your AI Team operates and extends it. [AI Rescue](/ai-rescue/) enters mid-ladder, for work that already exists. You can stop at any rung. The POC counts a reasoned "do not build" as a successful outcome, AI in a Day lists what to defer, and AI Rescue keeps a real off-ramp after week one (engagements page). The AI Compass and AI Rescue fees are credited toward a subsequent build, on the terms in the engagement letter. ## What if your situation fits two rungs? Four checks settle most close calls: a budget to approve, a named use case, a written bar already cleared on real data, and a stalled pilot. The table pairs each with the rungs it separates. Regulated work cuts across all of them: if the output will feed a regulated decision, say so in the first conversation, because access controls, audit evidence and review steps add work wherever it does (NewGenApps, [how much AI consulting costs](/how-much-does-ai-consulting-cost/), September 2026). ## Close calls and tie-breakers | If you hesitate between | The tie-breaker | |---|---| | AI in a Day and AI Compass | Must someone approve a budget on the output? If so, take the costed roadmap. | | AI Compass and POC in 3 Weeks | Can you already name the use case? A compass chooses between candidates; a proof tests one. | | POC in 3 Weeks and AI Production Build | Has the use case cleared a bar agreed in writing, on real data? If not, prove first. A passing proof means worth building, not ready to run. | | POC in 3 Weeks and AI Rescue | Is there an existing pilot that stalled? Rescue starts from what exists; a proof starts from a question. | | AI Production Build and Your AI Team | Is it one scoped system, or continuing demand? A build is phased and milestone-gated; a team is month to month. | ## When should you not hire an AI consultant at all? When the problem is not yet a decision someone owns, when ordinary software would solve it, or when the data it needs does not exist. Four cases, in our judgement: - **No one owns the decision.** If no named person can say what would change if the AI worked, and agree what "working" means, a proof has nothing to be graded against. The POC's pass bar is agreed with you in Week 1, so someone must be able to agree it. - **A fixed rule would do.** If the right answer is a rule over structured inputs that rarely changes, ordinary software is usually cheaper to build, run and audit than a model. AI earns its place where inputs are unstructured or the rules cannot be written down. - **The data does not exist, or you cannot reach it.** If you already know the data is not captured or is blocked, the first spend belongs in the data. AI Compass rates data readiness per system, but do not buy a rating for an answer you have. - **A product you can buy already does the job.** Buy it. The booking page describes the first call as mapping where AI pays off in your business and where it does not (checked 2026-10-13), so hearing "not here" is within its scope. If you cannot yet measure success, fix that first; [measuring AI ROI in production](/measure-ai-roi/) shows what to instrument. "Not yet" is a legitimate answer at every rung. ## What does the free 30-minute call cover? A 30-minute working session on Google Meet, with no deck and no pitch, run by a senior engineer rather than a sales representative (booking page, checked 2026-10-13). You describe the problem; a sentence is enough to start. The session maps where AI would pay off in your business and where it would not, and scopes what a first proof of concept would involve. You leave with a view of what that proof would show, roughly what it would take and whether it is worth doing. It costs nothing (engagements page) and carries no obligation (booking page). If you would rather write first, use the [contact form](/contact/). ## How much does it cost to start? The first step costs nothing, and the first paid rung starts at $2,500, for AI in a Day. Prices are in US dollars, shown as starting points and set against a scoped engagement after the free call, so you see the deliverable and the price before you commit (engagements page). On the AI Production Build and Your AI Team, model usage, third-party licenses and cloud infrastructure are separate: billed at cost, or paid directly by you. Questions worth asking any firm are in [how to choose an AI partner](/how-to-choose-an-ai-partner/). ## What terms run across the engagements? Three published terms run across the ladder, each where it applies. First, independent verification, the basis of production AI, proven: every engagement that builds something ends with independent verification on the running system, and the person who builds it does not sign it off (the principle is in the [data-integrity contract](/ai-data-integrity-contract/)). Second, scope and price before you commit, for all six. Third, a way out where one is published: a POC can end in "do not build", AI Rescue keeps an off-ramp after week one, and Your AI Team has a structured, no-lock-in exit at no extra charge. The method is on [How we work](/how-we-work/); past work is described by sector, never by client, on the [Work](/work/) page. ## Related reading and sources *[Book a 30-minute working session](/book/). Read next by rung: [prioritizing AI use cases](/ai-use-case-prioritization/), [what a proof of concept should prove](/ai-proof-of-concept/), [rescuing a failing AI project](/how-to-rescue-a-failing-ai-project/), [what a production build includes](/what-does-an-ai-production-build-include/) and [the embedded AI team](/embedded-ai-team/). For context, see [AI consulting](/ai-consulting/) and [AI strategy](/ai-strategy/).* *Sources: NewGenApps, [engagements page](/engagements/); NewGenApps, [Book an AI working session](/book/) (read 2026-10-13); NewGenApps, [how much does AI consulting cost](/how-much-does-ai-consulting-cost/) (September 2026; the regulated-context sentence). No external statistics are cited.*