Writing

Most AI pilots die between the demo and production

8 min read Updated 3 Sept 2026

There is a specific silence in companies that tried AI last year. The demo went brilliantly. The pilot kicked off with a name and a channel and a bit of budget. And then, somewhere around month four, everyone stopped mentioning it. Nobody killed it. It just stopped being brought up, the way you stop bringing up a gym membership.

I build AI products inside a venture fund, which means I live in the gap this post is about: the distance between what a demo promises and what a production system has to do. None of what follows is hypothetical. It is the same short list of gaps, in roughly the same order, almost every time.

The demo is not a smaller version of the product

The demo lies — not maliciously, structurally. A demo is a happy path with the lights dimmed. It runs on hand-picked inputs, under the demo-giver’s control, with no authentication, no tenant isolation, no cost meter, no angry customer, and no Tuesday-afternoon data. Large language models are spectacular demo material because they are fluent even when they are wrong — the failure modes don’t look like error screens, they look like confidence.

A production system is the opposite object. It runs on inputs nobody picked, for users nobody briefed, at a cost somebody has to approve, with its mistakes landing on a real person. The demo and the product share a model. They share almost nothing else.

Most pilots die in that “almost nothing else”. Specifically, in six places.

The six gaps

1. Data access, not data science. The pilot ran on an export someone emailed. The product needs live access to systems with owners, permissions, personal data, and audit requirements. In UK companies this is where months go: not the model, the plumbing and the permission to use it. If nobody has answered “are we allowed to put this data through this system?” in writing, the pilot has a legal department in its future, and pilots do not survive surprise meetings with legal.

2. Nobody defined “right”. Ask what the pilot must score to ship and you usually get a pause. Without an evaluation set — even a modest one: a hundred real examples, marked by someone who knows the domain — “it seems pretty good” is the ceiling. Then the first visible mistake arrives, there’s no baseline to defend, and confidence collapses to zero as fast as it inflated. Teams that ship have a number. Teams that stall have vibes.

3. No cost or latency budget. Per-request pricing feels like nothing in a demo and becomes a line item at volume, and “the answer takes nine seconds” is a curiosity in a pilot and a dealbreaker in a checkout flow. If the unit economics were never written down, the CFO discovers them later, in the worst possible format: an invoice.

4. Failure has no UX. The model will be wrong sometimes; that is a property, not a bug. The product question is what a user sees in that moment — can they tell, can they correct it, does a human take over, does the system know its own confidence? Pilots skip this because in a demo nobody is wrong. Products live or die on it, because trust, once spent, does not refill.

5. Nobody is watching it. Traditional software fails loudly; models drift quietly. Prompts get edited, providers update models under you, the input mix shifts, and quality sinks without a single alert firing — unless someone built monitoring, logging, and a way to replay yesterday’s traffic against today’s system. That is unglamorous engineering, which is exactly why it gets skipped in the phase that is meant to feel like magic.

6. The pilot has no owner with a P&L. A champion is not an owner. When the enthusiast who started it changes role, the pilot is orphaned; when a real owner exists, the question changes from “is this cool?” to “does this move a number I am accountable for?” — which is the question that gets things either shipped or honestly killed. Both outcomes beat the silence.

The research, honestly read

MIT’s Project NANDA put a number on the silence in its 2025 report, The GenAI Divide: State of AI in Business (PDF): from a review of more than 300 publicly disclosed AI initiatives, interviews with 52 organisations and survey responses from 153 senior leaders, it concluded that about 95% of organisations were getting zero return from their GenAI spend. Only a sliver of pilots reached production with a measurable P&L impact attached.

Read it with the same honesty you’d want applied to your own pilot: it is a preliminary industry report, not a peer-reviewed trial, and the 95% figure has been argued over since the day it published — Ray Poynter’s reading of the same report is that the headline blurs several different claims together, and that roughly a quarter of the organisations that actually ran a pilot cleared the bar. But the shape of the finding — high adoption, tiny transformation, pilots stalling before production — matches what practitioners see everywhere, and its core explanation is the right one: the failures are rarely about model quality. They are about systems that don’t learn the workflow, and workflows that never made room for the system.

What the surviving pilots do differently

Nothing exotic. That is rather the point.

  • They start from a workflow, not a capability. Not “what can the model do?” but “which step, in which process, costs us the most tedium or error?” The narrower the step, the better the odds — boring targets ship.
  • They buy the undifferentiated parts. If the value isn’t in owning the pipeline, they don’t build the pipeline. Build-versus-buy discipline applies double when the technology is moving this fast.
  • They write the evaluation set before the integration. A hundred marked examples before a single sprint. It converts every future argument into a measurement.
  • They ship the boring wrapper. Permissions, logging, fallback, a human in the loop where the stakes demand one. The wrapper is the product; the model is a component.
  • They give it an owner and a kill criterion. A date, a number, and a named person who benefits if it works and answers if it doesn’t. Pilots with kill criteria die fast or graduate. Pilots without them just fade.

If you already have a stalled pilot

Most companies reading this aren’t at week zero — they’re at month five, with a pilot in the ambiguous state between alive and abandoned. The kind thing to do is decide, and it takes an afternoon, not a committee — though it does take someone whose job is that kind of decision, rather than whoever happens to be nearest the pilot.

Run the six gaps as an autopsy checklist. Does it have live, permitted data access or is it still on the emailed export? Is there a number it was meant to move, and has anyone measured it? Does anyone know what it costs per week to run and per answer to serve? Has a real user hit a real failure, and what did they see? Would anyone notice if its quality halved tomorrow? And is there a named owner whose own results improve if it graduates?

Then pick one of three honest outcomes:

  • Kill it well. If the workflow was wrong or the data can’t be used, stop — and write one page on what was learned, so the next attempt starts from evidence instead of enthusiasm. A cleanly killed pilot is a success of a modest kind; a zombie pilot poisons the appetite for the next one.
  • Re-scope it narrower. Most stalled pilots aimed at a process when they should have aimed at a step. “Automate support” stalls; “draft the reply for the human to approve, for the top three ticket types” ships. Shrink until the evaluation set is writable in a week.
  • Graduate it properly. If the number moved, fund the unglamorous part it never had — the permissions, monitoring, failure UX and ownership from the list above — as a budgeted piece of engineering, not a favour squeezed around the day job. Pilots that “graduate” without that budget just become production incidents with a head start.

The one move that is never right is the current one: leaving it running, unmeasured and unowned, as a standing reason for the next idea to be met with “didn’t we try AI?”

The uncomfortable summary

The demo was never evidence that the product would work. It was evidence that a model can be impressive under controlled conditions — which was already known, and which is free to find out. The expensive discovery, the one worth a pilot, is whether your data, your process, and your tolerance for error can be engineered into something that holds up on a wet Tuesday in production.

That discovery is available before the pilot starts. It mostly requires asking the six questions above in week zero, in writing, of someone with no stake in the answer being yes — for companies without that person in the building, that is most of how I work with companies on AI. The companies in the happy five per cent didn’t get better models than everyone else. They asked earlier.

Drizzlelabs is an independent software studio in Bristol that builds and ships AI-assisted tools under its own name — the about page has the story, and the footer has the proof.