When to Kill an AI Project | Decision Framework

Deciding whether to continue an AI project requires comparing its remaining costs, expected benefits, and risks. Past spending can influence that decision even when it cannot be recovered. This guide offers a review framework with example criteria.

Signs It’s Failing

An AI project can remain technically operational while failing to deliver enough value. Look for measured signs in your own project rather than relying on a general AI failure-rate claim. The following signals are prompts for review, not a validated predictive model.

Review these five areas using evidence from the project:

Accuracy below threshold. Compare the current results with the acceptance criteria on a representative test set. Record what was tried, what changed, and what additional work could plausibly close the gap.

Adoption below target. Check whether people use the system for the intended task and whether the output helps them. Low use can reflect accuracy, workflow fit, training, access problems, or a task that occurs less often than expected. Investigate the cause before deciding to stop.

Costs exceeding value. Include software, token usage, operating time, and corrections in the ROI calculation. Compare them with measured or explicitly estimated benefits; the absence of a calculation is a gap in evidence, not proof of a loss.

Scope growing without a revised case. If the project expands from one classifier to a platform with several use cases, reassess the budget and expected value. Extra scope is not itself evidence of failure, but it changes the decision being evaluated.

Team concerns. Ask users, engineers, and sponsors to identify specific failures or unmet needs. Reduced enthusiasm is a reason to investigate, not proof that the project cannot work.

Use these signals alongside the severity of the problem and the remaining cost of addressing it. A serious unresolved safety issue may justify stopping on its own; several minor issues may still be fixable.

The Sunk Cost Trap

“We’ve already spent X” is irrelevant to the decision. Money already spent is gone whether you continue or stop. The only question that matters: from today, is this worth continued investment?

Separate unrecoverable past spending from costs and benefits that still depend on the decision. Existing assets may have reuse value, and stopping may create contractual or transition costs.

A cleaner test: if this project landed on your desk tomorrow as a new proposal. same remaining cost, same remaining chance of success. would you fund it? If no, continuing is not perseverance. It is throwing good money after bad.

This is also why custom versus off-the-shelf should be revisited. Sometimes the kill is “stop the custom build and buy a narrower tool.” That is still a stop.

Kill Criteria to Set Upfront

Agree review criteria before launch. They make it easier to explain a later decision to continue, narrow, pause, or stop.

Write these down before the first production user:

Minimum accuracy. A number, not a feeling. “90% of invoices extracted with all required fields correct” is a criterion. “Pretty good most of the time” is not.

Maximum cost. A monthly ceiling for model spend, tooling, and internal time. When the run-rate crosses it without a matching value story, the project stops or shrinks. no extra meeting required.

Adoption floor. Set a usage measure and review date that match the task frequency. Investigate why it is missed rather than treating low use as an automatic instruction to cancel.

Timeline gate. Proof of concept by week four. Production-shaped pilot by week twelve. Miss the gate and the default is stop. Continuing needs a new written case. not a verbal “we’re close.”

Agree review thresholds with the project owner before launch. The example accuracy targets and week-four/week-twelve gates above illustrate how to make a decision measurable; they are not industry standards or evidence that every project should use those numbers.

The Pivot Option

Kill is not the only honest outcome. Sometimes a smaller version works, or the system is solving a different problem. Ask three questions before you shut it down:

Is there a smaller version that works? A classifier limited to well-tested enquiry types, with other cases sent for review, may be useful. A drafting assistant reduces the scope of automated action, but its outputs and data access still need controls.

Is it solving a different problem? Evaluation may show acceptable results for triage but not generation, or the reverse. If the accidental use is valuable, rename the project and drop the rest.

Is a different approach worth one short test? Rules plus a smaller model. Retrieval instead of generation. Buying instead of building. Approve a time-boxed test only if the expected information justifies its cost. Further attempts need new evidence and a revised case.

Pivot when you can name the new success criteria in a paragraph. Kill when you cannot. Hovering between the two. “let’s keep exploring” with no new gate. is how zombie projects survive.

Hallucination risk can justify narrowing the scope. If people must review each draft, include that review time in the business case. A drafting assistant can still be useful automation, but it should not be described or priced as an unattended decision-maker.

Post-Mortem Value

Capture the findings from a cancelled project while the details are available. Record what was tested and what the evidence showed so a later team can assess whether circumstances have changed.

Write a short post-mortem while the details are fresh:

What you believed at the start, and which of those beliefs were wrong
What the data actually looked like versus what you assumed
Where accuracy stopped improving, and what you tried after that
What the team would refuse to attempt the same way again

Keep prompts, evaluation cases, and integration notes where retention rules and access permissions allow. They can inform the next SaaS versus custom decision. Record the actual cause of the failure rather than assuming it was always product fit.

Tell the organisation what you learned. Silence after a kill teaches people that cancellation is shameful. A write-up teaches them that stopping is part of running the work properly.

If the decision is to stop, name an owner and transition date. Arrange alternative handling, revoke unneeded access, and retain or delete data according to the applicable requirements. For broader risk-management context, see the voluntary NIST AI Risk Management Framework.

Where Fernside Fits

Advisory is the right place for this conversation when you are too close to the project to score it honestly. we review accuracy, cost, and adoption against kill criteria, and tell you whether to stop, shrink, or continue.

Want an objective assessment of a struggling AI project? Book a discovery call and we will score it against the gates above before anyone spends another quarter on it.