Skip to content

AI

Which AI projects are worth funding: the three conditions

What to check before you fund an applied AI project, and what to build before the model.

Computing America. Applications, integrations, data analysis and AI/ML engineering.

Name the decision
3 min readComputing America
A paper target with a tight cluster of hits at the center and a few scattered misses.

In short

  • Three conditions are worth checking before you start. The second, historical examples to evaluate against, is the one you can build yourself in a day.
  • A vendor's benchmark measures their examples. An evaluation set built from your own messy cases measures yours, and it's the figure that matters.
  • Aim for calibrated confidence, not maximum coverage: set the review threshold at your tolerance for error.
  • Any AI feature on a critical path needs a degraded mode decided at design time: a second provider, a simpler model, or an honest queue.
  • If the task is rule-shaped, use rules.

Fund an applied AI project only if it passes three checks, and the one most teams skip is an evaluation set built from your own cases.

Output that is almost right is the hazard to plan for. In Stack Overflow’s 2025 developer survey (opens in a new tab) about AI coding tools, the top frustration, at 66%, was output that is almost right but not quite, and 45% chose "debugging AI-generated code is more time-consuming."

The three conditions

  • A repetitive, expensive, judgment-light task. Someone does the same structured thing hundreds of times a week. If every instance needs expert judgment, you're automating the wrong step.
  • Enough historical examples to evaluate against (a few hundred labeled cases is a workable start). You may not need them to train on, but without an evaluation set you can't tell improvement from confident nonsense.
  • A tolerable cost of being wrong, or a place to put a human. Either the error is cheap, or the workflow has a review point where uncertain outputs go to a person.

Build the evaluation set before the model

Have your domain experts spend a day labeling real examples. Size the set to the error rate you must prove: 300 cases fix a measured rate to within about 6 points at worst (95% interval), which is enough to compare against a vendor but not to show a low error rate. Use the messy ones: the smudged fax, the handwritten amendment, the form somebody filled in wrong. It defines what correct means, which often surfaces disagreement inside the team. It gives you a number, so “it seems better” becomes a measurement. And you own it: when a vendor's benchmark says 94% and your evaluation says 71%, yours is the figure that matters.

Confidence beats coverage

Maximizing the share handled automatically is the wrong objective. Give each output a calibrated confidence score. Run half of your evaluation set through the system and take the lowest threshold at which the accepted outputs meet your tolerance for error. Confirm it on the other half, because a threshold chosen on a set cannot also certify itself. Route everything below it to a person with the uncertain parts highlighted. Zero errors in n accepted cases only bounds the rate near 3/n. At 78% accepted, a 150-case confirmation half yields about 117 accepted cases, which can show an error rate under about 2.6%, not 0.4%. Showing 99.6% takes roughly 750 accepted cases with no errors; if your tolerance is that tight, size the set for it from the start.

Fig. 1

errors let through unreviewed, per 1,000 documents

  • Handles 78%, 99.6% accurateabout 3
  • Handles 95%, 91% accurateabout 86

Of 1,000 documents, the system that automates more lets about 86 errors through unreviewed, because nothing marks its wrong answers; the other lets 3.

SourceWorked example with assumed rates, rounded: 780 x 0.4% and 950 x 9%. Not a measurement.

Design the fallback at design time

Model providers have outages, retire versions, change pricing and shift behavior between releases. An AI feature on a critical path needs a degraded mode: a secondary provider, a simpler local model, or a queue that tells users the assist is unavailable. Decide which before launch.

How these projects fail

  • The chatbot with no defined task. Its scope is “answer questions,” so it can't be evaluated or finished.
  • Accuracy as a feeling. Nothing monitors the outputs after launch, and the evidence is a demo that impressed the room in March.
  • A rules engine would have won. If the task is rule-shaped, use rules: deterministic logic is cheaper to run, gives the same answer every time and is easier to audit.

Before you commission any AI project, write down the decision, who makes it today, how often, and what a wrong one costs.

Sources

  1. 1.AI (opens in a new tab), Stack Overflow Developer Survey 2025

Next step

Name the decision you want a model to make.

Who makes it today, how often, and what a wrong one costs. We will say which of the three conditions it meets and which it does not, including when the answer is that it needs no model.

Reply
A person replies, not a sequence: within one business day, from someone who would be on the engagement.

Packages

Start with a package

Four packages, each with its price on the page. Pick one and buy it online.

Billing