AI
Which AI projects are worth funding: the three conditions
What to check before you fund an applied AI project, and what to build before the model.
Computing America. Applications, integrations, data analysis and AI/ML engineering.
Name the decision
In short
- Three conditions are worth checking before you start. The second, historical examples to evaluate against, is the one you can build yourself in a day.
- A vendor's benchmark measures their examples. An evaluation set built from your own messy cases measures yours, and it's the figure that matters.
- Aim for calibrated confidence, not maximum coverage: set the review threshold at your tolerance for error.
- Any AI feature on a critical path needs a degraded mode decided at design time: a second provider, a simpler model, or an honest queue.
- If the task is rule-shaped, use rules.
Fund an applied AI project only if it passes three checks, and the one most teams skip is an evaluation set built from your own cases.
Output that is almost right is the hazard to plan for. In Stack Overflow’s 2025 developer survey (opens in a new tab) about AI coding tools, the top frustration, at 66%, was output that is almost right but not quite, and 45% chose "debugging AI-generated code is more time-consuming."
The three conditions¶
- A repetitive, expensive, judgment-light task. Someone does the same structured thing hundreds of times a week. If every instance needs expert judgment, you're automating the wrong step.
- Enough historical examples to evaluate against (a few hundred labeled cases is a workable start). You may not need them to train on, but without an evaluation set you can't tell improvement from confident nonsense.
- A tolerable cost of being wrong, or a place to put a human. Either the error is cheap, or the workflow has a review point where uncertain outputs go to a person.
Build the evaluation set before the model¶
Have your domain experts spend a day labeling real examples. Size the set to the error rate you must prove: 300 cases fix a measured rate to within about 6 points at worst (95% interval), which is enough to compare against a vendor but not to show a low error rate. Use the messy ones: the smudged fax, the handwritten amendment, the form somebody filled in wrong. It defines what correct means, which often surfaces disagreement inside the team. It gives you a number, so “it seems better” becomes a measurement. And you own it: when a vendor's benchmark says 94% and your evaluation says 71%, yours is the figure that matters.
Confidence beats coverage¶
Maximizing the share handled automatically is the wrong objective. Give each output a calibrated confidence score. Run half of your evaluation set through the system and take the lowest threshold at which the accepted outputs meet your tolerance for error. Confirm it on the other half, because a threshold chosen on a set cannot also certify itself. Route everything below it to a person with the uncertain parts highlighted. Zero errors in n accepted cases only bounds the rate near 3/n. At 78% accepted, a 150-case confirmation half yields about 117 accepted cases, which can show an error rate under about 2.6%, not 0.4%. Showing 99.6% takes roughly 750 accepted cases with no errors; if your tolerance is that tight, size the set for it from the start.
errors let through unreviewed, per 1,000 documents
- Handles 78%, 99.6% accurateabout 3
- Handles 95%, 91% accurateabout 86
Of 1,000 documents, the system that automates more lets about 86 errors through unreviewed, because nothing marks its wrong answers; the other lets 3.
SourceWorked example with assumed rates, rounded: 780 x 0.4% and 950 x 9%. Not a measurement.
Design the fallback at design time¶
Model providers have outages, retire versions, change pricing and shift behavior between releases. An AI feature on a critical path needs a degraded mode: a secondary provider, a simpler local model, or a queue that tells users the assist is unavailable. Decide which before launch.
How these projects fail¶
- The chatbot with no defined task. Its scope is “answer questions,” so it can't be evaluated or finished.
- Accuracy as a feeling. Nothing monitors the outputs after launch, and the evidence is a demo that impressed the room in March.
- A rules engine would have won. If the task is rule-shaped, use rules: deterministic logic is cheaper to run, gives the same answer every time and is easier to audit.
Before you commission any AI project, write down the decision, who makes it today, how often, and what a wrong one costs.
Sources
- 1.AI (opens in a new tab), Stack Overflow Developer Survey 2025
Next step
Name the decision you want a model to make.
Who makes it today, how often, and what a wrong one costs. We will say which of the three conditions it meets and which it does not, including when the answer is that it needs no model.
- Phone
- (214) 723-2510
- Reply
- A person replies, not a sequence: within one business day, from someone who would be on the engagement.
Packages
Start with a package
Four packages, each with its price on the page. Pick one and buy it online.
$199/month
Billed monthly
$182.42/month
Billed yearly
$499/month
Billed monthly
$457.42/month
Billed yearly
- Scammers stopped from sending email as you
- Your website and systems scanned every week
- Your email settings checked every week
- Everything in the 1 to 10 plan
- Covers up to 50 people
$199/month
after a one-time setup fee
- A site planned, designed and built for you
- Hosting and updates every month
$499/month
Billed monthly
$457.42/month
Billed yearly
$999/month
Billed monthly
$915.75/month
Billed yearly
- Your Google profile, fully set up
- A Google post every week
- 2 social posts a week
- Everything in Get found
- 4 social posts a week
- Your top 10 Google searches tracked
from$500/month
after a one-time setup fee
- Your data gathered from the tools you use
- A first analysis that answers your questions
- New findings every month
Related reading
Delivery4 min readAI coding tools: the typing was never the expensive part
What the 2025–26 evidence on AI coding tools shows, and the three controls to have in place before your change volume rises.
Data2 min readWhy departments report different revenue numbers, and how to fix it
The meeting should be about the decision, not about whose number is right. Write each metric definition once and make every report read it.
Strategy3 min readBuild vs buy software: sort each process step before you write requirements
Sort every process step as commodity, differentiating or contextual before anyone writes a requirements document.
