Delivery
How to evaluate a software vendor without reading code
Four requests that show whether a firm builds, tests, tracks its dependencies and learns from failure.
Computing America. Applications, integrations, data analysis and AI/ML engineering.
Send the repository
In short
- Each request exposes something: source access for the build, business-rule names in the test run, a map of known weaknesses in the dependency report, and a client’s failure in the write-up. Agree terms first, then ask.
- Ask the vendor to build the system on a machine that hasn't built it before. The result is pass or fail, and it can't be argued with.
- A coverage percentage tells you little about a test suite. Ask which test fails if a named business rule is broken, then ask to see it fail.
- A dependency report lists every component and its license. Run it through a vulnerability scanner to learn which components have a published fix, and read the licenses for what you've taken on without noticing.
- Read an incident write-up for what it names. A cause, what was missing and a change that would have prevented it come from a team that operates things; blame on a person, a vendor or a user does not.
Ask the vendor for four artifacts: a build somebody else can run, a test run with its output, a dependency and license report, and an incident write-up. Each has a plain-language answer you can judge without reading a line of code.
The usual alternatives are weak. A technical advisor costs money, references come from the vendor, and rapport isn't evidence. These requests are cheap for a firm that already does the work, but each exposes something, listed under it, so agree terms (a nondisclosure agreement, a project the vendor may show, anonymization) before you ask.
1. A build somebody else can run¶
Ask them to check out the project on a machine that hasn't built it before and produce a running copy, with the instructions they followed. A demo of the finished software doesn't show you that. A project that only builds on its author's laptop can't be moved, handed over or given a second opinion.
What it exposes: the source code and build configuration, because a build needs both. Use a project the vendor owns or has the client's permission to show, or watch the build over a screen share.
A good answer comes with a page of instructions. A bad one explains why the request is unusual, or involves one specific person.
2. A test run, with its output¶
You aren't going to read the tests. You're checking three things you can see: there are some, they pass, and a run takes minutes, short enough to do routinely. The output exposes the names of tests, which often read as a list of the product's business rules.
Then ask the question that matters more than any coverage number. Name a business rule that would be expensive to get wrong (an order can't ship before payment clears, a user can't see another region’s records) and ask which test fails if somebody breaks it. Ask to watch it fail.
Whether a broken rule fails a test: A test fails or Nothing notices
How much code the tests run: Little or Every line
A test fails · Little
Whether a broken rule fails a test: A test fails. How much code the tests run: Little.
Narrow, but it notices
- The costly rule has a test that fails
A test fails · Every line
Whether a broken rule fails a test: A test fails. How much code the tests run: Every line.
Wide, and it notices
- Break the costly rule and a test fails
Nothing notices · Little
Whether a broken rule fails a test: Nothing notices. How much code the tests run: Little.
Blind
- Little runs
- A wrong answer passes
Nothing notices · Every line
Whether a broken rule fails a test: Nothing notices. How much code the tests run: Every line.
Coverage looks fine
- Every line runs, nothing is asserted
- A broken rule passes
A suite can run every line and still not notice a broken rule, so ask which test fails when a costly rule breaks, and watch it fail.
SourceThe test-run request this article describes; a classification, not measured data.
3. A dependency and license report¶
Every modern system is mostly other people’s code. Ask for the list with each item’s version and license. It is a software bill of materials, and tools generate one in a standard format, SPDX or CycloneDX: npm sbom (opens in a new tab) for a Node project, or syft (opens in a new tab) (run as syft ./project) for most other languages. An SBOM lists versions and licenses, not which versions have a published fix. So run it through a vulnerability scanner such as grype or osv-scanner (npm audit for a Node project), ask for the output, and define "too far behind" as a version with a published fix for a known vulnerability.
It answers two questions. What's stale: a maintenance cost you're about to inherit and a security exposure you're about to own. And what you've taken on legally: a dependency under a strong copyleft license in a product you intend to sell is a conversation your lawyers would rather have now. Some items will be behind. You want a team that knows which ones, and why. What it exposes: the name and version of every component, which is a map of the software’s weak points, so ask for it under a nondisclosure agreement.
4. A write-up of something that broke¶
Ask for an incident write-up from a real production system: what happened, what the customer experienced, why, and what changed afterward. Anonymized is fine, because it exposes a client’s failure and what the vendor does about it.
Read it for tone. A write-up that blames a person, a vendor or a user tends to come from a firm that hasn't learned from the incident. One that names a cause, admits what was missing, and describes a change that would have prevented it comes from a team that operates things.
Ask for all four before you sign¶
None of the four is exotic, so a refusal with no alternative is the signal. Agree the terms, send the requests in one email, give a week, and compare what comes back across vendors.
Next step
Send us the repository you inherited.
Or just tell us what you were given when the last engagement ended. We will tell you whether it builds, what it depends on, and what somebody taking it over would have to work out for themselves.
- Phone
- (214) 723-2510
- Reply
- A person replies, not a sequence: within one business day, from someone who would be on the engagement.
Packages
Start with a package
Four packages, each with its price on the page. Pick one and buy it online.
$199/month
Billed monthly
$182.42/month
Billed yearly
$499/month
Billed monthly
$457.42/month
Billed yearly
- Scammers stopped from sending email as you
- Your website and systems scanned every week
- Your email settings checked every week
- Everything in the 1 to 10 plan
- Covers up to 50 people
$199/month
after a one-time setup fee
- A site planned, designed and built for you
- Hosting and updates every month
$499/month
Billed monthly
$457.42/month
Billed yearly
$999/month
Billed monthly
$915.75/month
Billed yearly
- Your Google profile, fully set up
- A Google post every week
- 2 social posts a week
- Everything in Get found
- 4 social posts a week
- Your top 10 Google searches tracked
from$500/month
after a one-time setup fee
- Your data gathered from the tools you use
- A first analysis that answers your questions
- New findings every month
Related reading
Strategy4 min readSoftware support agreements: ask how hard it is to leave
The exit terms to check first, the three questions every support agreement must answer, and why a warranty is not support.
Delivery4 min readStatement of work template for software projects: what a SOW must include
A checklist to run before you sign, ending in a test you can give to someone who has never heard of the project.
Engineering3 min readLegacy software modernization: why rewrites get canceled, and a safer bet
A big-bang rewrite is a bad bet, not a bad pitch. Replacing the system one slice at a time changes the bet and gives the budget review something it can approve.
