Skip to content

Delivery

How to evaluate a software vendor without reading code

Four requests that show whether a firm builds, tests, tracks its dependencies and learns from failure.

Computing America. Applications, integrations, data analysis and AI/ML engineering.

Send the repository
3 min readComputing America
An inspector's clipboard, flashlight and moisture meter in front of a finished wall.

In short

  • Each request exposes something: source access for the build, business-rule names in the test run, a map of known weaknesses in the dependency report, and a client’s failure in the write-up. Agree terms first, then ask.
  • Ask the vendor to build the system on a machine that hasn't built it before. The result is pass or fail, and it can't be argued with.
  • A coverage percentage tells you little about a test suite. Ask which test fails if a named business rule is broken, then ask to see it fail.
  • A dependency report lists every component and its license. Run it through a vulnerability scanner to learn which components have a published fix, and read the licenses for what you've taken on without noticing.
  • Read an incident write-up for what it names. A cause, what was missing and a change that would have prevented it come from a team that operates things; blame on a person, a vendor or a user does not.

Ask the vendor for four artifacts: a build somebody else can run, a test run with its output, a dependency and license report, and an incident write-up. Each has a plain-language answer you can judge without reading a line of code.

The usual alternatives are weak. A technical advisor costs money, references come from the vendor, and rapport isn't evidence. These requests are cheap for a firm that already does the work, but each exposes something, listed under it, so agree terms (a nondisclosure agreement, a project the vendor may show, anonymization) before you ask.

1. A build somebody else can run

Ask them to check out the project on a machine that hasn't built it before and produce a running copy, with the instructions they followed. A demo of the finished software doesn't show you that. A project that only builds on its author's laptop can't be moved, handed over or given a second opinion.

What it exposes: the source code and build configuration, because a build needs both. Use a project the vendor owns or has the client's permission to show, or watch the build over a screen share.

A good answer comes with a page of instructions. A bad one explains why the request is unusual, or involves one specific person.

2. A test run, with its output

You aren't going to read the tests. You're checking three things you can see: there are some, they pass, and a run takes minutes, short enough to do routinely. The output exposes the names of tests, which often read as a list of the product's business rules.

Then ask the question that matters more than any coverage number. Name a business rule that would be expensive to get wrong (an order can't ship before payment clears, a user can't see another region’s records) and ask which test fails if somebody breaks it. Ask to watch it fail.

Fig. 1

Whether a broken rule fails a test: A test fails or Nothing notices

How much code the tests run: Little or Every line

A test fails · Little

Narrow, but it notices

  • The costly rule has a test that fails

A test fails · Every line

Wide, and it notices

  • Break the costly rule and a test fails

Nothing notices · Little

Blind

  • Little runs
  • A wrong answer passes

Nothing notices · Every line

Coverage looks fine

  • Every line runs, nothing is asserted
  • A broken rule passes

A suite can run every line and still not notice a broken rule, so ask which test fails when a costly rule breaks, and watch it fail.

SourceThe test-run request this article describes; a classification, not measured data.

3. A dependency and license report

Every modern system is mostly other people’s code. Ask for the list with each item’s version and license. It is a software bill of materials, and tools generate one in a standard format, SPDX or CycloneDX: npm sbom (opens in a new tab) for a Node project, or syft (opens in a new tab) (run as syft ./project) for most other languages. An SBOM lists versions and licenses, not which versions have a published fix. So run it through a vulnerability scanner such as grype or osv-scanner (npm audit for a Node project), ask for the output, and define "too far behind" as a version with a published fix for a known vulnerability.

It answers two questions. What's stale: a maintenance cost you're about to inherit and a security exposure you're about to own. And what you've taken on legally: a dependency under a strong copyleft license in a product you intend to sell is a conversation your lawyers would rather have now. Some items will be behind. You want a team that knows which ones, and why. What it exposes: the name and version of every component, which is a map of the software’s weak points, so ask for it under a nondisclosure agreement.

4. A write-up of something that broke

Ask for an incident write-up from a real production system: what happened, what the customer experienced, why, and what changed afterward. Anonymized is fine, because it exposes a client’s failure and what the vendor does about it.

Read it for tone. A write-up that blames a person, a vendor or a user tends to come from a firm that hasn't learned from the incident. One that names a cause, admits what was missing, and describes a change that would have prevented it comes from a team that operates things.

Ask for all four before you sign

None of the four is exotic, so a refusal with no alternative is the signal. Agree the terms, send the requests in one email, give a week, and compare what comes back across vendors.

Next step

Send us the repository you inherited.

Or just tell us what you were given when the last engagement ended. We will tell you whether it builds, what it depends on, and what somebody taking it over would have to work out for themselves.

Reply
A person replies, not a sequence: within one business day, from someone who would be on the engagement.

Packages

Start with a package

Four packages, each with its price on the page. Pick one and buy it online.

Billing