Skip to content

Security

Penetration test vs vulnerability scan: what you are actually buying

Both are legitimate purchases. A sample report and a look for chaining show which one a quote is selling, and four questions test the vendor behind it.

Computing America Cyber. Security assessments, penetration testing and staff training.

Send the quotes
3 min readComputing America
A steel padlock clamped in a bench vise, a pry bar resting against it.

In short

  • Chaining separates the two: real intrusions combine several unremarkable weaknesses, and a scan doesn't try that. One CVE per finding is scan output, whatever the title page says.
  • Ask a vendor what happens when they find nothing. The answer shows whether you're buying an assessment or a report.
  • Scope written as a list of addresses buys a test of those addresses. Scope written as a goal (reach this data, reach this controller) buys a test of whether the goal is reachable.
  • Put the raw evidence and a retest in the contract. A finding you can't reproduce is one you can't verify was fixed.
  • If a questionnaire is the driver, say so up front. Buying a broader test than the form asks for is a decision to make deliberately.

A vulnerability scan runs automated checks for known flaws and reports each one. A penetration test is a person trying to reach something you care about, exploiting flaws and chaining them on the way.

The failure is buying a scan while believing you bought a test, then telling a customer or an insurer you've been penetration tested. A quote can be a scan with a title page. We sell this service, so the useful thing we can do is show you how to interrogate a quote, ours included.

Compared onVulnerability scanPenetration test
CadenceAutomated, repeatableManual, a point-in-time judgment by a person
Right forTracking hygiene across a large estateWhether a determined attacker reaches what you care about
Report shapeOne CVE per finding, CVSS score, generic fix textA narrative: this got us here, that let us do the next thing
What each purchase is

Chaining is what separates them: real intrusions combine several unremarkable weaknesses into one serious one, and a scan doesn't try that. NIST's technical guide to information security testing (opens in a new tab) describes vulnerability scanning and penetration testing as separate techniques. Ask for a sanitized sample report before you sign: a list of single CVEs sorted by severity is scan output, formatted.

Scope it as a goal, not a list

Most scopes are lists of addresses or applications, because those are easy to write and easy to price. You get a test of those things, which helps only if your question was about those things. It usually wasn't.

The stronger form is a goal. From the guest wireless, can you reach the file share holding customer drawings? From a phished sales account, can you reach the production controllers? A goal-scoped test can answer yes by a route your list didn't include. A list-scoped test can't produce that finding, because it stays off the system the route ran through. Goal scoping makes you say what you're protecting, and that can be harder than the technical work.

Ask four questions before you sign

  1. 1.What happens if you find nothing? A good answer is “we would tell you, and here is what we'd have had to do for that answer to mean anything.” If the vendor can't describe that outcome, ask what the report would say.
  2. 2.Who does the work, and can I read something they wrote? Testing quality belongs to individuals, and the name on the proposal can differ from the name on the keyboard.
  3. 3.Do I get the raw evidence (requests, responses, screenshots, timestamps) or only the write-up? Without it, a disputed finding can't be settled and a fix can't be verified.
  4. 4.Is a retest included, and for how long? Findings get fixed weeks later. A test with no retest ends in claims about a system that no longer exists.

When a test is the wrong first purchase

A test earns its fee when two things hold: you can't yet name your worst exposure, and someone owns the fixing and has a window to do it. Call the second one capacity to act. Cross the two and the picture below shows which purchase comes first.

Fig. 1

Capacity to act on findings: Owner and a window to fix or None

How well you already know the exposure: Unknown or Known

Owner and a window to fix · Unknown

Buy the test

  • Scope it as a goal
  • Retest included

Owner and a window to fix · Known

Fix first, test afterward

  • Flat network, shared admin credentials, no logging

None · Unknown

Build capacity first

  • Scans track hygiene meanwhile

None · Known

Defer the test

  • A report would only add documentation

A test is worth buying only where the exposure is still unknown and someone owns the fixing; in the other three quadrants, fix, build capacity or wait.

SourcePlanning heuristic from this article, not measured data.

If a questionnaire is the driver, say so up front. Read what the form asks for, because the scope should answer that question and it came from somebody else's form. Buying a broader test than the form asks for is a decision. Make it deliberately and tell the vendor which one you're making.

Write one sentence before you request quotes

"The thing I'm afraid of is someone reaching X by way of Y." If you can write it, put it in the scope. Let three firms say how they would go after it, and compare those answers before the prices. If you can't write it, that's the work to do first.

Sources

  1. 1.SP 800-115: Technical Guide to Information Security Testing and Assessment (opens in a new tab), NIST,

Next step

Send us the quotes.

We will tell you what each one actually commits to, where they differ, and which questions to put back to the vendors, including where our own quote would be the weaker one.

Reply
A person replies, not a sequence: within one business day, from someone who would be on the engagement.

Packages

Start with a package

Four packages, each with its price on the page. Pick one and buy it online.

Billing