Skip to content

Security

OT network anomaly detection with fuzzy hashing (TLSH)

How a TLSH similarity hash over Modbus/TCP packet clusters flags drift, the alternatives that failed, and what the method misses.

Computing America Cyber. Security assessments, penetration testing and staff training.

Send a capture
5 min readComputing America
An industrial network switch in a control cabinet, one cable running to a passive capture device.

In short

  • Industrial control traffic is largely deterministic, so a deviation from its baseline is a credible signal.
  • Cryptographic hashes are built to destroy similarity. Similarity-preserving hashes do the opposite. Small input changes produce small digest changes, and a baseline needs exactly that.
  • A single Modbus request can fall below TLSH's 50-byte minimum, so there is nothing to hash. The unit that works is a cluster of packets from one device conversation.
  • A sliding window of cluster digests follows normal operational drift. A patient attacker can still walk it.
  • The technique is a primary indicator. It tells you something is off without saying what, and it needs a second check behind it, against another source such as a change record or the device's own logs.

We flag anomalies in Modbus/TCP traffic by hashing clusters of packets from one device conversation with TLSH, then scoring each new digest against a sliding window of recent ones. We published the method as "Anomaly Detection in ICS Networks with Fuzzy Hashing" at the 2024 IEEE Cyber Awareness and Research Symposium (opens in a new tab), where clustering worked best of the four approaches the paper tried. The paper evaluated it on two public capture datasets. Our own tuning is labeled as ours below.

Why regular traffic helps

Enterprise networks are noisy: people browse, applications update, and normal is a wide distribution. Industrial networks send the same function codes between the same controllers at the same intervals, day after day. That predictability helps an attacker, but it also makes anomaly detection tractable. A departure from the pattern is rare here.

Why not an ordinary hash

A cryptographic hash changes completely when one byte changes. Two frames that differ only in a transaction ID get unrelated digests, so it can't measure similarity. Fuzzy (locality-sensitive) hashes invert that: small input changes produce small digest changes, and comparing two digests is cheap. The technique comes from malware clustering, where it's mature. The paper compared three candidates and chose TLSH (opens in a new tab).

CandidateResultVerdict
SSDEEPOn packet captures the paper found it gave "exclusively false positives" in most approaches, and it mostly returned "0 or near 0." The likely cause is ours, not the paper's: ssdeep derives its block size from input length, so short inputs share few trigger points.Dropped.
NILSIMSASlower than TLSH (about a minute per ten megabytes, single-threaded) and prone to false positives outside email.Dropped.
TLSHFast, accurate, scores inputs as small as fifty bytes, and per the paper more resistant to manipulated input than the others: its sliding window and k-skip n-grams offer "a defense against algorithmic file alterations intended to preserve high similarity scores."Chosen.
Three candidates the paper compared on packet captures. Resistance to crafted input weighed in the choice, but it concerns manipulated bytes in one input and says nothing about an attacker who shifts the baseline slowly.

Packets are too small to hash

Why not hash each packet? A Modbus/TCP message carries a 7-byte header, a function code and a small payload, and the protocol caps it at 260 bytes (the serial form caps at 256). The Modbus specification (opens in a new tab) sets those limits. A read request is 12 bytes: the header, a function code, a starting address and a quantity. The TLSH README says "In default mode the data must contain at least 50 bytes to generate a hash value" (opens in a new tab), so a request that short gets no digest at all. The paper needed a bigger unit of input and tried four.

  • Signature analysis (hash known-bad packets, compare to a blacklist). Failed: the frames are too short and too similar for a blacklist to discriminate.
  • Single-packet inspection. Failed for the size reason above, and drowned in false positives.
  • Whole-capture hashing against a reference set. It shows a capture came from the same network but can't localize an attack inside it, so it's a forensics tool and no detector.
  • Cluster hashing over a sliding window: group the packets of one device conversation, hash each cluster, compare each new digest to the recent window. This worked.

How the bytes are represented matters as much as the unit. The paper tested four encodings (string, binary, hex and sorted bytes) and measured average similarity separately on known-similar and known-different pairs. String separated the two best.

Fig. 1

% similarity

  • String, similar pairs91.33%
  • String, different pairs4.80%
  • Binary, similar pairs74.97%
  • Binary, different pairs9.83%
  • Hex, similar pairs89.13%
  • Hex, different pairs31.03%
  • Sorted bytes, similar pairs99.17%
  • Sorted bytes, different pairs95.43%

A high score on similar pairs means nothing if different pairs score high too, and sorted bytes failed that test.

SourceAverage similarity between packet hashes over multiple runs, from section IV of the CARS 2024 paper.

The window keeps the baseline honest

One digest per cluster is a measurement. The detector scores each new digest against the recent window of digests for the same device pair and surfaces clusters whose best similarity to it is unusually low. Because the window advances, the baseline follows normal drift.

  1. Span port

    Passive capture. No agent, no scanning.

  2. Cluster packets by conversation

    Sized to about one full repeating cycle.

  3. One TLSH digest per cluster

  4. Compare against the recent window

    Same device pair.

  5. Flag low similarity

  6. Second check

    Against a change record or device logs.

  7. Human review

Capture is passive: a span port copies traffic, and nothing in the chain sends a packet to a controller.
detector/score.pyPython
def score(cluster: list[Packet], window: Deque[str]) -> float | None:    """Score a conversation against the rolling baseline, not a packet    against a rule. Plant floors are periodic; the rule never fires."""    h = tlsh.hash(b"".join(p.payload for p in cluster))    if h is None:        return None          # under 50 bytes of entropy: no opinion offered    return min(tlsh.diff(h, prior) for prior in window) / DIFF_CEILING
A detector that is allowed to return nothing is worth more on an industrial network than one that always produces a number, because a confident score on 12 bytes of Modbus is noise wearing a threshold.

Cluster size is the main tuning knob; the paper uses a variable size N. Too small and you're back to the single-packet failure. Too large and a short attack is diluted by a long stretch of ordinary traffic. In our own use we tuned it per device pair, to roughly one full cycle of that conversation.

The paper evaluated the method on the CIC Modbus 2023 dataset (opens in a new tab) and the ICS PCAPS dataset on GitHub, both with labeled attack windows. The paper says the cluster method "worked the best" of the four approaches. It states no detection or false-positive rate, so read the ranking as qualitative.

What it does not do

  • It isn't a verdict. A similarity score says something is off, not what, and it belongs in front of a second check against a change record or the device's logs, and a person.
  • It can be walked. A sliding baseline follows an attacker who shifts traffic slowly enough that each cluster resembles the last few, and every step scores as normal. The paper doesn't test this. Scoring against a fixed, reviewed reference set alongside the window would catch slow drift, at the price of flagging legitimate change; that is a design option, not something we measured.
  • It covers one protocol, Modbus. Real plants also speak DNP3, BACnet, EtherNet/IP and a long tail of vendor protocols. The approach should generalize, but each needs its own parser and cluster sizing.
  • Canonicalizing cluster bytes before hashing, as intrusion detection rules normalize content, should cut false positives. It's future work, as is layering learned attribution on the similarity signal.

Ask how any plant detector was tested

Before you buy any plant detector, ask what its unit of input is, which attacks it was tested against, and on whose equipment. Ask what detection and false-positive rates it reports, and whether anyone tested slow drift.

Sources

  1. 1.Anomaly Detection in ICS Networks with Fuzzy Hashing (opens in a new tab), IEEE Cyber Awareness and Research Symposium (CARS) 2024,
  2. 2.TLSH: Trend Micro Locality Sensitive Hash (opens in a new tab), Trend Micro
  3. 3.CIC Modbus Dataset 2023 (opens in a new tab), Canadian Institute for Cybersecurity, University of New Brunswick
  4. 4.MODBUS Application Protocol Specification V1.1b3 (opens in a new tab), Modbus Organization,

Next step

Send us a capture.

Twenty minutes of traffic off one segment, and we will tell you whether it is regular enough to baseline and where this technique goes blind, which sets what an assessment of that segment should test.

Reply
A person replies, not a sequence: within one business day, from someone who would be on the engagement.

Packages

Start with a package

Four packages, each with its price on the page. Pick one and buy it online.

Billing