Skip to content

Data

Machine learning intrusion detection: what the data stages taught us

What we learned building an ML intrusion detector from raw packet capture to live alerts, and which stages were data work.

Computing America. Applications, integrations, data analysis and AI/ML engineering.

Describe the detection
5 min readComputing America
Seven connected lab vessels in a row, the liquid clearing from cloudy in the first to clear in the last.

In short

  • On our captures, many extreme values were attack traffic, so dropping rows beyond 1.5 times the interquartile range discarded some of it. Switching to clipping at 3 times raised validation F1; the cleaning section gives the size, and why it cannot say which of the two changes mattered.
  • Labeling individual packets makes many labels wrong, because a malicious session is full of innocuous traffic. Label flows.
  • On our engineered flow statistics, gradient-boosted trees scored F1 0.981 and the convolutional network 0.974, on the same leaked validation split. The gap is too small to rank them.
  • Build the pipeline as decoupled, replayable stages, so a crash at stage five doesn't force a re-run of stage one.
  • Profile before optimizing the part that feels expensive. In our live detector, inference was sub-millisecond and packet parsing was the slower step.

Four of the seven stages in our intrusion detector are data work, and one cleaning change visibly moved validation F1. We built seven decoupled stages from raw capture to a live detector, with four classical models and a convolutional network at stage five, in roughly fifty hours.

pipeline/main_pipeline.pyPython
def run_pipeline(self):    self.collect_data()        # 1. PCAPs -> data/raw_pcaps/{normal,malicious}    self.extract_features()    # 2. PCAPs -> packet/flow CSVs (scapy)    self.clean_data()          # 3. dedupe, IQR clip, engineer, scale    self.select_features()     # 4. ensemble vote across 4 methods    self.train_models()        # 5. SVM/RF/XGB/MLP + 1D CNN    self.evaluate_models()     # 6. metrics, ROC, confusion matrices    self.deploy_model()        # 7. live scapy sniff -> predict_proba -> alert
One line of the seven is the modeling.

Label flows, not packets

A naive extractor labels every packet and trains on packets. That yields millions of samples, many with false labels, because a malicious session contains plenty of ordinary handshakes and keepalives. Aggregating packets into flows, keyed on source and destination address, both ports and the protocol, removes many wrong labels, though every flow in a malicious capture still inherits that capture's label. Per flow we compute packet counts, byte totals, duration, packet-size and inter-arrival distributions, and derived rates.

pipeline/pcap_processor.py: flow keyingPython
flow_key = (    packet[IP].src,    packet[IP].dst,    src_port,    dst_port,    protocol,)flows[flow_key].append(packet)
Every packet with the same five values lands in one list, and each list is then summarized into a single row.

Two flow features, packet rate and average packet size, with one threshold each, already make a detection rule. Move the cuts in the demo below and the confusion matrix moves with them, before any model exists.

Interactive · Applied machine learning

Two features and a threshold rule

Flow explorer · 146 synthetic flows

drag the sliders

log₁₀ packets / seclog₁₀ avg packet bytes01231.522.533.5
rule

confusion matrix

pred 0
pred 1
true 0
90
0
true 1
18
38
precision1.000
recall0.679
f10.809
accuracy0.877
Blue dots are normal flows, red dots are malicious. The dashed red lines are the rule’s decision boundaries; flows in the flagged region get a white outline. The same exercise on real PCAP-derived features is what feature selection and model training automate.
What this isSynthetic flows drawn from the shape of the real capture, not the capture itself. Move the thresholds and the confusion matrix moves with them: the point is how much of the result is decided by the feature extraction and the cut, before any model is chosen.

Clean the data: clip outliers and deduplicate

Switching from dropping rows at one and a half times the interquartile range to clipping at three times it moved validation F1 by roughly 3.5 points. That is one figure for two changes, dropping to clipping and 1.5 to 3 times, so it does not say which one mattered. It is also optimistic: our pipeline oversampled before the train and validation split, so copies of minority rows sat on both sides and inflated the score. Treat 3.5 as the direction of the effect, not a size to expect.

pipeline/data_cleaner.py: outlier treatmentPython
numeric_columns = df.select_dtypes(include=[np.number]).columnsfor col in numeric_columns:    if col != 'label':        Q1 = df[col].quantile(0.25)        Q3 = df[col].quantile(0.75)        IQR = Q3 - Q1        lower_bound = Q1 - 3 * IQR        upper_bound = Q3 + 3 * IQR        df[col] = df[col].clip(lower_bound, upper_bound)
One of the two changes is the word clip; the other is the multiplier. Dropping those rows throws away the attack traffic, because on our captures the extreme values were not noise: many of them were the attack traffic.

Deduplication was the other surprise. Dropping duplicate flows removed about 22% of rows, and we assumed the counter was broken. Scans and floods produce many flows with identical feature vectors, so after a random split the same row sits in both train and test, and a model is partly scored on rows it has already seen. Deduplicate before splitting.

Select features by disagreement

We took an ensemble vote across an F-test, mutual information, recursive feature elimination and random-forest importance, and the value is where they disagree. The F-test measures how far the class means separate, a linear signal. Mutual information, which measures the dependency between two variables (opens in a new tab), counts any relationship with the label, linear or not. Engineered binary features ranked high on mutual information and low on the F-statistic, so selecting with the F-test alone would have dropped real signal.

pipeline/feature_selector.py: the votePython
top_f    = set(f_scores['feature'][:20].tolist())top_mi   = set(mi_scores['feature'][:20].tolist())top_rfe  = set(rfe_features)top_tree = set(tree_features) for feature_set in [top_f, top_mi, top_rfe, top_tree]:    for feature in feature_set:        feature_counts[feature] = feature_counts.get(feature, 0) + 1 # Keep anything at least two of the four methods asked for.self.selected_features = [f for f, c in feature_counts.items() if c >= 2]
The threshold on the last line is the only tunable in the stage, and it is a judgment rather than a result. Two votes keeps features a single method would have dropped; four keeps only what everything agrees on, which on this dataset is a much smaller set than it sounds.

Interactive · Feature selection

Feature selection by ensemble vote

Ensemble feature voter · 24 features · 4 methods

mirrors feature_selector.select_best_features()

min votes

19 / 24 selected

dot orderF-testMIRFERF imp.

packet_rate
In the top set of F-test, MI, RFE and RF imp.
4
packets_per_second
In the top set of F-test, MI, RFE and RF imp.
4
byte_rate
In the top set of F-test, MI, RFE and RF imp.
4
avg_iat
In the top set of F-test, MI, RFE and RF imp.
4
avg_packet_size
In the top set of F-test, MI, RFE and RF imp.
4
std_iat
In the top set of F-test, MI and RF imp.
3
std_packet_size
In the top set of F-test, MI and RFE
3
packet_count
In the top set of F-test, RFE and RF imp.
3
duration
In the top set of MI, RFE and RF imp.
3
total_bytes
In the top set of F-test and RF imp.
2
avg_bytes_per_packet
In the top set of F-test and MI
2
min_packet_size
In the top set of F-test and RF imp.
2
max_iat
In the top set of MI and RF imp.
2
max_packet_size
In the top set of MI and RFE
2
min_iat
In the top set of RFE and RF imp.
2
is_well_known_port
In the top set of RFE and RF imp.
2
is_registered_port
In the top set of RFE and RF imp.
2
dst_port
In the top set of MI and RFE
2
is_dynamic_port
In the top set of MI and RF imp.
2
src_port
In the top set of MI
1
syn_ack_ratio
In the top set of RFE
1
protocol
In the top set of MI
1
dst_ip
In no method’s top set
0
src_ip
In no method’s top set
0

features per method (top set)

F-test11
MI15
RFE14
RF imp.15

vote distribution

exactly 0 methods2
exactly 1 method3
exactly 2 methods10
exactly 3 methods4
exactly 4 methods5
Each row is a flow feature. Colored dots mark the selection methods that placed it in their top-K. The sort is by vote count, then F-score. Move the threshold up and the selected set tightens around the features that multiple methods agree on - exactly what selected = [f for f, c in counts.items() if c >= 2] does in the project.
What this isFour selection methods vote on twenty-four features; you set how many votes a feature needs to survive. The per-method scores are illustrative, calibrated so the ensemble reproduces the kind of rank disagreement the project’s report describes; the vote arithmetic over them is real. Require more votes and watch the features everyone agrees on stay while the ones a single method liked drop out.

The convolutional network did not beat boosted trees

The network had the full regularization stack: batch normalization, dropout, weight decay and early stopping. On the same validation split it scored F1 0.974 against 0.981 for gradient-boosted trees, as the output below shows. That split carries the oversampling leak, which favors a model that memorizes rows, so a gap this small may not survive a clean split, and we did not test why. We did not try raw packet sequences, which would be a different pipeline.

Fig. 1

validation F1, leaked split

  • Gradient-boosted trees0.981
  • Convolutional network0.974

The convolutional network scored 0.007 F1 below gradient-boosted trees, on a validation split the oversampling leak inflates: too small a gap to rank them.

SourceStage 6 output of our pipeline, reproduced in this article. Both scores are on the validation split that the oversampling leak inflates, so neither is a clean estimate; a clean split could move both.

The literature points the same way for MLP-like networks. Grinsztajn, Oyallon and Varoquaux benchmarked trees against neural networks on a standardized set of tabular problems and found trees still state of the art. They traced it to three properties: such networks are less robust to uninformative features, are rotation invariant (which discards the meaning of each column), and are biased toward overly smooth solutions, so they struggle with irregular target functions. The benchmark's datasets have heterogeneous columns, as flow statistics do. The finding covers medium-sized data, around ten thousand samples, and says nothing about ten million rows or raw sequences.

Sources

  1. 1.Why do tree-based models still outperform deep learning on typical tabular data? (opens in a new tab), Grinsztajn, Oyallon & Varoquaux,
  2. 2.mutual_info_classif (opens in a new tab), scikit-learn
stage 4/7: feature selection, ensemble votepipeline output
feature                F     MI   RFE  Tree   votesflow_iat_std           *      *     *     *      4fwd_pkt_len_max        *      *     *     .      3init_win_bytes_fwd     *      .     *     *      3bwd_pkt_len_mean       .      *     .     *      2   kept at the threshold--stage 6/7  gradient-boosted trees   F1 0.981stage 6/7  1D convolutional net     F1 0.974
The two models are 0.007 apart on a validation split the oversampling leak inflates, while the cleaning change in stage 3 moved F1 by about 3.5 points on that same split. We report that because a client who is told the impressive model won learns nothing about where to spend the next dollar.

The threshold is a business decision

A classifier doesn't raise alerts. A threshold on its output does, and every position on the slider trades a missed detection against a false one. The math can't say which is right. What does an investigation cost, what does a miss cost, and how many alerts will the receiving team work before closing them unread? Show the receiving team a curve with a handle on it and let them choose.

Interactive · Evaluation

Where to put the alert threshold

Alert threshold sweep · 600 simulated predict_proba outputs

mirrors anomaly_detector.alert_threshold

t = 0.700.000.250.500.751.00predict_proba (malicious class)countnormalmalicious
presets

Lower the threshold to catch more attacks (higher recall) at the cost of more false alerts. Raise it for cleaner alerts at the cost of misses. The project ships at 0.7 in config.json, where this synthetic set happens to produce no false positives; around 0.5 both error columns fill and the trade is easier to read.

at this threshold

precision1.000
recall0.665
f10.799
FPR0.000
TP / FN113 / 57
FP / TN0 / 430
alerts / hour~678

assuming 1 flow/sec

Each bar is a probability bin; blue = normal flows the model scored at that probability, red = malicious. The model is honest but not perfect: the two distributions overlap. The job of the threshold is to pick the cut that minimizes the cost of whichever error is worse for your operation.
What this isSix hundred simulated scores, one slider, and the two error columns that a client actually argues about. The distribution is synthetic; the trade-off is not, and neither is the fact that the defensible threshold is a business decision rather than an optimum.

Three structural decisions that made it survivable

  • Every stage writes its output to disk and reads its input from disk, so a crash at stage five doesn't force a re-collection at stage one. Explicit pipeline state records which stages have completed.
  • The selected feature column list is written at training time and read back at inference. Column order is part of the contract between training and serving, and rebuilding the list at inference produces confident nonsense with no error.
  • One configuration file holds paths, thresholds and hyperparameters, so someone who wasn't there can reproduce a run.

Profile before optimizing

We expected inference to be the live detector's bottleneck, since it feels expensive. Scoring a flow is sub-millisecond. The packet parsing and field extraction in front of it is slower, so throughput means replacing the parser, and touching the model would accomplish nothing.

What we would change next time

  • Model the pipeline as a dependency graph, so stages re-run individually, cache by content hash and parallelize.
  • Split first and fit the oversampler, scaler, outlier bounds and selector on the training fold only, from day one. We oversampled before the split, and the honest answer to a reviewer asking how leakage was avoided is that it wasn't.
  • Version intermediate datasets with the code, so “why did the score drop on Monday” takes two minutes to answer.
  • Plot the per-class distribution of every feature before training anything. Half the bugs and most of the wins are visible in those histograms.

Before you commission a model

Write down what you are trying to detect and how you would know you were right. Then hold out a capture from a different day or network, and set how many alerts the receiving team will work before you choose a threshold.

Next step

Tell us what you are trying to detect.

And how you would know you were right. We will say whether the data you already hold can support it, before anybody chooses an algorithm.

Reply
A person replies, not a sequence: within one business day, from someone who would be on the engagement.

Packages

Start with a package

Four packages, each with its price on the page. Pick one and buy it online.

Billing