Machine Learning for Anomaly Detection: A Beginner's Guide to Network Security

Your firewall logs a connection to an odd port at 3:14 a.m. Nothing matches a known signature. Is it a misconfigured backup job, or someone probing for a way in? Signature-based tools stay quiet, because they only catch what someone has already described. Machine learning takes another route: learn what normal looks like on your network, then flag whatever drifts away from it. The idea is simple. The execution takes care. What follows covers the parts that actually decide whether a project succeeds — framing the question, preparing traffic data, picking a model, and judging the results before you wake anyone up at night.

Why "Normal" Is Harder to Define Than It Sounds

Networks have a rhythm. People log in around nine, backups run overnight, patching causes a spike once a month, and the marketing team uploads a large file every Friday. Anomaly detection learns that rhythm statistically instead of from a list of rules, which makes it useful for the things rules handle badly: beaconing to a new domain, a host scanning ports it has never touched, a service account suddenly transferring ten times its usual volume.

It also has a blind spot. On a noisy, flat network where every day looks different, "normal" is a very wide band, and everything looks slightly unusual. Before writing a line of code, sit with the person who runs the network and ask them to describe last week. Their answer tells you whether you have a signal worth modelling.

Start With a Question, Not an Algorithm

"Can we use machine learning?" is not a question you can evaluate. "Can we catch internal hosts that start contacting destinations they have never contacted before?" is. A narrow question decides your features, your labels, and how you measure success.

Then write down what you would do with an alert. If the honest answer is "someone would look at it eventually" or "we would ignore it," stop here. Detection without a response path just produces a dashboard nobody opens.

Preparing the Data

Collect traffic you are allowed to use

Start with flow records rather than full packet capture. NetFlow, IPFIX, or Zeek logs give you who talked to whom, how much, and for how long — compact enough to store for months and rich enough for most behavioural features. Full packet capture is expensive to keep and holds far more personal data than you need at this stage.

Confirm your authorisation in writing before you capture anything, and involve whoever owns privacy decisions. Monitoring employee traffic is a policy question as much as a technical one, and getting it wrong is far more damaging than a missed detection.

Build features that describe behaviour

Aggregate per host into short windows — five minutes is a reasonable starting point — and describe each window with numbers rather than addresses. Useful features include:

• Counts of distinct destinations, ports, and protocols contacted
• Bytes sent and received, plus the ratio between them
• Number of failed or refused connections
• Entropy of destination addresses, which rises during scanning
• Average session duration and bytes per packet
• Hour of day and whether the host is usually active then

Raw IP addresses and port numbers are usually poor features, because a model that memorises them will not generalise to next month's infrastructure.

Split and clean with time in mind

A random train-test split quietly leaks the future into the past. Split by time instead: train on the first four weeks, test on the fifth. Apply a log transform to heavily skewed values like byte counts, scale features if your model depends on distances, and decide in advance how to handle missing windows rather than letting them become zeros.

Labels are the scarce resource. Most teams do not have a clean, complete set of confirmed incidents. That is fine — it just pushes you toward unsupervised methods or toward using a handful of known incident timestamps as a sanity check rather than as training data.

Choosing a Model Without Overthinking It

Build a baseline first: thresholds on connection counts and bytes. If your model cannot beat a well-tuned threshold, it is not earning its place.

From there, an isolation forest is a sensible first attempt on tabular flow features. It needs no labels, assumes little about the shape of your data, and produces a score you can rank. One-class approaches and autoencoders work well when you have plenty of clean normal traffic and very few attacks, using reconstruction error as the signal. If you do have decent labels, gradient-boosted trees on the same features are usually hard to beat.

Treat the score as a ranking, not a probability. The model does not know that 0.87 means "attack"; it only knows that 0.87 is more unusual than 0.42.

Evaluating When Attacks Are Rare

Accuracy is the wrong headline number. If 99% of your windows are normal, a model that always says "normal" scores 99% and catches nothing. Work from the confusion matrix instead, and talk about it in plain language: true positives are attacks you caught, false positives are alerts that waste an analyst's morning, false negatives are the ones that got through.

Precision tells you how many of your alerts are real. Recall tells you how many real incidents you found. For rare-event detection, the precision-recall curve is more informative than the ROC curve, because it does not flatter you with a huge number of true negatives. But the metric that convinces a security team is alert volume: a model with slightly lower recall that produces forty alerts a day will be used, while one producing five thousand will be switched off.

Keep these habits from the start:

• Test on a later time period than you trained on, always
• Report alerts per day alongside any ratio
• Break performance down by attack type; averages hide the misses that matter
• Compare against your rule-based baseline, every time
• Watch for leakage — if a feature only exists because someone investigated an incident, the model learns the investigation, not the attack

A First Project You Can Finish in a Week

1. Pick one behaviour to detect, such as hosts contacting an unusual number of destinations.
2. Collect two months of flow logs from one network segment, with written approval.
3. Aggregate into five-minute windows per host and compute a small set of features.
4. Build the threshold baseline and record its alert volume and hit rate.
5. Train an isolation forest on the first six weeks, test on the remaining two.
6. Review the top twenty alerts by hand and write down what each one actually was.

Those twenty reviews teach you more than another week of tuning.

Keeping It Useful After Launch

Networks change, and a model trained in March will quietly decay by September. Run new versions in shadow mode for two weeks before anyone depends on them, and watch for a creeping rise in alert volume — that is usually the first sign of drift. Keep a feedback loop: analysts mark false positives, you review them weekly, and the labels come back as better features rather than as blind retraining.

Keep the model as one signal among many, documented well enough that a colleague can explain why an alert fired. Start small, prove it on a single behaviour, and expand only when the alerts it produces are ones your team actually acts on.

Photo: panumas nikhomkhai / Pexels