Common Mistakes in AI Phishing Detection and How to Fix Them

Most phishing detectors look impressive in a demo and disappointing by the second month in production. The model is usually not the problem. What breaks is everything around it: the data it learned from, the way alerts get triaged, the feedback loop nobody owns, and the slow drift that turns a sharp detector into a noisy one. Here are the mistakes that show up again and again, and what to do about each of them.

Training on a Snapshot Instead of a Moving Target

A detector trained on a fixed corpus learns that corpus. If last year's campaigns leaned on "your mailbox is full" subject lines and a handful of spoofed domains, the model gives those signals far too much weight. Then a new campaign arrives with a one-line message, a PDF attachment and no links at all, and it slips straight through.

Overfitting also hides behind flattering test scores. When you split a dataset randomly, near-duplicate emails land in both halves, so the model appears to generalise when it has really just memorised. A campaign sent to 300 employees in slightly different wording can effectively leak the test set into the training set.

Fix it with time-based validation. Train on January through March, validate on April, and report results on May. Keep a rolling window so old campaigns age out instead of teaching the model about threats that no longer exist. Test against the phishing messages your own users reported in the last 30 days — that is the closest thing you have to ground truth about what is actually arriving. Then break the results down by segment: internal senders, external partners, non-English mail, each business unit. An aggregate number hides the group the detector is quietly failing.

Measuring the Wrong Number

Accuracy is a terrible headline metric when phishing is a small fraction of traffic. A model that labels everything clean can score above 99% and catch nothing at all. Metrics worth putting on a dashboard:

• Recall on phishing — how much malicious mail actually reaches inboxes.
• False-positive rate at the live threshold — what gets blocked or quarantined by mistake.
• Time to detection — minutes from delivery to alert, not hours.
• Noise ratio — the share of alerts that required no action at all.

Set the threshold by consequence rather than habit. A missed credential-phishing email delivered to a finance team costs more than a newsletter wrongly quarantined. Many teams run two thresholds: an aggressive one for auto-quarantine on high-confidence signals, and a conservative one that simply adds a warning banner. Write down who can change those thresholds and why, because a silent tweak in March will confuse everyone debugging alerts in June.

Alert Fatigue Is a Triage Design Problem

Detection quality collapses the moment analysts stop reading alerts properly. If the console produces 400 items a day and the team has two people, bulk-closing is not laziness — it is arithmetic. And every bulk close teaches the system nothing.

1. Group related alerts into campaigns by sender infrastructure, subject pattern and URL family, then review the campaign instead of 80 near-identical items.
2. Auto-remediate the top confidence tier: quarantine, strip or rewrite links, and notify the recipient. Reserve human attention for the ambiguous middle.
3. Attach a reason code to every alert — lookalike domain, first-time sender, urgency language, unusual attachment type. Analysts triage faster, and the codes become useful labels later.
4. Rotate triage duty and cap the daily queue. An analyst who has reviewed 200 alerts in a row will miss things, and that is a workload failure, not an attention failure.

Starving the Feedback Loop

User reports are the cheapest threat intelligence a security team has, and most organisations waste them. Someone forwards a suspicious email to the service desk, it sits in a shared mailbox, and nobody labels it. Meanwhile the reporter hears nothing back, concludes that reporting is pointless, and stops.

Close the loop deliberately. Give people a one-click report button that works from mobile, send an automatic acknowledgement, and tell reporters when a message turns out to be malicious. Capture the analyst verdict — phish, spam, legitimate — as a structured label rather than a comment in a ticket. Schedule labelling on a fixed cadence, weekly if volume allows, and feed those labels back into retraining so the model learns your environment and not a vendor's generic sample set.

Ownership matters here too. Name a person responsible for reviewing performance monthly, watching for drift after vendor model updates, and checking whether a legitimate sender suddenly started getting blocked after a marketing tool migration. Allowlist entries without expiry dates pile up and become permanent blind spots.

Blind Spots Beyond the Inbox

Plenty of detectors read the email body as plain text and stop there. That leaves a long list of tricks unexamined: QR codes embedded in images, attacks written entirely into a screenshot, HTML attachments that render a fake login page locally, calendar invitations with malicious links, and chat or SMS messages that never touch the mail gateway.

Lookalike infrastructure is the other gap. If nobody monitors for newly registered domains that resemble yours, the first person to notice will be a customer entering a password. Domain monitoring is unglamorous and it works. So does extracting text from image attachments and checking links against reputation services at click time rather than only at delivery, because a URL that was clean on Monday can be hostile on Thursday.

A Short Starting Checklist

If the current setup feels shaky, do not rebuild everything at once. Three moves cover most of the ground:

1. Re-run validation with a time-based split and publish the numbers by segment, so you can see where detection actually fails.
2. Pick one metric that reflects cost of error, put it on a monthly review, and give it an owner.
3. Make the report button visible, acknowledge every report, and start labelling what users send you.

None of that requires a new platform. It requires someone to decide that a detector is a system to be operated, not a product to be installed and forgotten.

Photo: geralt / Pixabay