Why I don't trust a perfect detection score
During my thesis on ML-based DDoS detection, one model hit 100% on every metric. That was the moment I knew something was wrong.
The suspicious result
Accuracy, precision, recall, F1: all perfect. In security, that almost never means you built a perfect detector. It usually means the model has seen the answers.
# first run (training data) accuracy 1.000 f1 1.000 <- too good to be true
Where the answers leaked in
- Scoring on training data. The model was graded on rows it had memorised.
- Balancing before splitting. SMOTE creates synthetic samples from real ones. Run it before the split and near-copies of test rows end up in training.
- Give-away features. Some fields identify the capture more than the attack, like a timestamp-style field. The model learns the dataset, not the traffic.
Rule: split first. Everything else (scaling, SMOTE, feature selection) is fitted on the training fold only.
The fix
- Split the data before any preprocessing.
- Apply SMOTE inside each training fold only.
- Use 5-fold cross-validation and report the mean, not the best fold.
- Drop identifier-like features and check what the model actually relies on.
The honest numbers
After the fix, the best model scored an F1 of around 0.99, and the runner-up around 0.96 while training in a fraction of the time. Still strong, but now it's a number I can defend.
model A f1 ≈ 0.99 train ≈ 525 s model B f1 ≈ 0.96 train ≈ 12 s # exact per-class figures: see thesis appendix
What it means for a SOC
- A detector that is 99% on a lab dataset can still be noisy on real traffic. Test it on data from a different source.
- Speed matters. A slightly weaker model that retrains in seconds can be the better operational choice.
- Always ask: what does the model look at? If you can't answer, you can't trust the alert.