Rigorous Evaluation of Machine Learning Intrusion Detection for Water Treatment Systems on SWaT Network Traffic

Abstract

Intrusion detection systems (IDSs) for industrial control networks are commonly evaluated using random stratified splits, placing rows from every recorded attack in both training and test sets. Although convenient, this practice measures a model’s ability to recognise repetitions of patterns it has already seen rather than its ability to detect novel attacks. We revisit supervised and unsupervised machine-learning IDSs on the Secure Water Treatment (SWaT) dataset’s network-traffic modality, extending a prior conference study, and quantify the effect of more rigorous evaluation protocols. We evaluate five model families (XGBoost, a convolutional–MLP hybrid, a bidirectional LSTM classifier, an unsupervised LSTM-Autoencoder, and a temporal convolutional network) under three protocols: stratified random, attack-held-out, and leave-one-attack-out (LOO). Under LOO on a 30-file subsample, every supervised classifier scores below random on the majority of held-out attacks; the unsupervised LSTM-Autoencoder retains the best solo mean of 0.550 with a strongly bimodal per-attack distribution spanning 0.046 to 0.894. A sign-adjusted oracle-bound ensemble flips members whose per-attack AUROC inverts achieves a mean LOO AUROC of 0.844; adding the TCN as a fourth ensemble member does not improve the result, providing evidence that what is needed is an additional detection mode rather than another supervised classifier. We additionally report recall at a 5% false-positive-rate budget, paired Wilcoxon significance tests, and bootstrap confidence intervals. The full preprocessing, evaluation, and ensemble pipeline is released, and we argue that attack-held-out and LOO should be standard protocols for network-traffic IDS benchmarks on SWaT.