Adaptive Anomaly Detection in Database Transactions: Bridging Security Gaps with Reinforcement Learning

Abstract

Anomaly detection in database transactions is critical for safeguarding sensitive information and ensuring the integrity of operations in industries like finance, healthcare, and e-commerce. Existing techniques, including rule-based, machine learning, and deep learning methods, face challenges such as high false positive rates, poor adaptability to evolving patterns, and limited scalability in imbalanced datasets. This research proposes a novel Reinforcement Learning (RL)-based anomaly detection system to address these limitations. The model employs a dynamic reward mechanism and anomaly scoring system to classify transactions accurately while reducing false positives. It leverages the Kaggle Anomaly Detection in Transactions Dataset and a synthetically generated dataset for training and evaluation. Experimental results show that the RL-based model outperforms traditional methods, achieving a precision of 95.2%, recall of 92.4%, and an AUC-ROC score of 97.2%, significantly higher than Autoencoders, Isolation Forest, and Support Vector Machines. The proposed model’s adaptability and robustness make it a scalable solution for real-time anomaly detection, addressing critical gaps in existing techniques. This study advances database security by offering a highly accurate, adaptive, and efficient system for detecting anomalies in complex transactional environments.