MTL-DoHTA: Multi-Task Learning-Based DNS over HTTPS Traffic Analysis for Enhanced Network Security

Abstract

The adoption of DNS over HTTPS (DoH) has significantly enhanced user privacy and security by encrypting DNS queries. However, it also presents new challenges for detecting malicious activities, such as DNS tunneling, within encrypted traffic. In this study, we propose MTL-DoHTA, a multi-task learning-based framework designed to analyze DoH traffic and classify it into three tasks: (1) DoH vs. non-DoH traffic, (2) benign vs. malicious DoH traffic, and (3) the identification of DNS tunneling tools (e.g., dns2tcp, dnscat2, iodine). Leveraging statistical features derived from network traffic and a 2D-CNN architecture enhanced with GradNorm and attention mechanisms, MTL-DoHTA achieves a macro-averaging F1-score of 0.9905 on the CIRA-CIC-DoHBrw-2020 dataset. Furthermore, the model effectively handles class imbalance and mitigates overfitting using downsampling techniques while maintaining high classification performance. The proposed framework can serve as a reliable tool for monitoring and securing sensor-based network systems against sophisticated threats, while also demonstrating its potential to enhance multi-tasking capabilities in resource-constrained sensor environments.