Detection of malicious PDF files using a two-stage machine learning algorithm

Abstract

Portable document format (PDF) files are increasingly used to launch cyberattacks due to their popularity and increasing number of vulnerabilities. Many solutions have been developed to detect malicious files, but their accuracy decreases rapidly in face of new evasion techniques. We explore how to improve the robustness of classifiers for detecting adversarial attacks in PDF files. Content replacement and the n-gram are implemented to extract robust features using proposed guiding principles. In the two-stage machine learning model, the objects are divided based on their types, and the anomaly detection model is first trained for each type individually. The former detection results are organized into tree-like information structure and treated as inputs to convolutional neural network. Experimental results show that the accuracy of our classifier is nearly 100% and the robustness against evasive samples is excellent. The object features also enable the identification of different vulnerabilities exploited in malicious PDF files.