Malware Clustering Using Family Dependency Graph

Abstract

Malware brings a major security threat on the Internet today. It is not surprising that much research has concentrated on detecting malware. Unfortunately, the current malware detection approaches suffer from ineffective detection of new malware samples. These models effectively identify the known malware samples but not new variants. To address this issue, we propose a novel malware detection approach based on the family graph. First, we trace the API calls of the monitored application, and then we generate the dependency graph based on the dependency relationship of the API calls. At last, we construct the family dependency graph via clustering the graphs of a known malware family. In this way, we can determine whether a new sample belongs to a known malware family. The evaluation results show that our approach is effective with small overhead compared to other existing approaches.