Malware analysis and detection via activity trees in user-dependent environment

Abstract

We present a new system that offers detection and analysis of modern complex malware including user-oriented and targeted attacks. These attacks stem from users’ misbehavior, e.g. misinterpreting or ignoring security alerts, which lead to proliferation of malicious objects inside trusted perimeter of cyber-security systems (e.g. exclusion list of AVs). The attack mechanisms include strategic web compromise, spear phishing, insider threat and social network malware. Moreover, targeted attacks often deliver zero-day malware that is made difficult to be detected, e.g. due to distributed malicious payload. The system provides a secure container enabling user-dependent environment in malicious activity analysis, which is achieved by user interaction simulation in real time. The user interaction simulator recognizes GUI components and clicks through them according to click patterns of a typical user, e.g. office employee. To provide effective malware detection, our team developed a new technology for deep dynamic inspection of system-wide behavior, which is based on structural analysis of so-called activity trees defined in the domain of system functionalities. We use Modified Hierarchical Colored Petri Nets for run-time recognition of system functionalities including obfuscated and distributed ones. We our system with corpus of real malware families. Results show high efficiency of our system in detecting and blocking malware while having low system overhead.