A three-way decision making approach to malware analysis

Abstract

Malware analysis techniques generally classify software behaviors as malicious (i.e., harmful) or benign (i.e., not harmful). Due to ambiguous nature of application behavior, there are cases where it may not be possible to confidently reach two-way conclusions. This may result in higher classification errors which in turn affect users trust on malware analysis outcomes. In this paper, we investigate a three-way decision making approach based on probabilistic rough set models, such as, information-theoretic rough sets and game-theoretic rough sets, for malware analysis. The essential idea is to add a third option of deferment or delaying a decision whenever the available information is not sufficient to reach certain conclusions. We demonstrate the applicability of the proposed approach with an example from system call sequences of a vulnerable Linux application.