We present a method for detecting new malicious executables, which comprise the following steps: (a) in an offline training phase, finding a set of (not necessary consecutive) system call sequences that are characteristic only to malicious files, when such malicious files are executed, and storing said sequences in a database, (b) in a real time detection phase, for each running executable, continuously monitoring its issued system calls and comparing with the stored sequences of system calls within the database to determine whether there exists a match between a portion of the sequence of the run-time system calls and one or more of the database sequences, and when such a match is found, declaring said executable as malicious. We have evaluated our method and the preliminary results are promising and justify the use of system calls sequences for the purpose of detection of new malicious executables.
Rozenberg, B., Gudes, E., Elovici, Y., & Fledel, Y. (2011). A method for detecting unknown malicious executables. In Proc. 10th IEEE Int. Conf. on Trust, Security and Privacy in Computing and Communications, TrustCom 2011, 8th IEEE Int. Conf. on Embedded Software and Systems, ICESS 2011, 6th Int. Conf. on FCST 2011 (pp. 190–196). https://doi.org/10.1109/TrustCom.2011.27