Extended abstract - Detection and classification of malware based on symbolic execution and machine learning methods

Bertrand Van Ouytsel, Charles-Henry [UCL] Legay, Axel [UCL]

These last years, the number of malware detected by antivirus always continues to grow drastically. Most of these malware are just variants of previously observed malware. Actually, malware authors just tweak them in such a way that previous detection methods are bypassed. For this purpose, they continuously create new obfuscation/evasion techniques. Thus efficient malware detection stays a big challenge for all security analysts. We present a new technique for detecting malware. The approach represents all the behaviors of an instance of the malware in the form of an object obtained by symbolic analysis: namely a system call dependency graph (SCDG). Next, we use learning techniques to extract the signature of a malware family from a set of known instances and use this to predict the shape of new instances of malwares of the same family. On one hand, symbolic analysis avoids the inherent problems of dynamic analysis: thanks to symbolic analysis, all execution paths can be explored and evasion methods can be accounted for. On the other hand, learning avoids the problem of rigidity and obfuscation of the signature encountered with static analysis. This technique is currently implemented in a toolchain allowing us to extract SCDGs from several malware types and to build a signature thanks to the Gspan algorithm. Using symbolic analysis in this context leads to several challenges such as path explosion or environment modeling.