Detecting encryption functions via process emulation and IL-based program analysis

Abstract

Malware often encrypts its malicious code and sensitive data to avoid static pattern detection, thus detecting encryption functions and extracting the encryption keys in a malware can be very useful in security analysis. However, it's a complicated process to automatically detect encryption functions among huge amount of binary code, and the main challenge is to keep high efficiency and accuracy at the same time. In this paper we propose an enhanced detection approach. First we designed a novel process level emulation technique to efficiently analyze binary code, which is less resource-consuming compared with full system emulation. Further, we conduct program partitioning and assembly-to-IL(intermediate language) translation on binary code to simplify the analysis. We applied our approach to sample programs using cryptographic libraries and custom implemented version of typical encryption algorithms, and showed that these routines can be detected efficiently. It is convenient for analysts to use our approach to deal with the encrypted data within malware automatically. Our approach also provides an extensible interface for analysts to add extra templates to detect other forms of functions besides encryption routines. © 2012 Springer-Verlag.