Anti-malware emulation systems and methods

[0001] The invention relates to systems and methods for detecting viruses and other malware, and in particular to anti-virus detection systems using behavior-based heuristics.

[0002] Known methods of scanning for malicious software (malware) such as viruses, worms, and Trojan horses include content-based methods and behavior-based methods. In content-based methods, the contents of a suspected file are compared to a database of known malware signatures. If a known virus signature is found in the suspected file, the file is labeled as a virus. Content-based methods are generally effective at identifying known malware, but may not be able to identify new threats. Also, malware writers often employ countermeasures aimed at evading signature scanners. For example, polymorphic viruses mutate periodically in order to make their identification more difficult.

[0003] Behavior-based methods typically involve allowing a suspected program to execute in an isolated virtual environment, commonly called a sandbox, and observing the program's resulting behavior. Programs that exhibit malicious behavior are identified and removed or contained. While conventional behavior-based methods may be effective in identifying new threats, such methods may be relatively computationally intensive and require relatively long analysis times.

[0004] According to one aspect, a malware detection method comprises performing an accelerated emulation of an untrusted computer file by emulating a first code sequence of the untrusted computer file in a virtual environment; and, upon identifying a known code sequence in the untrusted computer file, executing a native-code routine functionally implementing the known code sequence. A result of the accelerated emulation is used to determine whether the untrusted computer file is malicious.

[0005] According to another aspect, a malware detection method comprises: upon identifying a known code sequence in an untrusted computer file, executing a native code routine functionally implementing the known code sequence; employing a result of said executing the native code routine to emulate a second code sequence of the untrusted computer file; and determining whether the untrusted computer file is malicious according to a result of emulating the second code sequence.

[0006] According to another aspect, a malware detection method comprises determining whether a part of an untrusted computer file matches a known code signature; and when the part matches the known code signature, natively executing a native code routine functionally implementing the part, and when the part does not match the known code signature, emulating the part in a virtual machine.

[0007] According to another aspect, a malware detection method comprises emulating a first code sequence of an untrusted computer file in a virtual environment; upon identifying a virtual environment operating system call generated by said emulating the first code sequence, executing a native code routine functionally implementing the virtual environment system call; employing a result of said executing the native-code routine to emulate a second code sequence of the untrusted computer file; and determining whether the untrusted computer file is malicious according to a result of emulating the second code sequence.

[0008] According to another aspect, a malware detection method comprises emulating an unpacking of an untrusted computer file in a virtual environment to generate an unpacked content in a virtual memory of the virtual environment; executing a native code file reconstruction routine to assemble a reconstructed file on a real host of the virtual environment from the unpacked content in the virtual memory; and examining a content of the reconstructed file to determine whether the untrusted computer file is malicious.

[0009] According to another aspect, a malware detection system comprises an emulator configured to emulate an untrusted computer file in a virtual environment, and to recognize a known code sequence in the untrusted computer file; and a native-code acceleration routine responsive to the emulator and functionally implementing the known code sequence. The emulator calls the native-code acceleration routine upon recognizing the known code sequence, receives a result of an execution of the native-code acceleration routine, and employs the result of the execution to continue an emulation of the untrusted computer file.

[0015] In the following description, it is understood that all recited connections between structures can be direct operative connections or indirect operative connections through intermediary structures. A set of elements includes one or more elements. Any recitation of an element is understood to refer to at least one element. Unless otherwise required, any described method steps need not be necessarily performed in a particular illustrated order. A first element (e.g. data) derived from a second element encompasses a first element equal to the second element, as well as a first element generated by processing the second element and optionally other data. Making a determination or decision according to a parameter encompasses making the determination or decision according to the parameter and optionally according to other data. Unless otherwise specified, an indicator of some quantity/data may be the quantity/data itself, or an indicator different from the quantity/data itself. The term “native code” refers to code running on a real host machine, and is distinct from code running on a virtual machine defined by an emulator. Computer readable media encompass storage media such as magnetic, optic, and semiconductor media (e.g. hard drives, optical disks, flash memory, DRAM), as well as communications links such as conductive cables and fiber optic links. According to some embodiments, the present invention provides, inter alia, computer systems programmed to perform the methods described herein, as well as computer-readable media encoding instructions to perform the methods described herein.

[0016] The following description illustrates embodiments of the invention by way of example and not necessarily by way of limitation.

[0017] FIG. 1 shows an anti-malware detection/protection system 20 according to some embodiments of the present invention. System 20 includes a scanning engine 26, a protection/notification module 30 connected to scanning engine 26, and an update module 32 connected to scanning engine 26. Scanning engine 26 includes a content analyzer 34 and a behavior analyzer 36.

[0018] Analyzers 34, 36 receive one or more untrusted target files 24, analyze the files, and generate a set of scan results 38. Target files 24 may include contents of an email or other electronic messages, and/or contents of one or more drives or folders to be scanned for malware. Target files 24 may include executable (e.g. portable executable, .exe., .cmd), batch (.bat), archive (e.g. .zip, .tar), script (e.g. Javascript, Visual Basic Script), and other content capable of malicious functionality. Content analyzer 34 analyzes a content of target files 24, and behavior analyzer 36 analyzes a behavior of target files 24. Scan results 38 include indicators of whether target files 24 are malicious and, for malicious files, the type of threat posed by target files 24. Protection/notification module 30 uses scan results 38 to take appropriate protection and notification steps if target files 24 are determined to be malicious. Protective actions may include removing and/or quarantining malicious files, while notification actions may include notifying a user of the type of malicious content that has been identified. Update module 32 may periodically update a set of content signatures used by content analyzer 34 and/or a set of behavior evaluation parameters used by behavior analyzer 36.

[0019] In some embodiments, anti-malware protection system 20 may be provided on end-user client computers running an operating system such as Windows, Macintosh or Linux. Anti-malware protection system 20 may be provided as part of a security suite. In some embodiments, anti-malware protection system 20 may be provided on a mail server or other external server. A malware protection system provided on a server may be used to identify malware present in emails or other messages sent to multiple clients, and to take appropriate action (e.g. remove or quarantine) before the messages are delivered to the clients.

[0020] FIG. 2 is a diagram of behavior analyzer 36 according to some embodiments of the present invention. Behavior analyzer 36 receives a target file 24 and analyzes the behavior of target file 24 to generate a set of scanning results/malware indicators 38 indicating whether target file 24 is malicious, and a type of threat posed by target file 24. Behavior analyzer 36 includes a simulation engine 50, and a plurality of native-code routines connected to simulation engine 50: a set of native-code emulation-acceleration routines 58, a set of native-code unpacked file reconstruction routines 52, a set of native-code virtual system call routines 54, and a set of content inspection modules 62, 64, 68. The set of content inspection modules includes a virtual machine content inspector 62, a memory-based pattern matching module 64, and a file-based pattern-matching module 68.

[0021] Simulation engine 50 forms a virtual host computer for target file 24. Simulation engine 50 may be thought of as a sandbox, in which target file 24 is allowed to play and possibly harm the virtual computer created by simulation engine 50, without harming the real computer hosting simulation engine 50. Simulation engine 50 includes a system emulator 82 and a virtual hardware emulator 86 connected to system emulator 82. System emulator 82 includes a processor emulator 92 and a virtual memory controlled by a virtual memory manager 94. Virtual hardware emulator 86 includes a virtual hard drive 96 and other virtual hardware 98. Virtual hardware 98 may include a virtual display, keyboard, mouse and floppy drive. In some embodiments, system emulator 82 is configured to emulate an IA-32 (x86-32) instruction-architecture, protected-mode microprocessor and associated memory. In protected mode, different programs are generally prevented from corrupting one another. In some embodiments, system emulator 82 may be configured to emulate a real-mode microprocessor or a processor using other instruction architectures.

[0022] Each acceleration routine 58 implements in native code the algorithmic functionality of a corresponding known, safe code sequence encountered by simulation engine 50, or of a class of known code sequences having identical functionality. In general, the corresponding native-code routine's machine code need not be identical to the machine code of the known code sequence processed by the emulator, although the machine code sequences may be identical. In some embodiments, acceleration routines 58 include routines that are commonly encountered in malware scans and whose emulation is relatively computationally-intensive and time-consuming. In some embodiments, the set of acceleration routines 58 includes decompression, decryption, and/or checksum routines.

[0023] Each native-code unpacked file reconstruction routine 52 reconstructs a real file from the contents of the virtual memory of system emulator 82 following an unpacking of the file by system emulator 82. When system emulator 82 has unpacked a file in virtual memory, system emulator 82 calls an unpacked-file reconstruction routine 52, which receives the unpacked data from the virtual memory and assembles the data into a file for scanning by file-based pattern scanning module 68. In some embodiments, when system emulator 82 processes a packed file, system emulator 82 proceeds through a set of unpacking instructions. The original entry point of the packed file is a target of a code flow change. When reaching the original entry point of the packed file, system emulator 82 checks for a signature as described below, and upon finding a signature match calls an unpacked file reconstruction routine 52.

[0024] Each virtual system call routine 54 implements in native code a corresponding virtual operating system call of system emulator 82, or part of a virtual operating system call. For example, the set of virtual system call routines 54 may implement in native code the functions of a set of dynamic link libraries (DLLs) employed by system emulator 82. Upon encountering a recognized system call during emulation, system emulator 82 calls a corresponding native-code virtual system call routine 54, and receives the results of the virtual system call from the called native-code routine 54, rather than by emulating the virtual system call.

[0025] System emulator 82 is capable of accessing, (reading and updating) a known safe code signature database 88 maintained on the real host computer, as described in detail below. Each entry in safe-code signature database 88 includes a safe-code signature uniquely identifying a corresponding safe code sequence, and an identifier of a corresponding routine selected from acceleration routines 58 and unpacked file reconstruction routines 52.

[0026] FIG. 3 shows an exemplary safe code signature 300 according to some embodiments of the present invention. Signature 300 represents a sequence of bytes stored in database 88. Signature 300 identifies a known safe code area, and is searched by system emulator 82 to determine whether to call a native-code routine (an acceleration routine 58 or file reconstruction routine 52 (FIG. 2)). Signature 300 includes a hash zone 310 beginning at a starting address 320, a pre-hash zone 312 immediately preceding hash zone 310, and a post-hash zone 318 immediately subsequent to hash zone 310. Starting address 320 is an address following a code flow change—e.g. the target of a jump or call instruction. Hash zone 310 may have a predetermined extent, e.g. 16 bytes, for all signatures. Hash zone 310 may form an index of the database entry of safe code signature 300.

[0027] In some embodiments, system emulator 82 compares a safe-code signature database entry to the data at a current emulation address in two steps: in a first step, a predetermined number of bytes (e.g. 16 bytes) in virtual memory starting at the current address is processed by a predetermined hash function and checked against all signature hashes stored in safe code signature database 88. If no match is found, the search process stops. If a hash match is found, system emulator 82 compares the pre-hash and post-hash zones of the matching signature(s) to corresponding bitmasked pre-hash and post-hash zones preceding and subsequent to the current address in virtual memory. System emulator 82 retrieves the sizes of pre-hash zone 310 and post-hash zone 318 from database 88, as well a bitmask for each zone. The bitmask of each zone defines a subset of bits to be used for comparison. For example, address fields and/or other fields may not be useful or necessary to uniquely identify a given machine code sequence, and such fields may be masked out. A suitable bitmask for a given signature may be chosen by a system designer. System emulator 82 applies the bitmasks to corresponding regions preceding and following the hash-zone in virtual memory, and compares the bitmasked regions to the corresponding pre-hash and post-hash signature zones in database 88.

[0028] System emulator 82 is also capable of accessing (reading and updating) a safe-code virtual address cache 90 (FIG. 2) maintained on the real host computer. Each entry in safe-code address cache 90 includes a safe-code virtual memory address as well as an identifier of a corresponding routine selected from acceleration routines 58 and file reconstruction routines 52. Safe-code address cache 90 may be configured in a structure optimized for fast searching using the address field as a search key. In some embodiments, safe-code address cache 90 may include a hash table, with each table entry formed by an address, and each table index formed by a hash value generated by applying a hash function to the address. In some embodiments, safe-code address cache 90 may be configured as a binary tree such as a self-balancing binary search tree. An exemplary self-balancing binary search tree is an AVL (Adelson-Velsky-Landis) tree. In some embodiments, safe-code address cache may also store addresses corresponding to virtual system calls to be handled by virtual system call routines 54, as well as identifiers of the corresponding virtual system call routines 54.

[0029] As simulation engine 50 moves through a sequence of addresses holding the data of target file 24, system emulator 82 searches safe code signature database 88 for matches to the data of target file 24. When a safe code signature match is found, the current address is added to safe code address cache 90, and a pre-stored native code acceleration routine 58 or unpacked file reconstruction routine 52 is executed. If an address in acceleration address cache 90 is accessed again during the emulation of target file 24, simulation engine 50 may proceed directly to retrieving and natively executing the corresponding routine, without re-searching for a signature match in database 88.

[0030] FIG. 4 shows a sequence of steps performed by simulation engine 50 in analyzing target file data at a current virtual memory address according to some embodiments of the present invention. In a step 200, system emulator 82 (FIG. 2) selects a current memory address for emulation. The current memory address holds a part of target file 24, which may include machine code instruction(s) and/or operand(s). Generally, emulation starts at the beginning of target file 24 and proceeds as described below.

[0031] In a step 202, system emulator 82 determines whether the current address corresponds to a recognized virtual operating system call. If yes, in a step 226, system emulator 82 calls a corresponding native-code virtual system call routine 54 and does not emulate the virtual system call. System emulator 82 receives the results of the virtual system call from the called routine 54, and selects a new current memory address for emulation (step 200).

[0032] In a step 204, system emulator 82 checks whether the current address is present in safe code address cache 90 (FIG. 2). If the current address is present in cache 90, system emulator 82 identifies a native-code acceleration or file reconstruction routine corresponding to the current address and calls that routine (step 224). If the current address is not present in cache 90, system emulator 82 scans the contents of the current address and/or neighboring addresses for matches to acceleration signatures stored in safe code signature database 88 (FIG. 2) (step 208), and determines whether a signature match is found as described above (step 212). If no signature match is found, system emulator 82 continues emulating instructions until a code flow change (step 216). For example, a code flow change occurs upon a jump (conditional or unconditional), call, or interrupt instruction. In some embodiments, system emulator 82 performs step 204 selectively only after each code flow change. If a matching signature is found, system emulator 82 adds the current address to safe code address cache 90, together with an identifier of a native code routine corresponding to the signature (step 220), and calls the corresponding native code routine (step 224).

[0033] If the native code routine is a file reconstruction routine 52 (FIG. 2), system emulator 82 continues execution and does not wait for results from the file reconstruction routine 52. The selected file reconstruction routine 52 reconstructs a real file from the virtual memory contents received from system emulator 82, and sends the resulting file to file-based pattern-matching module 68. File-based pattern-matching module 68 performs a content-based scan on the received data and generates a scan result/malware indicator 38.

[0034] If the native code routine is an acceleration routine 58 (FIG. 2), system emulator 82 suspends execution until acceleration routine 58 returns its results and a new current address for system emulator 82. The called routine executes a sequence of pre-defined native code instructions. Upon finishing execution, the called acceleration routine 58 returns to system emulator 82 the results of its execution and a new current address for system emulator 82. System emulator 82 uses the results returned by the acceleration routine to update its virtual registers and memory.

[0035] Memory-based pattern matching module 62 examines the contents of the virtual memory controlled by virtual memory manager 94 as simulation engine 50 proceeds, and generates a scan result/malware indicator 38. In some embodiments, memory-based pattern matching module 62 examines the virtual memory contents upon every code flow change, every time a virtual process terminates within system emulator 82, and at the end of the emulation process for target file 24.

[0036] Virtual machine content inspector module 64 examines the contents of virtual hard drive 96 and the virtual memory controlled by virtual memory manager 94 when the emulation of target file 24 is completed. Content inspector module 64 performs a behavior-based detection of maliciousness by determining whether certain behaviors have occurred during emulation. For example, target file 24 may be deemed malicious if any executable dummy file (goat file) on virtual hard drive 96 is modified, or if the memory contents of a specific process (e.g. “explorer.exe”) have been modified by another process.

[0037] When the emulation of target file 24 is completed, file-based pattern matching module 68 examines the contents of virtual hard drive 96 and any other virtual storage devices of virtual hardware emulator 86 for malware signatures, and generates a scan result/malware indicator 38.

[0038] The exemplary systems and methods described above allow accelerating the performance of behavior-based malware scanners employing emulators. Selected known code sequences encountered by the emulator are executed natively using pre-tailored native-code routines functionally implementing the code sequences, rather than emulated. Executing native code may be significantly faster than emulation in a virtual environment. A system designer may manually choose a number of native-code routines (e.g. on the order of tens of routines) that are sufficiently commonly encountered and that are computationally-intensive to emulate. The native-code routines are then placed in a database and searched periodically during the emulation process. According to some embodiments, pre-defined native-code routines are also used to implement virtual environment operating system calls and to reconstruct unpacked files from the contents of the emulator's virtual memory.

[0039] It will be clear to one skilled in the art that the above embodiments may be altered in many ways without departing from the scope of the invention. Accordingly, the scope of the invention should be determined by the following claims and their legal equivalents.