Document exploit detection using baseline comparison

[0001] The present invention relates generally to detection of malware on a computer. More specifically, the present invention relates to detection of document exploits using baseline behavior.

[0002] Attacks upon computer systems are increasingly becoming more sophisticated and targeted. One particular type of threat, known as an advanced persistent threat (APT), refers to targeted attacks that aggressively pursue and compromise chosen targets, and is commonly associated with a government or other group that has the resources to maintain such an attack. Often, such a long-term pattern of attacks is aimed at other governments, companies, and political activists. Individuals (such as individual hackers) are usually not referred to as being an advanced persistent threat because they rarely have the resources to launch a sophisticated attack or be persistent.

[0003] An advanced persistent threat is characterized by: targeting a specific organization or individual; gaining a foothold; accessing the target network; deploying additional tools; and covering tracks in order to maintain future access. One common method of attack, and usually the first vector of an advanced persistent threat, is to exploit a vulnerability in an application program, typically through one of its documents, in order to cause harm. The vulnerability may be some type of flaw, error or poor coding technique in the application program that allows the attacker to exploit the program for a malicious purpose.

[0004] This so-called “document exploit” can affect many types of software applications and their corresponding documents. For example, standard computer document types such as Flash files, PDF files, Word documents, Excel documents, PowerPoint documents, RTF files, etc., can be exploited because of flaws in their corresponding application programs. For example, one family of malware modifies PDF files in order to exploit vulnerabilities in Adobe Acrobat and Adobe Reader by executing JavaScript code when the file is opened. The embedded JavaScript may contain malicious instructions to download and install other malware. A computer may become infected when the user visits a compromised Web site or opens the malicious PDF file. This family may exploit over a dozen known vulnerabilities.

[0005] Even lesser-known software applications can be the subject of a document exploit, such as the Korean proprietary word processing application Hangul and its HWP file types. Even file types that users would not normally create and that would seem above suspicion are at risk. For example, the help files (extension “.HLP”) in the Microsoft operating system are being used in targeted attacks because malware authors can use these files to call an operating system API for a malicious purpose.

[0006] A database of common vulnerabilities and exposures (the CVE database) keeps track of publicly known vulnerabilities using unique, common identifiers. For example, two of the most common vulnerabilities exploited by malware in Microsoft Word are CVE-2010-3333 and CVE-2012-0158. Not surprisingly, the methods used to exploit a Microsoft Word document (for example) will differ based upon the particular vulnerability chosen by a malicious program. Often, the payload delivered by the malware falls into particular categories such as launching another malicious process, crashing the computer, downloading another malicious file from the Web, or dropping a file from the original malware.

[0007] In addition to the vulnerabilities shown in the CVE database, many of the attacking methods used by a document exploit are well-known such as the stack-based buffer overflow attack, the heap spray attack, use of shell code, or invoking an unsafe method. Accordingly, and unfortunately, most if not all of the prior art detection techniques are based upon the known CVE database or based upon the known attacking methods. For example, static techniques based upon virus signatures only work for known document exploits; these techniques will not work for unknown exploits for which no signature yet exists. Emulation-based techniques have associated overhead, rarely open certain types of files, and often cannot monitor the real behavior of a document because of the emulation.

[0008] Other techniques such as private memory usage monitoring, NOP sled detection, string detection and null page allocation often are not helpful because they all attempt to detect the exploit known as “Heap Spray.” If no heap spray technique is used in the document, these techniques will not be helpful. And, protection techniques such as ASLR and DEP are not able to stop well-constructed exploits. For example, exploit techniques such as “Return Oriented Programming” and “Information Leak” are ways to bypass both of these protection techniques. Finally, the above detection techniques can be unsuccessful or at best, inefficient, in the case of a zero-day attack.

[0009] Accordingly, a new method to detect document exploits that is more efficient, that does not adversely impact system performance, and that is effective in the case of the zero-day attack is desirable.

[0010] To achieve the foregoing, and in accordance with the purpose of the present invention, a document exploit detection technique is disclosed that is able to detect unknown exploits such as zero-day attacks, does not rely upon static signatures, and has none of the overhead of emulation.

[0011] In a first embodiment, a pattern is created for detecting a document exploit. A document file known to include a document exploit is executed within its corresponding software application and its behaviors are monitored and recorded while a document file is opened and executes for a predetermined amount of time. Monitoring uses software hooks or internal drivers. Relevant information from the resulting behavior report is extracted in order to create a pattern file used to detect a document exploit. The pattern file may include regular expressions, text strings or another format. A document file known to be free of malware may also be monitored while it is opened in order to create a behavior report and resulting pattern file indicative of malware not being present.

[0012] In a second embodiment, a method of detecting a document exploit opens a suspicious document file in a software application corresponding to a particular malicious pattern that has already been created. The suspicious document file actually does not contain any malware. Behaviors of the suspicious document are monitored and recorded in a report file. The behaviors in the report file are compared to the malicious pattern, and weight values are assigned to behaviors in the report file that match with expressions in the malicious pattern. When the threshold is not reached, then an output indicates that the suspicious document file does not contain a document exploit. The report file may also be compared to a benign pattern that corresponds to a normal document free of malware.

[0013] In a third embodiment, a method of detecting a document exploit opens a suspicious document file in a software application corresponding to a particular malicious pattern that has already been created. The suspicious document file does contain malware. Behaviors of the suspicious document are monitored and recorded in a report file. The behaviors in the report file are compared to the malicious pattern, and weight values are assigned to behaviors in the report file that match with expressions in the malicious pattern. When the threshold is reached, then an output indicates that the suspicious document file does contain a document exploit.

[0023] In order to detect known or unknown document exploits, the present invention is able to record the baseline behavior of both known normal and known malicious documents being opened and executing within a corresponding software application. Both the normal and the malicious behavior can be extracted into baseline patterns which may then be compared to the behavior of a suspicious document that is opened and executes within the same corresponding software application. If any abnormalities are detected, then an alert may be raised that a document exploit has been detected.

[0024] FIG. 1 is a block diagram of a computer system 5 for implementing the present invention. The present invention may be implemented upon a variety of computers (including computer system 900) such as laptops, desktops, server computers, backend computers, user computers, etc. Executing upon a computer is any suitable operating system 10 upon which is executing one or more software applications 20. This software application may be, for example, an application such as Microsoft Word, Adobe Acrobat or Adobe Reader, Excel, PowerPoint, Adobe Flash Player, Hangul, or Ichitaro (a Japanese word processor produced by JustSystems).

[0025] The computer system may be executing only a single software application at a time, or may be executing numerous of these software applications. Known application documents 24 refer to those so-called “document” files that are opened and possibly executed by the corresponding software applications listed above. For example, computer document types such as Flash files, PDF files, Word documents, Excel documents, PowerPoint documents, .RTF files, .HLP files, .HWP files and .JTD files are all considered document files.

[0026] During operation of the invention, a given document will be opened and possibly executed by its corresponding software application; for example, a Microsoft Word file (“example.doc”), will be opened by its corresponding software application, namely, Microsoft Word. These known application documents 24 include those known to be free of malware (including any document exploits) so that their opening and execution will only produce normal results that would be expected of a document that is free of malware. Application documents 24 also include those known to include a document exploit, so that their opening and execution will produce malicious results that would be expected of a document that contains malware.

[0027] By contrast, a suspicious document file 28 is one of the computer document types listed above but it is unknown as to whether this document includes any malware such as a document exploit. This suspicious document 28 will be opened by its corresponding software application 20. If there is a document exploit present in suspicious document 28 it is likely that the malicious code will take advantage of a vulnerability in the corresponding software application in order to engage in malicious behavior. For example, a PDF file might contain malicious code that attacks a vulnerability in Adobe Reader, the software application corresponding to the PDF file. It is also possible that a hybrid document exploit is present in suspicious document 28, in which case the malicious code will not take advantage of its corresponding software application, but instead will attack a vulnerability of a different software application. For example, a Flash object may be embedded within an Excel document, a Word document, a PDF document or a Web page. Instead of attacking a vulnerability in Excel, Word, Adobe Reader or Safari, the Flash object will attack a vulnerability in Flash Player and perform malicious activity in that fashion. Of course, other types of hybrid document exploits are possible in which a software object corresponding to a particular software application is embedded within a document corresponding to a different software application (e.g., many types of documents are able to be embedded within a Microsoft Word document as software objects, even though they are not DOC files).

[0028] Monitor module 30 is a software module or modules present and executing upon a computer system 5 and may include any number of hook modules or hooking code, as well as internal software drivers used to monitor the behavior of an application document (either document 24 or document 28) when it is opened by its software application. Module 30 need not be a discrete module that includes all of the computer code used to monitor and record the behavior of the application 20 and a document that it opens. For example, various hook modules and hooking code inserted in various places in the operating system may be considered as part of the monitor module, as well as any internal drivers that are executing within the operating system.

[0029] In one specific embodiment, module 30 includes a number of discrete monitor modules that are responsible for capturing behaviors during execution time and include a file system behaviors module, a registry behaviors module, and a network behaviors module. These three modules may be implemented as internal drivers in order to capture the specific behaviors. For example, the file system behaviors module captures these behaviors: add file, write file and delete file. The registry behaviors module captures these behaviors: add key, write key and delete key. The network behaviors module captures these behaviors: DNS query and HTTP request. Of course, other behaviors may be captured as well.

[0030] In addition to these specific modules, many process-related behaviors are monitored such as: create Mutex, create process, delete file, drop executable, execute dropped file, modify file, sleep, etc. And a variety of hooking methods may be used to capture process behaviors. Some of the API functions that are hooked include: CopyFileEx, MoveFileWithProgress, CreateFile, GetFileAttributes, CreateDirectory, RemoveDirectory, LoadLibrary, LoadImage, GetProcAddress, IsDebuggerPresent, CheckRemoteDebuggerPresent, Process32Next, CreateThread, InternetGetConnectedStateEx, InternetOpen, InternetOpenUrl, HttpOpenRequest, InternetConnect and FtpOpenFile. Of course, other functions may be hooked as well.

[0031] Behavior report 40 is a report listing details of the executing software application, the document it has opened, and the behavior produced by that document (whether a known document or a suspicious document). Examples of behavior reports are presented below. Pattern creation module 50 is a software module used to create a baseline pattern from a behavior report for a known document. Its operation is described below. Baseline pattern database 60 is a database of patterns produced from the behavior of known documents that have been opened. For example, database 60 may contain baseline patterns for each of the software applications such as Microsoft Word, Adobe Reader, etc. For example, a known normal baseline pattern and a known malicious baseline pattern may be stored in database 60 for Adobe Reader.

[0032] Comparison module 70 is a software module used to compare one of the baseline patterns (representing normal or malicious behavior of a document) with a behavior report concerning the opening and execution of a suspicious document. Based upon this comparison, the computer system may output 80 that a document exploit has been detected or not.

[0033] Briefly, any number of known normal and known malicious sample documents are opened, executed and analyzed in the system to create behavior reports. Normal patterns are extracted from the behaviors of normal samples. Malicious patterns are extracted from the behaviors of malicious samples. Further, the malicious patterns may then be compared to the normal patterns to further refine the malicious patterns.

[0034] FIG. 2 is a flow diagram describing one embodiment by which a baseline pattern is created. In step 104 the behavior detection techniques are set up. Preferably, the behaviors that are monitored are registry behaviors, file system behaviors, process behaviors and network behaviors. Examples of behaviors are presented above and below.

[0035] These behaviors may be detected in different manners. In one specific embodiment, hook modules and hooking code are used to detect process behaviors, while internal software drivers are used to detect registry, file system and network behaviors. Internal drivers are processes internal to the operating system that may be written specifically to detect these behaviors.

[0036] Once these the detection techniques are in place, then in step 108 any such processes are executed (such as any internal drivers used) to begin monitoring the behavior of a document when opened. In step 112 the software application 20 under consideration begins execution. It is possible that other software applications are also executing at the same time, although preferably only a single software application is executing. In step 116 a known document corresponding to the executing software application is opened; this document is either known to be free of any malware including document exploits, or is known to include document exploits. Steps 108-116 may be executed manually by a user in the customary fashion (i.e., double-clicking on Microsoft Word and then double-clicking on a particular document), although it is preferable to use a command line to perform these actions. For example, first the Console application (or Terminal application) of the operating system is started and then a command line is used to launch these processes and to open a particular document.

[0037] In step 120 the monitoring software described above collects all behaviors of the document that has been opened and generates a report. These behaviors may include actions taken with respect to the opening of the document, and any other actions thereafter, including code that may be executed by virtue of the document being opened. Example of generated reports are shown in FIG. 4 (normal) and FIG. 5 (malicious).

[0038] When the computer system has recorded enough information concerning the behaviors of the opened document and its corresponding software application, then in step 124 the document is closed and the report is complete. The document may be closed after any suitable time; in one embodiment, a document is kept open for about 30 seconds while its behaviors are monitored and recorded. Of course, a document may be kept open for less time or for more time. Preferably, the document is also closed using a command line. Because the goal is to observe behavior and create a baseline pattern that is repeatable, preferably the user of the software application does not interact with the software application or the open document, and does not perform any operations with the document while open.

[0039] In another embodiment, it is possible to open and record the behaviors of more than a single document (either normal or malicious documents) corresponding to the executing software application. If so, the reports generated from each of these documents may be combined to create a single pattern. For example, since in the organization of a report the behaviors are organized by processes, the behaviors belonging to different sample documents will be categorized as children nodes of the parent processes of the report.

[0040] Once the document is closed then in step 128 a baseline pattern is created for the opened document and its executing software application using the report generated in step 120. The pattern is created by extracting useful information from the report. Useful information may be extracted and a pattern maybe created in different ways. In one particular embodiment, the pattern is a regular expression or a series of regular expressions used to match text and information strings in future behavior reports that are generated from suspicious documents 28. Examples of created patterns are shown in FIG. 6 (normal) and in FIG. 7 (malicious). Once created, the pattern is stored in database 60 for later comparison. In addition to regular expressions, keywords may also used as patterns, and are simply plain text.

[0041] In addition, as mentioned above, a pattern created from a known malicious document may be further refined by comparison with a pattern created from a known benign document. For example, it is possible that the regular expressions created for a malicious pattern may actually include a regular expression that matches with a benign behavior. Considering the regular expression 624 of FIG. 6 that matches with creation of a message box, this regular expression 624 may be removed from any malicious pattern because creation of a message box is relatively benign. In an alternative embodiment, benign regular expressions that are found within a malicious pattern may simply be assigned a weight value of 0.

[0042] FIG. 3 is a flow diagram describing one embodiment by which a document exploit may be detected. Steps 204-212 may be performed in a similar fashion as steps 104-112, either as a session occurring later on the computer system or immediately subsequent to execution of the steps of FIG. 2. For example, once a baseline pattern or patterns have been created, a suspicious document 28 may be opened by the already executing software application 20 and it would not be necessary to perform steps 204-212 because steps 104-112 have just been performed. In another example, FIG. 3 is performed after the computer system has been shut down and restarted, in which case step 204-212 will be executed.

[0043] In step 216 a suspicious document 28 is opened by its corresponding software application 20, the same software application used previously to open the known document. For example, if a known normal pattern and a known malicious pattern have been created for the Excel application, then a suspicious Excel document 28 may be opened and compared. The suspicious document may be opened and monitored in a similar manner as described above for known documents. If the suspicious document does in fact contain a document exploit (or other type of malware) then malicious code within the suspicious document may take advantage of a vulnerability in the executing software application 20. If a hybrid document exploit is present, then the malicious code may take advantage of a vulnerability in a software application other than executing application 20.

[0044] In step 220 a report is generated for this suspicious document, detailing its processes started and details of behaviors for those processes. FIG. 5 shows an example of a report generated for execution of a suspicious document, which in this case, does contain malware.

[0045] In step 224 the report for the suspicious document is compared to a baseline pattern created in FIG. 2 for the same corresponding software application. In one embodiment, the suspicious report is compared to the malicious pattern first. If no exploit is detected, then the suspicious report is compared to the normal pattern.

[0046] Preferably, the comparison is handled automatically using any of a variety of computer algorithms. For example, a string search algorithm or a regular expression match algorithm may be used to compare the pattern to the report and to find similarities and differences. Or, both the baseline pattern and the suspicious report may be translated into the same internal format (such as Apilog format, an internal message format for logging behaviors) in order to compare the two more easily.

[0047] Weights are assigned to specific behaviors that match with specific text strings, regular expressions or rules of the baseline pattern that are indicative of malware. All of the weights are then summed to provide a final weight which is compared against a threshold number. A final weight greater than a threshold indicates that it is likely that the suspicious document includes a document exploit, while a final weight less than the threshold indicates the suspicious document is likely free of malware. In one particular embodiment, when each regular expression is matched (either a regular expression of FIG. 6 indicating a benign document or a regular expression of FIG. 7 indicating a malicious document), a weight value will be added to a running total. If the final weight total exceeds a threshold, then it is determined that malicious sample is detected.

[0048] For example, if a regular expression of FIG. 7 representing “drops an executable file” matches with a portion of the behavior report for a suspicious document then a weight of “2” is added to the running total. Other regular expressions of that pattern are also attempted to be matched. If the final total weight value exceeds 10 then a determination is made that the suspicious document is malicious. In one embodiment, if a regular expression in the normal pattern of FIG. 6 matches a portion of the behavior report for a suspicious document then a weight of 0 or a negative value may be added to the running total. In other embodiments, matching a regular expression in a normal pattern does not add any weight value to the running total, and only matching a regular expression in the malicious pattern adds a positive value to the running total.

[0049] In step 228 a result from this comparison is output, such as by generating an alert on the user computer, printing a result on paper, writing to a computer file, sending a message over a network, etc. The result indicates whether or not the suspicious document includes a document exploit or not.

[0050] As can be seen in FIG. 4, a report typically contains three parts. A first part 410 includes general information about this document and the generation of this report, a second part 420 (the call tree) provide information on processes running, and a third part 430 provides more detail on the behaviors of these processes. Of course, a report detailing the opening and execution of a particular document may take many forms; in this specific embodiment the form is that of a log file that provides very detailed information. The report of FIG. 4 has been greatly redacted in order to show exemplary information in a single drawing.

[0051] As shown in the first part 410 general information includes names of any antivirus software running, the name of the opened document, the type of document, the time when the document is opened and closed and any the decision made after the analysis process.

[0052] As shown in the second part 420, the call tree lists each of the various processes that have been started, each process identifier, the relevant file name, a start reason and a termination reason. The call tree may also describe which processes are subordinate to others. In this example, shown are first process 422 and a second process 424; information concerning a third process has been omitted for clarity.

[0053] As shown in the third part 430, shown are behaviors for each particular process organized by behaviors. For example, for the first process identified at 432, the behaviors listed are registry behaviors 434, file behaviors 436, file system behaviors 438, system behaviors 440, Mutex behaviors 442 and window behaviors 444. Many behaviors in each category have been redacted for clarity. In addition, behaviors for the second and third processes are not shown. This detail provides baseline information describing what are normal behaviors for a normal document free of malware.

[0054] FIG. 5 shows a behavior report for a document deemed to be malicious. As mentioned above, behaviors in this report are compared with a previously created malicious pattern, and weight values are assigned to matched regular expressions, leading to a conclusion of malicious if the final total is over a threshold.

[0055] A first part 510 includes general information about this document and the generation of this report, a second part 520 (the call tree) provide information on processes running, and a third part 530 provides more detail on the behaviors of these processes. In this specific embodiment, the form of the report is that of a log file that provides very detailed information. The report of FIG. 5 has been greatly redacted in order to show exemplary information in a single drawing.

[0056] As shown in the second part 520, the call tree lists each of the various processes that have been started, each process identifier, the relevant file name, a start reason and a termination reason. The call tree may also describe which processes are subordinate to others. In this example, shown are first, second, third, fourth and fifth processes.

[0057] As shown in the third part 530, shown are behaviors for each particular process organized by behaviors. For example, for the third process, the behaviors listed are registry behaviors 534, file behaviors 536, file system behaviors 538, system behaviors 540, Mutex behaviors 542, process behaviors 544, service behaviors 546, and network behaviors 548. Many behaviors in each category have been redacted for clarity. In addition, behaviors for the first, second, fourth and fifth processes are not shown. This detail provides information describing behaviors for a malicious document that contains malware such as a document exploit. Note that this reports also list behaviors under the “process,” “service,” and “network” categories which are not present in the normal behavior report of FIG. 4. In this example, the malicious pattern extracted from this report covered behaviors involving the registry, file system, processes, services and network. In most situations, the malware performed malicious operations in these areas.

[0058] FIG. 6 shows a normal pattern created from a normal behavior report. Shown are any number of regular expressions. These regular expressions were extracted from a number of normal behaviors and may be regarded as a baseline for comparison. In this example, expression 604 matches adding a registry; expression 608 matches modifying a registry; expression 612 matches adding a file; expression 616 matches modifying a file; expression 620 matches creating a Mutex; expression 624 matches creating a message box; expression 628 matches creating a mail slot; expression 632 matches creating an event; and expression 636 matches creating a process. As known in the art, a regular expression is a concise, accurate way to match text and information strings in a document.

[0059] FIG. 7 shows a malicious pattern created from a malicious behavior report. Shown are patterns organized as regular expressions. The patterns were extracted from a number of malicious behaviors. In this example, expression 704 matches dropping an executable; expression 708 matches dropping a file; expression 712 matches downloading an executable; expression 716 matches establishing a network connection and has two parts; expression 720 matches requesting a URL and has two parts; expression 724 matches a process residing in memory; expression 728 matches causing a document reader to crash; expression 732 matches using a heap spray technique to execute code; and expression 736 matches attempting to use a document exploit.

[0060] FIGS. 8A and 8B illustrate a computer system 900 suitable for implementing embodiments of the present invention. FIG. 8A shows one possible physical form of the computer system. Of course, the computer system may have many physical forms including an integrated circuit, a printed circuit board, a small handheld device (such as a mobile telephone or PDA), a personal computer or a super computer. Computer system 900 includes a monitor 902, a display 904, a housing 906, a disk drive 908, a keyboard 910 and a mouse 912. Disk 914 is a computer-readable medium used to transfer data to and from computer system 900.

[0061] FIG. 8B is an example of a block diagram for computer system 900. Attached to system bus 920 are a wide variety of subsystems. Processor(s) 922 (also referred to as central processing units, or CPUs) are coupled to storage devices including memory 924. Memory 924 includes random access memory (RAM) and read-only memory (ROM). As is well known in the art, ROM acts to transfer data and instructions uni-directionally to the CPU and RAM is used typically to transfer data and instructions in a bi-directional manner Both of these types of memories may include any suitable of the computer-readable media described below. A fixed disk 926 is also coupled bi-directionally to CPU 922; it provides additional data storage capacity and may also include any of the computer-readable media described below. Fixed disk 926 may be used to store programs, data and the like and is typically a secondary storage medium (such as a hard disk) that is slower than primary storage. It will be appreciated that the information retained within fixed disk 926, may, in appropriate cases, be incorporated in standard fashion as virtual memory in memory 924. Removable disk 914 may take the form of any of the computer-readable media described below.

[0062] CPU 922 is also coupled to a variety of input/output devices such as display 904, keyboard 910, mouse 912 and speakers 930. In general, an input/output device may be any of: video displays, track balls, mice, keyboards, microphones, touch-sensitive displays, transducer card readers, magnetic or paper tape readers, tablets, styluses, voice or handwriting recognizers, biometrics readers, or other computers. CPU 922 optionally may be coupled to another computer or telecommunications network using network interface 940. With such a network interface, it is contemplated that the CPU might receive information from the network, or might output information to the network in the course of performing the above-described method steps. Furthermore, method embodiments of the present invention may execute solely upon CPU 922 or may execute over a network such as the Internet in conjunction with a remote CPU that shares a portion of the processing.

[0063] In addition, embodiments of the present invention further relate to computer storage products with a computer-readable medium that have computer code thereon for performing various computer-implemented operations. The media and computer code may be those specially designed and constructed for the purposes of the present invention, or they may be of the kind well known and available to those having skill in the computer software arts. Examples of computer-readable media include, but are not limited to: magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD-ROMs and holographic devices; magneto-optical media such as floptical disks; and hardware devices that are specially configured to store and execute program code, such as application-specific integrated circuits (ASICs), programmable logic devices (PLDs) and ROM and RAM devices. Examples of computer code include machine code, such as produced by a compiler, and files containing higher-level code that are executed by a computer using an interpreter.

[0064] Although the foregoing invention has been described in some detail for purposes of clarity of understanding, it will be apparent that certain changes and modifications may be practiced within the scope of the appended claims. Therefore, the described embodiments should be taken as illustrative and not restrictive, and the invention should not be limited to the details given herein but should be defined by the following claims and their full scope of equivalents.