VMM-BASED INTRUSION DETECTION SYSTEM

[0001] This application is a non-provisional application of provisional application Ser. No. 60/063,296 filed Feb. 1, 2008 and entitled “Intrusion Detection System Using Virtualization-Based Profiling And Pattern Classification Algorithms,” and provisional application Ser. No. 61/147,913 filed Jan. 28, 2009 and entitled “VMM-Based HIDS,” the entire contents of each of which are hereby incorporated by reference for all purposes.

[0002] This invention relates generally to providing intrusion detection by monitoring events in a virtualized environment.

[0003] Zero day attacks and hidden malware are threats to computer users. Malicious software can degrade the performance of computing systems, leak sensitive information, and disable entire computing infrastructures. Information security is a major concern for any computer-based commercial or government entity that deals with online information. A 2007 report by the US Government Office of Accountability documents that cybercrime (computer crime, identity theft and phishing) cost the U.S. economy $117.5B in 2006.

[0004] All industries are susceptible to cybercrime. Some of the most susceptible markets are: financial institutions, online retailers, credit card companies, and data repositories. Most commercial IT organizations employ a first-line of defense such as anti-virus and firewall software. To date, however, these widespread security measures have proven to be ineffective in guarding against these types of intrusions because these solutions can only thwart known attacks, i.e., ones that have been seen before or ones that may have already done harm. Anti-virus and firewall software also require continual updates of their signature databases and configuration information, and they provide no defense against zero-day attacks (i.e., new classes of attacks).

[0005] An alternative approach is to utilize an Intrusion Detection System (IDS), and specifically a Host-based Intrusion Detection Systems (HIDS). These systems look for anomalous behavior on a computing system, tracking activity at either the application level or the operating system level to look for abnormal behavior. Problems with these approaches include: a) the inability of the IDS to capture both application and operating system behavior (which limits completeness); b) the significant amount of overhead introduced into the runtime system (which impacts performance); and c) the inability of the IDS to avoid being compromised by malicious software (which impacts security).

[0006] Security mechanisms that are able to differentiate regular (normal) behavior from malicious (abnormal) behavior may promise new ways to detect, counter and ultimately prevent the execution of zero day attacks and hidden malware. To date, however, these IDSs have not been able to do so in a manner that is not resource intensive or without impairing normal operation.

[0007] New security measures are essential to secure computer systems, protect digital information and restore user confidence.

[0008] An intrusion detection system collects architectural level events from a Virtual Machine Monitor where the collected events represent operation of a corresponding Virtual Machine. The events are consolidated into features that are compared with features from known normal system operation. If an amount of any differences between the collected features and the normal features exceeds a threshold value, a compromised Virtual Machine may be indicated. The comparison thresholds are determined by training on normal and/or abnormal systems, and analyzing the collected events with machine learning algorithms to arrive at a model of normal operation.

[0014] Current state-of-the-art security systems use anti-virus software and firewall programs to safeguard a system. These solutions introduce significant overhead into the application environment. Further, anti-virus software depends upon having a repository of known exploits, i.e., a signature database, that can be used to scan binaries. Anti-virus software is heavily dependent on keeping the repository up to date and cannot identify or disable viruses that are just being deployed (known as zero-day attacks).

[0015] Embodiments of the present invention do not rely on known bad behavior, but instead are based on profiles of known good behavior. The approaches described herein are proactive by design and remain adaptive, identifying and defending against new exploits and attacks as they are deployed.

[0016] Advantageously, embodiments of the present invention provide a VMM-based Intrusion Detection System (VIDS) that utilizes the virtual machine monitor (VMM) layer in a virtualized system to extract VMM-level semantics or information during runtime. By extracting VMM-level information that, in one embodiment, is optimized to a particular VMM and architecture, the IDS is easier to deploy and manage as a part of a VMM.

[0017] As will be understood from the description below, one or more embodiments of the present invention do not depend on a particular operating system running in a corresponding virtual machine (VM). Different versions of operating systems, e.g., Windows and Linux, are supported without the need for any modifications.

[0018] Further, as part of a virtualization platform, security for multiple systems can be managed and controlled from a single centralized point. This reduces the cost and overhead associated with deploying VIDS across an organization.

[0019] While it may appear that a VMM-based IDS introduces a semantic gap between the program-level behavior of malware and the information that is extracted from the VMM, embodiments of the present invention address this concern by employing advanced data mining techniques. As will be described below, VMM-level events are extracted and features are developed that, when combined with sophisticated machine learning algorithms, accurately identify security intrusions in compute-server appliances.

[0020] The entire contents of United States provisional patent application Ser. No. 60/063,296 filed Feb. 1, 2008 and entitled “Intrusion Detection System Using Virtualization-Based Profiling And Pattern Classification Algorithms,” and provisional application Ser. No. 61/147,913 filed Jan. 28, 2009 and entitled “VMM-Based HIDS” are hereby incorporated by reference for all purposes and wherein a copy of the latter application is provided herewith in an appendix.

[0021] It is to be appreciated that embodiments of the methods and apparatuses discussed herein are not limited in application to the details of construction and the arrangement of components or steps set forth in the following description or illustrated in the accompanying drawings. The methods and apparatuses are capable of implementation in other embodiments and of being practiced or of being carried out in various ways. Examples of specific implementations are provided herein for illustrative purposes only and are not intended to be limiting. Particular acts, elements and features discussed in connection with any one of the embodiments are not intended to be excluded from a similar role in any other embodiments. Also, the phraseology and terminology used herein are for the purpose of description and should not be regarded as limiting. The use herein of “including,” “comprising,” “having,” “containing,” “involving,” and variations thereof, is meant to encompass the items listed thereafter and equivalents thereof as well as additional items.

[0022] As will become clear from the description below, embodiments of the present invention do not require any particular hardware platform. Nonetheless, because it is so common, and by way of example only, it is assumed below that an x86 architecture, e.g., as used in the Intel IA32 line of microprocessors, is being used.

[0023] Prior to a description of one or more embodiments of the present invention, a brief overview of virtualization technology will be presented. It should be noted that this is not intended to be a full explanation of virtualization and the concepts therein. One of ordinary skill in the art will understand that there are further details not shown as they are not necessary to understand embodiments of the present invention.

[0024] Referring now to FIGS. 1A and 1B, representations of the two most common types of virtualization systems are presented. Both virtualized systems will be described in order to subsequently describe embodiments of the present invention.

[0025] FIG. 1A presents an un-hosted virtualization system. In this model a hypervisor or virtual machine monitor (VMM) 130 is installed directly on system hardware 100. The VMM 130, as known, includes a VM control module or function 131, a resource allocation module or function 133 and a resource sharing module or function 135.

[0026] The system hardware 100 includes a central processor (CPU) 102, which may be a single processor, or multiple cooperating processors in a known multiprocessor arrangement. As in other known systems, the hardware includes, or is connected to, memory 107, disk 106 and network I/O 104.

[0027] Virtual machines (VMs) 114 are instantiated on top of the VMM 130, and provide a software abstraction to the guest operating system 118 and hosted applications 128. The virtual machine provides virtualized system elements including virtual CPUs (VCPUs) 116, virtual memory 120, virtual disks 122 and virtual drivers 126 as needed for controlling and communicating with various devices and the system hardware 100. Embodiments of the present invention do not presuppose any particular host operating system, and because the characteristics and functions of operating systems are so well known, the Guest Operating System (Guest OS) 118 need not be discussed in greater detail.

[0028] At least one virtual machine (VM) 114, is installed to run as a Guest on the host system hardware and software. As is well known in the art, a VM 114 is a software abstraction—a “virtualization”—of an actual physical computer system. As such, each VM 114 will typically include one or more virtual CPUs 116 (VCPU), Guest OSs 118 (which may, or may not, be a copy of a conventional, commodity OS), a virtual system memory (VMEM) 120, a virtual disk (VDISK) 122, and drivers (VDRV) 126, all of which are implemented in software to emulate components of an actual or physical computer. Although the key components of only one VM 114 are illustrated in FIG. 1A, the structure of any other VMs may be essentially identical.

[0029] FIG. 1B is a representation in block diagram form of the structure of a hosted VMM 130. As known to one of ordinary skill in the art, the hosted VMM is installed on top of a Host Operating System (HOS) 112, which is then installed on the physical hardware 100. The structure of the VM 114 on a hosted VMM 114 remains essentially the same as for the unhosted VMM. The HOS 112 carries out many of the services associated with managing the physical hardware resources 100, whereas in an unhosted VMM 130 the VMM is responsible for this management.

[0030] Most computers are intended to run various applications, and VMs are no exception. Consequently, by way of example, as shown in FIG. 1A, a group of applications 128 (which may be a single application) is installed to run at user level on the Guest OS 118; any number of applications, including none at all, may be loaded for running on the Guest OS 118, limited only by the capacity or requirements of the VM. In some virtualization systems, an application (or a user of the application) will not “know” that it is not running directly on “real” hardware. As known to one of ordinary skill in the art, the applications and the components of the virtual machine 114 are instructions and data stored in memory, just as any other software. The concept, design and operation of virtual machines are well known in the field of computer architecture.

[0031] The VMM 130 operates as an interface between a VM 114 and the hardware 100 in the case of an unhosted VMM and between an HOS 112 and the hardware 100 in a hosted VMM. As functionality of the virtualization software may alternatively be implemented in hardware or firmware, a more general term for this functionality is “virtualization logic.” For the hosted model, a HOS 112 is real in the sense of being either the native OS of the underlying physical computer, or the OS (or other system-level software) that handles actual I/O operations, takes faults and interrupts, etc. In a hosted VMM, the hardware 100 and the OS 112, together, are responsible for executing VM-issued instructions and transferring data to and from the actual, physical memory 108 and the storage devices 106.

[0032] In either model, the virtualization software generally takes the form of a virtual machine monitor (VMM) 130, which is usually a “thin” piece of software that runs directly on top of either a HOS 112 in the hosted VMM model or directly on the hardware in the unhosted model and virtualizing all, or at least some subset of, the resources of the machine.

[0033] The interface exported by a VMM 130 to a respective VM 114 can be the same as the hardware interface of the machine, or at least of some predefined hardware platform, so that the Guest OS 118 cannot determine the presence of the VMM. The VMM 130, in the unhosted model, schedules and handles all requests by its VM for machine resources and handles various faults and interrupts. In the hosted model, the VMM 130 handles some requests directly and may track and forward others to the HOS 112, as well as handle various faults and interrupts. The general features of VMMs are known in the art and therefore need not be discussed in further detail here.

[0034] One advantage of virtualization is that each VM can be isolated from all others, and from all software other than the VMM, which itself will be transparent to the VM; indeed, as above, the user of a VM will be completely unaware that it is not running directly on the hardware, but on a virtualized set of hardware resources.

[0035] One embodiment of the present invention is securing those systems known as “software appliances,” and applications servers in general servers. The basic architecture of this class of system is a commodity hardware platform (typically an X86-based system) with an MS/Windows or Linux/Unix operating system installed, and with a select set of applications present. Given the somewhat static nature of an appliance-based or application server environment, one can track and inspect execution on these systems, as the expected execution is based on a set number of installed applications. Typically, users will not login to these systems, and so the only execution should be applications initially installed on the system. Of course, one of ordinary skill in the art will understand that embodiments of the present invention are applicable to other systems, specifically those where applications can be installed over time, and not limited only to a software appliance or application server.

[0036] As above, embodiments of the present invention address the need to protect data, applications and an operating system from malicious code attacks and insider threats. Advantageously, embodiments of the present invention obtain information about both application and operating system behavior without introducing significant runtime overhead into the execution environment.

[0037] This advantage is provided at least by: a) the use of a virtualization layer to provide access to the execution stream below the Guest operating system, producing feature-rich execution profiles, and b) the use of machine learning and pattern classification algorithms to identify abnormal behavior, given such profiles.

[0038] Obtaining information in the virtualization layer has many advantages as compared to working at either an application layer or at the operating system layer. These advantages include, but are not limited to: i) the virtualization layer is essentially invisible to a potential attacker as it is located below the Guest operating system and isolates the profiling system from an attacker, ii) the described approach has a relatively small execution footprint and so introduces very little performance overhead, iii) the approach is transparent to the guest operating system (in that the OS has any knowledge of it), iv) the approach is portable and does not depend upon any particular OS, guest or otherwise, and v) the approach is relatively straightforward to deploy and manage.

[0039] Definitions

[0040] The following terms are used throughout this description:

[0041] Events—Raw run-time information collected directly at the virtualization layer. Events form an event stream when collected over time. An event is considered to be at the architectural level of the VMM, e.g., machine state such as register values, hardware—software interface information, etc.

[0042] Features—Processed event streams, capturing information on system behavior suitable for off-line profiling and/or on-line monitoring. Features include, but are not limited to, event frequency, event correlation, and other information extracted from the event stream.

[0043] Profile—An aggregation of features, representing a “snapshot” of system behavior either at a specific point in time or over a period of time. Profiles can be used off-line (pre-deployment) to construct a model of normal system behavior, and can be used on-line (post-deployment) to detect anomalous behavior, in conjunction with a model of normal behavior.

[0044] Profiling—The act of collecting a system profile, either off-line (pre-deployment) or on-line (post-deployment).

[0045] Off-line modeling—The act of constructing a model of normal system behavior. Off-line modeling occurs pre-deployment and includes the collection of events, the production of features, the generation of a system profile, and the use of machine learning techniques to generate a model of system behavior.

[0046] Execution Model—A characterization of normal system behavior, generated during the off-line modeling stage.

[0047] On-line monitoring—The act of monitoring a running system in order to detect anomalous behavior. On-line monitoring typically occurs post-deployment and may include one or more of: the collection of events, the production of features, the generation of a system profile, and the use of a model of normal system behavior, together with machine learning techniques, to identify anomalous system behavior.

[0048] Detection—The act of identifying anomalous behavior while monitoring a running system, in either post-deployment or in off-line modeling (also referred to as training).

[0049] True positive—A correct detection of anomalous behavior during monitoring.

[0050] False positive—A misidentification of normal behavior as being anomalous during monitoring.

[0051] False negative—A misidentification of anomalous behavior as being normal during monitoring.

[0052] Remediation—The steps taken following detection during monitoring. Remediation includes, but is not limited to, conveying information back to the virtualization layer detailing the cause and nature of the detection.

[0053] As an overview of the system, referring to FIG. 2, via the VMM monitoring, in the front-end 300 event extraction 302 provides streaming events from which features are constructed 304. The streaming features are used as an execution profile provided to the back-end system 204 where off-line modeling 312 is implemented, resulting in a model 314. In addition, on-line modeling 316 uses a received execution profile and the model 314 to provide anomaly detection and subsequent remediation.

[0054] In one embodiment of the present invention, as shown in FIG. 3, a virtualized software appliance (VSA) 201 sits above a virtualization layer 130 that provides access to the hardware 100. A virtual intrusion detection system (VIDS) module 202 is provided within the virtualization layer 130 and communicates with an analysis system 204. The function of the VIDS module 202 will be described in more detail below. Further, the virtualization layer 130 may be in communication with a network, e.g., the Internet, not shown, and have access to other systems. It should be noted that the VIDS module 202 is a software construct running in the virtualization layer 130 and while it is referred to as a “module” it is only meant to be a term for explanatory purposes and does not necessarily indicate a limitation as to any stand-alone structure or configuration. The VIDS 202 has access, via an API at the VMM level, for example, to VMM architectural level, i.e., machine state, information. In commercial products, VMware's ESX Server provides the VMsafe APIs and Xen provides Xentrace, both of which provide the event information discussed in this invention.

[0055] As part of the VMM, or as a module inside it, the VIDS 202 has access to the following types of raw run time information which are used to generate events:

• • [0056] 1. VM architectural state information such as the VCPU 116 architectural state (for example its registers) and virtual devices 124 architectural state such as the virtual disk 122 and memory 120.
  • [0057] 2. Virtualization layer state, i.e., the VMM state, including, for example, the number of VMs running, state of each VM, etc. Another class of information available in the VMM includes the execution state of the VM. In some VMMs, this execution state indicates whether the code running in the VM is executed directly on the hardware or if it is being emulated.
  • [0058] 3. System state that can include time of day timers, CPU usage, and other runtime metrics.

[0059] The VIDS 202 tracks the execution of the VM and can monitor events corresponding to changes to the architectural, virtualization layer and system software state. These changes include for example: a write to a virtual register, or an access (read to, write from, control of) virtual device (VM architectural state changes), operations such as VM create or destruct (in the context of software virtualization layer) and system software state changes such as the current time.

[0060] The majority of these events can be tracked by most Virtual Machine Monitors as they are necessary to maintain correctness and isolation. Tracking events is also supported by new hardware extensions (aimed to support virtualization at the hardware level) to the X86 architecture such as Intel VT-x and AMD AMD-V.

[0061] Instead of, or in addition to, tracking events occurring in the VMM, a VIDS 202 can modify the VMM code to provide additional information available in the VMM, but not available to a current API. The VMM is modified at the point of installation of the IDS, and remains in place during all execution of the VMM. Modification to the VMM can be performed in source code (if source for the VMM is available) or through binary modification of the VMM. Since the VMM virtualizes a CPU(s), as well as devices, it is possible to extract event information directly from the software code emulating the behavior of the CPU or devices through the API in the VMM or through the modified VMM. Similarly, information related to the state of the virtualization can also be extracted as the software itself is implementing the functionality. Whenever an event is recorded, the VMM state information can accompany the event record.

[0062] By operation of the VIDS module 202, the virtualization layer 134 is configured to track events associated with program and operating system execution as implemented or controlled by the VMM 130. The configuration of the VMM can be done at installation time utilizing a fixed set of events, the events can be specified by the person installing the IDS by specifying the class of server workload that will be running on the virtualized system, or can be set dynamically by the off-line monitoring system to better match the characteristics of the applications running on a VM These events can include, but are not limited to: disk reads and writes, network requests, privilege level changes by the operating system, page table misses, and a variety of other events available within the virtualization layer 130. Advantageously, all execution, i.e., events, may be captured (versus only application execution events or a subset of the operating system execution events). This helps to assure that an intruder cannot identify a potential hole in the present system because all execution may be analyzed and/or captured.

[0063] Referring to FIG. 3, the monitoring or analysis system 204 is shown as a standalone or separate system used to analyze an execution profile, as represented by the collected events and/or features, generated at the virtualization layer 130. In one embodiment of the present invention, the monitoring or analysis system 204 is decoupled from the profiling system, i.e., the hardware 100, thus reducing the overhead associated with analysis of the profile. While FIG. 3 represents the system 204 as being physically separate, this is just one possible implementation of the current invention. In an alternate embodiment, the monitoring system 204 could be implemented as another virtual machine hosted on the same virtualized platform, or on a different platform, as would be understood by one of ordinary skill in the art. Further, the monitoring system 204 may be in communication with the VIDS 202 via a network (not shown). Still further, the monitoring system 204 may be in networked communication with multiple VIDS 202 providing intrusion detection capabilities to multiple systems at the same time. Of course, one of ordinary skill in the art would understand that the system 204 would have the same, or similar, hardware as that described above in FIGS. 1A and 1B.

[0064] The analysis system 204 includes capabilities including one or more of:

[0065] Offline profiling and modeling—training the system based on normal behavior of the applications and operating system.

[0066] Profiling control and configuration—initializing the virtualization layer to track selected events and to generate selected features.

[0067] Profile classification algorithms—a combination of machine learning and data mining algorithms that can be used, in conjunction with a model of normal behavior, to identify abnormal execution behavior of a system.

[0068] Intrusion detection—the use of a model of normal behavior, classification algorithms, and user-defined thresholds to identify intrusions.

[0069] False positive filtering—utilizing algorithms to reduce the number of false positives.

[0070] Intrusion remediation—identifying the actions necessary to thwart a detected intrusion may then be communicated to the virtualization layer.

[0071] System diagnostics—performing self-testing of the underlying system.

[0072] Management and reporting—the logging and reporting of the health or status of the system.

[0073] Operational View

[0074] From an operational standpoint, and as will be described in more detail below, embodiments of the present invention work in at least two stages: 1) an off-line (pre-deployment) stage; and 2) an on-line (post-deployment) stage.

[0075] In the off-line (pre-deployment) stage, the profiling system is configured to collect selected events and generate selected features. These features are then aggregated over time to generate profiles, which in turn are used by machine learning algorithms to create a model of normal system behavior. As such, embodiments of the present invention profile a system in a pre-deployment phase. Such profiling can occur, for example, while a system is being configured and benchmarked, i.e., prior to deployment. While one cannot assume that system behavior is constant across time, initial profiling is conducted at the pre-deployment stage and can continue thereafter during the online stage to adapt to changes in normal behavior over time. This adaptation involves the system switching between on-line monitoring and back to off-line modeling. This capability will allow the backend system to learn the behavior of any new applications that may be installed over time.

[0076] In the on-line (post-deployment) stage, the system is monitored by continually profiling the running system and using these profiles, in conjunction with machine learning algorithms and the pre-generated model of normal behavior, to detect anomalous behavior. The sensitivity of the machine learning detection algorithms can be altered by settable detection thresholds. Thresholds can be set by the user to trade-off, i.e., adjust the detection rate (true positives) and the false-alarm rate (false positives) to provide acceptable performance. In one embodiment, an interface on the back-end is provided through which these thresholds can be set and, in another embodiment, the on-line system adjusts these thresholds dynamically based on the range of feature values observed.

[0077] Architectural View

[0078] As above, from an architectural standpoint, the present invention consists of two sub-systems: (1) the “front end” profiling sub-system and (2) the “back end” modeling and monitoring sub-systems, as shown in Figure H as the Analysis System 204. In an embodiment of the present invention, these subsystems can be separate or combined.

[0079] Front End

[0080] The front end consists of a profiling sub-system that collects events, produces features, and generates system execution profiles.

[0081] Events

[0082] An event is the data and/or information extracted by the VIDS from the VMM during execution. It is understood that the information that can be extracted may differ from a VMM provided by one vendor as compared to a VMM provided by a different vendor. These differences may have an effect on the resulting comparisons.

[0083] Embodiments of the present invention, however, target those VMMs that are similar to one another (in terms of functionality, performance, target architecture, etc.) such as the VMware Workstation, VirtualBox, ESX Server, and Xen. The virtualization layer provides a mechanism by which the events can be obtained. The events—some events are related to the VMM and are common across different VMMs—the rest of the events are related to the characteristics of the underlying hardware (as presented in the VCPU, Vmem, Vdisk and virtualized network). The VMM and architectural events that the VMM intercepts are used as building blocks of features. With further analysis they become the behavior profile of the system. These events include, for example and not meant to be limiting, execution of privileged instructions, access to shared resources (memory), and I/O, e.g., disk, network, device, etc. Using a common set of events provides a robust VMM-based IDS since the system will not need to modified when moving to a different VMM. Of course, one of ordinary skill in the art will understand that the teachings of the present invention found herein can be applied to any VMM with minor modifications. In this work the open source edition of VirtualBox was used to construct the front-end.

[0084] Events are divided into three classes:

[0085] 1) Virtual (VM) events—architectural-level and system events related to the virtualized guest OS executing inside the VM. For example, a guest modifying control registers, flushing the Translation Lookaside Buffer (TLB) or writing to the disk. These events may also include or correspond to particular applications running in the VM, e.g., a database application or the Guest OS. Thus, an application that is subsequently identified as suspicious, i.e., possibly malicious, can be isolated, terminated, etc.

[0086] 2) VMM events—these events are extracted from the VMM and relate to the state of the VMM itself (as influenced by the state of the guest OS or the interaction between the guest OS and VMM). For example, the VirtualBox implementation has two internal modes: one to execute the guest OS directly on the CPU without intervention (user mode instructions) and another to intercept, instrument, or emulate system mode instructions.

[0087] 3) Real events—these events can be extracted from within the VMM or from the host OS. The semantics of these events relate to the host OS. The real time clock is an example of such an event.

[0088] Features

[0089] A feature is derived from the events and is used to provide the information in an input format for processing by the back-end. The back-end (the machine learning algorithms) is given processed information that can be the result of filtering, aggregating, or transforming events. Features capture characteristics present in the event stream, so do not contain all the raw events contained in the original stream, but do quantify patterns that are effective in identifying normal and/or abnormal execution. Using features rather than events can help identify relevant execution patterns and behaviors.

[0090] There are multiple ways to construct features. Several feature dimensions can be used to extend the design space. These dimensions include the number of events used to construct the feature, the type of the event (virtual events, VMM events or real events) as well as the information used to measure time (virtual time, real time).

[0091] Rate Features

[0092] In embodiments of the present invention, rate features are constructed directly from the event stream in the following way, referring to FIG. 4. In one embodiment, rate features are constructed by counting the number of occurrences of events, or types of events, in each segment. For example, an event stream 402 is divided into consecutive segments 404 of equal (virtual) time, for example, SEGMENT_—0, SEGMENT_—1, SEGMENT_—2, (in this example, a segment is constructed on each *TIMER event). Timer events are available in the VMM and can related to virtual time or physical time, i.e., wall-clock time.

[0093] Next, the events in a segment 404 are characterized to provide or produce feature-values for each segment 404. Thus, for example, during SEGMENT_—0 there were two disk I/O operations and one network I/O operation. As shown, a first window WINDOW_0 406-1 is represented as a string <2 1 . . . >. Similarly, for a second window WINDOW _1 406-2, the events in the segment are represented as <1 0 . . . >. One of ordinary skill in the art will understand that the number of occurrences of events as represented in the windows 406 is not the same as those shown and these have been left out merely for the sake of clarity. It is understood that all or less than all of the events in a window would be accounted for (unless filtered out). Once all of the events in a window or segment have been accounted for, the feature-values are sent on for processing.

[0094] There are at least two advantages to using time-based windows. First, each window represents approximately the same amount of execution time and are therefore comparable to one another. Second, splitting the event stream into windows provides the ability to classify each window on its own, enabling on-line classification.

[0095] The length of the window (the virtual time captured in the window) introduces a trade-off, however, as longer windows capture more behavior while shorter ones reduce time to detection. Typically, a time interval is selected based on the class of applications present on the VM, since the intensity of the application and its interaction with the VMM will determine the number of events occurring per unit time. The time interval is set during off-line modeling. This allows each VM to have an associated time quantum in order to compute rate features. Alternatively, the user can be given the ability to set the time quantum. This time quantum provides the backend with sufficient information to make accurate classifications on a per window basis, and also allows the system to identify malicious activity within seconds from its execution.

[0096] Correlation Features

[0097] Correlation features are built with the intention of (augmenting) the information not covered by rate features. This may occur, for example, when different events have the same rate across windows. These windows can be differentiated from one another by, for example, accounting for the order that particular events took place, e.g., writes after reads, reads after writes, etc., where (deviation from an expected sequence may suggest and reveal the presence of an intrusion). For example, if during off-line modeling it was observed that in normal execution it is unlikely that the system would expect to see a high rate of disk writes in the same time quantum as a high rate of network reads, such an occurrence of events could indicate a detection of abnormal operation.

[0098] Profiles

[0099] A profile is an aggregation of features, representing a “snapshot” of system behavior in or over time. In one embodiment of the present invention, the profile consists of a set of windows and their associated features, as described above. Profiles are generated off-line (pre-deployment) in order to build an execution model of normal behavior and on-line (post-deployment) in order to identify anomalous behavior, in conjunction with an execution model and machine learning algorithms.

[0100] Back End

[0101] The back end consists of off-line modeling and on-line monitoring subsystems. Off-line modeling occurs pre-deployment, and in one embodiment of the present invention, consists of a feature analysis and selection phase followed by a model construction phase. On-line monitoring occurs post-deployment and consists of anomaly detection and remediation. Each of these aspects of the back end are discussed below.

[0102] Off-line Modeling

[0103] A model of the normal behavior of a given system is created off-line; such a model can be created, for example, while the system in question is being configured and “stress tested” pre-deployment. The model is captured in a vector containing the set of features being used by the backend system, and the feature values observed during off-line modeling.

[0104] Given the events, features, and profiles collected by the “front end” profiling sub-system, the back end “modeling” sub-system synthesizes a model of the normal behavior of the system. Synthesizing such a model involves analyzing the profiling information collected by the “front end” (some information is more “valuable” than others) followed by constructing a model from the most useful profiling information available.

[0105] Feature Analysis and Selection

[0106] In one embodiment of the present invention, the well-known Boosting Algorithm from Machine Learning is used to analyze the profiling information collected by the front end. A system executing a “normal” workload is injected with a diverse set of known malicious attacks. Events are collected by the front end and partitioned into contiguous blocks—“windows in time”—and within each window, features are constructed by aggregating the various events, e.g., by counting the various event types. Each window is represented by its set of features, and the aggregation of these windows (and associated features) constitutes a profile. Each window is labeled as “malicious” or “normal” depending on whether any attack was active during the window in question or not.

[0107] The Boosting Algorithm uses this labeled profile as training data to build an accurate classifier for discriminating between “malicious” and “normal” behavior. As known to those of ordinary skill in the art, thresholded features (“decision stumps”) are used as “weak learners” within the Boosting Algorithm, and given the labeled profile as training data, the Boosting Algorithm produces a weighted linear combination of these decision stumps as its classifier. This weighted linear combination effectively gives more total weight to those features most useful in discriminating “malicious” from “normal” behavior and less total weight to those features least useful for such purposes. Boosting identifies features useful in modeling normal behavior, and not merely those features useful in identifying the given specific malicious behavior. As such, boosting can be used as an effective method to analyze features, determining those most useful for modeling normal behavior in a given system. Other feature analysis and selection algorithms can be employed as well.

[0108] Model Construction

[0109] Given an informative feature set obtained from feature analysis and selection, a model of normal behavior can be constructed. In one embodiment of the present invention, a variant of the well-known one-nearest-neighbor (“1NN”) algorithm from Machine Learning is used. The profile information collected from a system executing a “normal” workload is collected, and the windows (and their associated features) are stored. Such a set of windows constitutes one model of normal behavior.

[0110] As known to those of ordinary skill in the art, the 1NN algorithm associates any new window with the most similar window in the 1NN model, where the similarity between two windows can be computed in any number of ways, as a function of the windows' respective feature sets. A new window which is similar to a known normal (model) window can be assumed normal, while a new window different from any known normal (model) window can be assumed anomalous. A similarity threshold can be used as a cutoff to delineate normal (i.e., similar) from anomalous (different), and this threshold can be tuned to trade-off false-positive and false negative errors. Such tuning can be accomplished through the use of a validation data set consisting, for example, of feature windows collected from known normal and malicious behaviors during pre-deployment model construction. The 1NN similarity values or thresholded predictions can also be filtered to enhance prediction accuracy, using state-of-the-art filtering techniques.

[0111] The on-line monitoring system keeps track of the classification of each window (normal vs. abnormal).

[0112] For example, an alarm could be raised based on one anomalous window, with respect to a predetermined threshold, being detected by the on-line monitoring system.

[0113] Alternately, an alarm could be raised if a predetermined number of consecutive windows, e.g., three, is detected as being abnormal.

[0114] Still further, the alarm could be raised when some amount of windows in a series of consecutive windows, i.e., a width of windows, are abnormal. For example, if the width is set to 10, and a window threshold is set to 60%, the alarm will be raised once any 6 out of the 10 windows are classified as abnormal. Note that this window threshold parameter may improve the detection rate (true positives) but might also increase false positives.

[0115] An alarm may be triggering using a confidence parameter that relaxes both width and window threshold parameters by allowing the raising of the alarm before the window threshold has been reached. This parameter accounts for the amount of the difference between the feature-value and its corresponding threshold. A level of confidence increases when the value is significantly higher than the corresponding threshold. This confidence parameter enables the system to decrease the time to detection by raising the alarm before the window threshold has been reached.

[0116] As such, an alarm is raised when the conditions specified in the filter are satisfied. Those familiar with the field of pattern classification will recognize that a large number of filters could be used to generate the alarm, and that the selection of the most appropriate alarm can be determined during off-line monitoring.

[0117] Other state-of-the-art unsupervised and semi-supervised machine learning and filtering algorithms can be employed as well for building an off-line model, as well as for providing for on-line monitoring.

[0118] On-line Monitoring

[0119] Given a model of the normal behavior of a given system constructed pre-deployment, the system in question can be monitored on-line, post-deployment. In one embodiment of the present invention, a 1NN model of normal behavior is constructed as described above, pre-deployment. Post-deployment, the running system is profiled by the front end, and this profile information (windows and associated features) is provided to the back end sub-system in order to detect anomalous behavior, as described above. The underlying model of normal behavior can also gradually evolve over time by returning to pre-deployment modeling mode to adapt to changing use patterns, using variants of these and other machine learning algorithms. Flagged anomalous behavior, together with information sufficient to decode and interpret the anomaly, e.g., the process number associated with the anomalous behavior, and the window features primarily responsible for incurring the low similarity, are provided for remediation to the VMM. The VMM can then take whatever action is appropriate based on the severity of the intrusion.

[0120] Further modifications to the embodiments of the present invention described herein include:

[0121] Changes to the Event format. In one embodiment, plain text is used to extract events and related information. It is envisioned that there is a 5-10% overhead charge associated with text (string) formatting. This overhead can be easily removed by using a binary format.

[0122] Filter unnecessary events. Rather than generating all events regardless of their usefulness to the analysis, filter them out preemptively. This is done at the VIDS and can be determined during off-line modeling.

[0123] On-the-fly trace consumption. In one embodiment, all information is stored in a trace on disk. While using traces may be simple and repeatable, it consumes large amounts of storage (GigaBytes) and can slow the physical host down. Alternatively, all information can be used/consumed directly.

[0124] Further, it is recognized that a high false positive rate can inhibit deployment in a production environment. This issue can be alleviated by improving the predictor, the quality of the features and the number of events extracted. Second, the IDS described herein can be deployed as one component of a system which can include multiple detectors (OS IDS as well as Application IDS). In such a case multiple sources will need to agree before raising an alarm.

[0125] VMM IDS suitability—As described above, one exemplary embodiment is directed to evaluating the IDS on regular environments such as software appliances. These environments, while limited, are common in data centers.

[0126] Evading the VMM IDS

[0127] A weakness shared by most types of IDSs, is one in which an attacker can study the defense methods and create new attacks evading the detectors. Although a VMM IDS may not be totally immune from attack, it is believed that it would be much harder to accomplish. An attacker would need to generate a low-level footprint such that it is either identical to the normal running workload or one that is very light and is, therefore, able to pass as being not abnormal. This task is both difficult to accomplish and is highly dependent on the target machine normal workload.

[0128] Timeliness—Timely detection is one of the main goals of any IDS. It is clear that a timely and early detection is advantageous. The presently described IDS is able to detect most malware within minutes of introduction. Although the detection is not always immediate, it is better to detect an attack after a few minutes than never. And while some damage can be done in the meantime, it is restricted to one VM.

[0129] Response—Generating a response to an attack is a followup to detecting the intrusion. While there may be no current industry framework for coordinating a response, it is envisioned that OS support can be used to interpret the low level data, identify the malware and generate a report useful for system administrators.

[0130] Additionally, several actions can be taken to resolve the attack. For example, a breached guest VM can be put offline while the attack is analyzed or an identified application can be quarantined, or the VM can be discarded and destroyed. Moreover, in many cases, appliances or a VM can be rolled back to a last known good configuration (a checkpoint). This action is relatively straightforward to implement in a VM environment.

[0131] Summary

[0132] A VMM-based IDS increases the ease of deployment across different operating systems and versions, and, as part of a VMM offers high manageability for computer-server appliances. A VMM-based IDS breaks the boundaries of current state-of-the-art IDSs and represents a new point in the IDS design space that trades a lack of program semantics for greater malware resistance and ease of deployment.

[0133] Embodiments of the above-described invention may be implemented in all software, all hardware, or a combination of hardware and software, including program code stored in a firmware format to support dedicated hardware. A software implementation of the above described embodiment(s) may comprise a series of computer instructions fixed on a tangible medium, such as a computer readable medium, e.g. diskette, CD-ROM, ROM, or fixed disk. The series of computer instructions embodies all or part of the functionality previously described herein with respect to the embodiment of the invention. Those skilled in the art will appreciate that such computer instructions can be written in a number of programming languages for use with many computer architectures or operating systems and may exist in machine executable format. It is contemplated that such a computer program product may be distributed as a removable media with accompanying printed or electronic documentation, e.g., shrink wrapped software, preloaded with a computer system, e.g., on system ROM or fixed disk.

[0134] Although various exemplary embodiments of the present invention have been disclosed, it will be apparent to those skilled in the art that changes and modifications can be made which will achieve some of the advantages of the invention without departing from the general concepts of the invention. It will be apparent to those reasonably skilled in the art that other components performing the same functions may be suitably substituted. Further, the methods of the invention may be achieved in either all software implementations, using the appropriate processor instructions, or in hybrid implementations that utilize a combination of hardware logic and software logic to achieve the same results. Such alterations, modifications, and improvements are intended to be part of this disclosure and are intended to be within the scope of the invention. Accordingly, the foregoing description and drawings are by way of example only, and the scope of the invention should be determined from proper construction of the appended claims, and their equivalents.