US 7,854,004 B2Grant
Computer immune system and method for detecting unwanted code in a computer system
Issue Date:2010-12-14
•20 Claims
•8 Drawing Sheets
Abstract
An automated analysis system detects malicious code within a computer system by generating and subsequently analyzing a behavior pattern for each computer program introduced to the computer system. Generation of the behavior pattern is accomplished by a virtual machine invoked within the computer system. An initial analysis may be performed on the behavior pattern to identify infected programs on initial presentation of the program to the computer system. The analysis system also stores behavior patterns and sequences with their corresponding analysis results in a database. Newly infected programs can be detected by analyzing a newly generated behavior pattern for the program with reference to a stored behavior pattern to identify presence of an infection or payload pattern.
Metadata
Assignee
- International Business Machines Corporation
Inventor
- Peter A. J. van der Made
Application Information
Application Number:US 11/094,489
Filing Date:2005-03-30
Priority Date:2000-07-14
Art Unit:2435
Classifications
IPC:
H04L9/00
Patent Drawings (8 sheets)
Description
Cross Reference to Related Application
[0001] This application is a continuation of and claims priority to U.S. application Ser. No. 09/642,625 filed Aug. 18, 2000, now U.S. Pat. No. 7,093,239 entitled, “Computer Immune System and Method for Detecting Unwanted Code in a Computer System”, the entire contents of which are hereby incorporated by reference. The 09/642,625 application claims priority from the U.S. provisional patent application Ser. No. 60/218,489, filed Jul. 14, 2000, which application is also hereby incorporated by reference in its entirety.
Background of the Invention
[0002] 1. Field of the Invention
[0003] The present invention relates to the field of computer security and specifically to the detection of computer programs that exhibit malicious or self-propagating behavior including, for example, computer viruses and trojans.
[0004] 2. Discussion of the Related Art
[0005] Detection of viruses has been a concern throughout the era of the personal computer. With the growth of communication networks such as the Internet and increasing interchange of data, including the rapid growth in the use of e-mail for communications, the infection of computers through communications or file exchange is an increasingly significant consideration. Infections take various forms, but are typically related to computer viruses, trojan programs, or other forms of malicious code. Recent incidents of e-mail mediated virus attacks have been dramatic both for the speed of propagation and for the extent of damage, with Internet service providers (ISPs) and companies suffering service problems and a loss of e-mail capability. In many instances, attempts to adequately prevent file exchange or e-mail mediated infections significantly inconvenience computer users. Improved strategies for detecting and dealing with virus attacks are desired.
[0006] One conventional technique for detecting viruses is signature scanning. Signature scanning systems use sample code patterns extracted from known malicious code and scan for the occurrence of these patterns in other program code. In some cases program code that is scanned is first decrypted through emulation, and the resulting code is scanned for signatures or function signatures. A primary limitation of this signature scanning method is that only known malicious code is detected, that is, only code that matches the stored sample signatures of known malicious code is identified as being infected. All viruses or malicious code not previously identified and all viruses or malicious code created after the last update to the signature database will not be detected. Thus, newly created viruses are not detected by this method; neither are viruses with code in which the signature, previously extracted and contained in the signature database, has been overwritten.
[0007] In addition, the signature analysis technique fails to identify the presence of a virus if the signature is not aligned in the code in the expected fashion. Alternately, the authors of a virus may obscure the identity of the virus by opcode substitution or by inserting dummy or random code into virus functions. Nonsense code can be inserted that alters the signature of the virus to a sufficient extent as to be undetectable by a signature scanning program, without diminishing the ability of the virus to propagate and deliver its payload.
[0008] Another virus detection strategy is integrity checking. Integrity checking systems extract a code sample from known, benign application program code. The code sample is stored, together with information from the program file such as the executable program header and the file length, as well as the date and time of the sample. The program file is checked at regular intervals against this database to ensure that the program file has not been modified. Integrity checking programs generate long lists of modified files when a user upgrades the operating system of the computer or installs or upgrades application software. A main disadvantage of an integrity check based virus detection system is that a great many warnings of virus activity issue when any modification of an application program is performed. It is difficult for a user to determine when a warning represents a legitimate attack on the computer system.
[0009] Checksum monitoring systems detect viruses by generating a cyclic redundancy check (CRC) value for each program file. Modification of the program file is detected by a variation in the CRC value. Checksum monitors improve on integrity check systems in that it is more difficult for malicious code to defeat the monitoring. On the other hand, checksum monitors exhibit the same limitations as integrity checking systems in that many false warnings issue and it is difficult to identify which warnings represent actual viruses or infection.
[0010] Behavior interception systems detect virus activity by interacting with the operating system of the target computer and monitoring for potentially malicious behavior. When such malicious behavior is detected, the action is blocked and the user is informed that a potentially dangerous action is about to take place. The potentially malicious code can be allowed to perform this action by the user. This makes the behavior interception system somewhat unreliable, because the effectiveness of the system depends on user input. In addition, resident behavior interception systems are sometimes detected and disabled by malicious code.
[0011] Another conventional strategy for detecting infections is the use of bait files. This strategy is typically used in combination with other virus detection strategies to detect an existing and active infection. This means that the malicious code is presently running on the target computer and is modifying files. The virus is detected when the bait file is modified. Many viruses are aware of bait files and do not modify files that are either too small, obviously a bait file because of their structure or have a predetermined content in the file name.
[0012] It is apparent that improved techniques for detecting viruses and other malicious types of code are desirable.
Summary of the Preferred Embodiments
[0013] One aspect of the present invention provides a method for identifying presence of malicious code in program code within a computer system, including initializing a virtual machine within the computer system. The initialized virtual machine comprises software simulating functionality of a central processing unit and memory. The virtual machine virtually executes a target program so that the target program interacts with the computer system only through the virtual machine. The method includes analyzing behavior of the target program following virtual execution to identify occurrence of malicious code behavior and indicating in a behavior pattern the occurrence of malicious code behavior. The virtual machine is terminated at the end of the analysis process, thereby removing from the computer system a copy of the target program that was contained within the virtual machine.
[0014] Another aspect of the present invention provides a method for identifying the presence of malicious code in program code within a computer system. The method includes initializing a virtual machine within the computer system, the virtual machine comprising software simulating functionality of a central processing unit, memory and an operating system including interrupt calls to the virtual operating system. A target program is virtually executed within the virtual machine so that the target program interacts with the virtual operating system and the virtual central processing unit through the virtual machine. Behavior of the target program is monitored during virtual execution to identify presence of malicious code and the occurrence of malicious code behavior is indicated in a behavior pattern. The virtual machine is terminated, leaving behind a record of the behavior pattern characteristic of the analyzed target program.
Brief Description of the Drawings
[0015] FIG. 1 illustrates a behavior pattern generated according to the analytical behavior method, showing the behavior pattern for code that is not infected and is infected with a computer virus. Each bit may be a flag indicating an action. The total stream of bits is a value indicative of the behavior of the program.
[0016] FIG. 2 shows a block diagram of components used in a preferred implementation of the analytical detection method.
[0017] FIG. 3 schematically illustrates the COM file format, used as an example of the function of the program structure extractor and program loader.
[0018] FIG. 4 illustrates an interface of the virtual PC to various program file formats. Before virtualization can take place, the program loader preferably extracts the correct entry point, code and initialized data from the program file. The file offset to the entry point code is given in the program header and varies depending on the type of file that contains the program.
[0019] FIG. 5 schematically illustrates the virtual PC memory map after loading a binary image (.COM) program and after loading a MZ-executable program. To virtualize the code in the desired manner, the structure of the virtual PC and its memory map contains the same information as it would if the code was executed on the physical PC which runs the virtual machine containing the Virtual PC.
[0020] FIG. 6 provides a detailed diagram showing components of a preferred implementation of the Virtual PC. The virtual PC contains the same components that are used in a physical computer, except that all Virtual PC components are simulated in software running as a virtual machine on a physical computer.
[0021] FIG. 7 illustrates aspects of an exemplary computing environment in which an embodiment of the invention is designed to operate. For example, exemplary computing system 705 can comprise software modules performing the processes described herein, such as the processes of a virtual machine. The software modules can be stored in exemplary computer-readable storage device 720 and be executed by exemplary CPU 710 via exemplary computer-readable memory 715. Computing device 705, computer-readable storage device 720, CPU 710, and computer-readable memory 715 are conventional computing components well known to those of ordinary skill in the art. Other embodiments of the invention may operate in other types of computing environments.
Detailed Description of the Preferred Embodiments
[0022] A particularly preferred embodiment of the present invention provides an automated analysis system that detects viruses and other types of malicious code within a computer system by generating and subsequently analyzing a behavior pattern for each computer program introduced to the computer system. New or modified computer programs are analyzed before being executed by the computer system. Most preferably the computer system initiates a virtual machine representing a simulation of the computer system and the virtual machine executes the new or modified computer program to generate a behavior pattern prior to the new computer program being executed by the physical computer system. An initial analysis is performed on the behavior pattern to identify infected programs upon initial presentation of the program to the computer system. The analysis system also stores behavior patterns and corresponding analysis results in a database. Newly infected programs can be detected by subtracting the stored behavior pattern for that program from a newly generated behavior pattern, and analyzing the resulting pattern to identify an infection or payload pattern associated with malicious code.
[0023] A variety of different terms are used in programming to describe different functional programming subunits. At different times and for different programming languages subunits of various sorts have been called functions, routines, subprograms, subroutines and other names. Such designations and the context or differences they represent are not significant to the present discussion and so this discussion is made simply in terms of programs, intending the term program to refer to functional programming units of any size that are sufficient to perform a defined task within a computer system or computing environment. Such specialized functions as those performed by macros within certain word processing programs, including for example, in Visual Basic macros for Microsoft Word documents, are included within this general discussion. In this sense, individual documents may be considered to be programs within the context of this discussion.
[0024] For convenience and brevity, this discussion references viruses in the known sense of that term as being a self-propagating program generally undesired in the infected computer system. As used here, the term Windows is intended to reference any of the personal desktop operating systems sold by the Microsoft Corporation under the Windows brand name. The term PC or personal computer is used, unless specifically modified to indicate otherwise, to indicate a computer system based on the well-known x86 architecture, including those machines that presently are based on the microprocessor sold by Intel Corporation under its Pentium brand name and successors to that microprocessor and architecture. This discussion is provided to illustrate implementation of aspects of the invention. Aspects of the present invention find application in a range of different computer systems in addition to the illustrated personal computer systems.
[0025] The present inventor has analyzed the behavior of a variety of different viruses and other malignant source code. Certain general characteristics of viruses have been identified. A virus needs to infect other programs and eventually other computers to propagate. Viruses consequently include infection loops that copy the virus into another executable program or sometimes into documents, in the exemplary case of Visual Basic macro viruses. Viruses and trojans generally contain payloads. The payload allows the virus to affect the infected system or communicate its presence. A payload might be, for example, a message that pops up to announce the virus or a malicious function that damages the infected computer, for example by corrupting or erasing the data on the hard disk or by altering or disabling the BIOS within the BIOS flash or EEPROM.
[0026] Another common characteristic of viruses is that the virus becomes resident in the memory. DOS viruses need to copy themselves into memory and stay resident. Most viruses do not use the obvious terminate and stay resident (TSR) call but instead use a procedure that copies the virus into high memory. The virus then can directly modify the data in the high memory blocks. In an additional aspect of this infection scheme, the interrupt vector is modified to point at memory blocks that have been modified by the memory resident virus or other malignant procedure. These modified memory blocks store the infection procedure. Windows specific viruses bump themselves into ringo, for example using a callgate or DPMI call, and go resident in a system utility such as the system tray.
[0027] These behaviors are characteristic of a virus and are not, in the aggregate, characteristic of other, non-malignant programs. Consequently, a program can be identified as a virus or infected with a virus if it possesses certain ones of these behaviors, certain collections of these behaviors or all of these behaviors. In preferred embodiments of the present invention, the occurrence of these behaviors or combinations of the behaviors is indicated by collections of bits in a behavior pattern data set representing behavior characteristic of the infected program. An example of behavior patterns for a normal and an infected file are illustrated in FIG. 1 .
[0028] In preferred embodiments of the present invention, the behavior of a newly loaded or called program is analyzed in a virtual machine that simulates a complete PC, or a sufficiently complete PC, in software and it is that virtual PC that generates the behavior pattern. The virtual PC simulates execution of the new or modified program, simulating a range of system functions, and the virtual PC monitors the behavior of the suspect program and makes a record of this behavior that can be analyzed to determine that the target program exhibits virus or malignant behaviors. The result of the virtual execution by the virtual machine is a behavior pattern representative of the new program. As discussed in greater detail below, the behavior pattern generated by the virtual PC identifies that a program is infected with a virus or is itself a virus. An advantage for the use of virtual execution and analysis of new programs for viruses is that the virtual machine is virtual and so, if the virtualized new program contains a virus, only the virtual machine is infected. The infected instance of the virtual machine is deleted after the simulation, so the infection is incomplete and the virus does not propagate. The behavior pattern survives the deletion of the virtual machine, allowing an analysis program to identify the existence of the virus and of the infection within the new program.
[0029] Most preferably, each time a new program is analyzed a new instance of the virtual machine is generated, free of modification by any previously virtualized programs including any earlier analyzed viruses. The new program then is run on the new instance of the virtual machine preferably followed by initiation of a modified interrupt caller procedure, described in greater detail below. While the virtual machine is executing the new program in cooperation with the modified interrupt caller procedure, the virtual machine monitors all system calls, DPMI/DOS interrupts and I/O port read/write (r/w) operations, setting bits in the behavior pattern register according to the observed behaviors. It is these bits in the behavior pattern that are retained after the simulation is complete and the virtual PC has been terminated. The bits stored in the behavior pattern register are the behavior pattern and indicate whether the virtually-executed program includes behaviors indicative of the presence of a virus or other malignant code.
[0030] The modified interrupt caller procedure calls the interrupts that the program being analyzed has modified within the virtual PC and generates a behavior pattern for each of those interrupt service routines as well. This allows particularly preferred embodiments of the present invention to identify certain types of viruses that initially modify only the interrupt service routines and do not begin propagating until the modified interrupt or interrupts are called by another program. By allowing the various interrupt service routines in the virtual machine to be modified and then analyzing the modified interrupts, these embodiments of the invention can detect this delayed propagation mechanism.
[0031] In some presently preferred embodiments, only the static, final version of the behavior pattern is analyzed. It is possible, and in some circumstances desirable, to monitor the sequence in which the bits in the behavior pattern register are set. The order in which the behavior pattern bits are set provides additional information allowing identification of additional virus behaviors. Tracking of the order in which the behavior pattern bits are set is accomplished within the virtual machine.
[0032] Preferred implementations of the analytical behavior method (ABM) proceed by extracting a behavior pattern and sequence from a modified, new, unknown or suspect program. The behavior pattern is preferably used to analyze the behavior of the unknown program to determine if the behavior of the unknown program is malicious. Identification of malicious behavior in this manner allows identification of virus carrying files prior to infection of the host computer system. The behavior pattern can also be stored in a database and the virtual machine can subsequently analyze the behavior of the program following modification to determine if its functionality has been modified in a suspect (malicious) manner. This provides post-infection analysis.
[0033] The described analytical behavior method differs from conventional virus detection methods in that it does not match program code to a set of stored patterns as do signature scanners and integrity checkers. Rather, a virtual machine is used to generate a behavior pattern and a sequence. The generated behavior pattern does not change significantly between version updates, but does change dramatically when a virus infects a program. For example, a word processor will still behave like a word processor when the program is replaced or updated with a new version of the program but the word processor changes significantly when the word processor is infected with a virus. The differences reflected in the behavior patterns are illustrated in FIG. 1 . When a word processor is infected with a file infector computer virus, the word processor now opens executable files and inserts the viral code into them, thereby infecting additional files. This is clearly reflected in the illustrated behavior patterns.
[0034] In particularly preferred embodiments of the invention, the analysis procedure specifically targets infection methods such as, but not limited to, the insertion of code to other executables or documents, submitting code to other applications to be transmitted or stored, insertion of code into high memory blocks and the modification of memory control blocks. Preferred implementations of the analysis method further look for destructive content, such as, but not limited to, functions that overwrite disk areas or the BIOS ROM, or delete files or directories. Most preferably, the analysis makes an exception and does not identify as infected a program whose other behavior characteristics indicate that the program is a development tool or software debugging tool and where the modifying behavior is an integral part of the tool's normal function. A viral infection of a development tool can be detected where an expressed function is not part of the tool's normal function, that is, within the development process. Both active (1) and inactive (0) flags present in the behavior pattern are significant in this analysis, as well as the sequence in which actions take place.
[0035] In accordance with preferred embodiments of the present invention, the virtual machine or virtual PC represents a simulation of a complete computer system. A complete computer system preferably includes an emulated central processing unit (CPU), emulated memory, input/output (I/O) ports, BIOS firmware, operating system and the operating system data areas. This stands in contrast to simple emulation of a processor, in which only the processor processes are emulated. In emulation, program instructions are converted from their native form to a stream of instructions that perform the same function on a different hardware platform. Some signature scanning software employs emulation to decrypt the body of a suspect program before the suspect program is scanned for signatures. In virtualization, the entire computer is simulated including operating system calls, which are not actually executed but seem to the calling program to perform the desired functions and return the correct values as if they were executed.
[0036] As discussed above, the virtual PC includes a CPU, memory, I/O ports, a program loader, and the operating system application program interface (API's) entry points and interface. Using such a complete virtual PC is particularly preferred because it gives the analytical behavior method a high level of control over the virtualized program, including over the sophisticated direct calls to the operating system API. The virtualized program is not given access to any of the facilities of the physical machine, thereby avoiding the risk that the potential virus or other malicious code escapes from the controlled environment to infect the host computer system.
[0037] FIG. 2 provides an overview of the preferred analytic behavior method architecture including the relationship between the virtual machine and components of the host computer system. Program code is delivered to the ABM engine and analysis system by direct access to the hard disk through I/O port bit manipulation, hooking into the operating system file system or by sequentially scanning the hard disk. The program code is checked against the database for ‘known’ files. If the file is new or modified, it is processed. The resulting behavior signature is analyzed or compared and stored. A virus warning is returned when analysis shows that the file contains malicious code. The analytical behavior method preferably includes: (1) file structure extraction; (2) change detection; (3) virtualization; (4) analysis; and (5) decision.
[0038] Before the program can be virtualized, the file format containing the target program has to be evaluated. The entry point code is extracted and loaded into the virtual computer's memory at the correct simulated offset. In a physical computer this function would be performed by the program loader function, which is part of the operating system. The operating system can execute programs that are held in a collection of different file formats, such as:
[0039]
| DOS 1.0 | Binary image file, loaded at 100 h in memory, |
| and/or CP/ | maximum size: 64K. |
| M COM | |
| DOS 2.0- | MZ-type executable, header determines CS:IP of |
| DOS 7.1 EXE | load address. |
| Windows | NE-type executable which contains both the |
| 3.0 | DOS MZ-header pointing at a DOS code area |
| executables | and a New Executable (NE) header containing |
| the entry point of the Windows (protected mode) | |
| code. NE files are segmented. | |
| OS/2 | LE/LX type executable which contains both the |
| executables | DOS MZ-header and DOS code area and a |
| protected mode section which is determined by | |
| the LE-header following the DOS code segment. | |
| Linear Executable (LE) files are used in | |
| Windows 3 for system utilities and device | |
| drivers. LE files are segmented. LX files | |
| incorporate some differences in the way the | |
| page table is stored and are intended for the | |
| OS/2 operating system. LE files are segmented | |
| and the segments are paged. | |
| 32-bit | PE-type executable which contains both the |
| executables | DOS MZ-header and DOS code area and the |
| Portable Executable header containing the | |
| entry point and file offset of the protected mode | |
| code. PE files are segmented. | |
| OLE | OLE compound files (COM) are document files |
| Compound | that can contain executable format streams, |
| Files | usually referred to as Macros. All office |
| components incorporate Visual Basic for | |
| Applications, as does Internet Explorer versions | |
| 4 and 5. Windows98 systems can execute Visual | |
| Basic code directly from a script file. The Visual | |
| Basic code is compiled and stored in a stream, | |
| which is paged according to its file offset | |
| references stored in a linked list in the file | |
| header. | |
| Binary | A binary image is used for the boot sector and |
| Image | Master Boot and Partition table. Both the boot- |
| sector and the MBR contain executable code | |
| which is loaded into memory at 0:7C00 during | |
| the start-up process. | |
| Driver | System Drivers are stored as a binary image |
| files | with a header. The header contains information |
| about the drivers stored within the file. | |
| Multiple drivers can be stored within the same | |
| file. | |
[0040] The virtual computer loader function is capable of dealing with the file formats and binary image files shown above. The loader function is performed by virtualizing the operating system program loader and so varies depending on the operating system used in the host computer. The file structure analysis procedure looks in the file header and file structure to determine the file format, rather than using the file extension because file extensions are unreliable in general use. The .EXE formats described above therefore include DLL, AX, OCX and other executable file format extensions.
[0041] Compound document files can contain executable streams such as Visual Basic code or macros. The structure of a compound document file is illustrated in the diagram shown in FIG. 3 . The header of a compound document file contains a linked list (or File Allocation Table) which is referenced in a directory structure that points to the entry point of the linked list. Each entry in the linked list refers to the next entry and a file offset. A value of −1 in the linked list indicates the end of a chain. Streams exist out of blocks, which may be scattered anywhere in the file in any order. In particularly preferred embodiments of the invention, code extracted from a compound document file is passed through a Visual Basic decompiler before it is presented to a Visual Basic emulator. Not all compound document files contain compiled Visual Basic code. Hypertext markup language (HTML) and Visual Basic Script (VBS) files can contain Visual Basic Script code as text. This code is preferably extracted and treated as a Visual Basic stream within the virtual machine.
[0042] The NE/PE/LE executable file formats are similar in complexity, except that no linked list is used; rather these file formats use a segment or page table. The PE file format is based on the COFF file specification. FIG. 4 illustrates how these file formats interface with the preferred virtual PC in accordance with certain embodiments of the present invention. In evaluating how aspects of the preferred virtual PC interfaces to a particular file, the file loader preferably decides if the file presented is a document file or a binary file.
[0043] After the file format has been evaluated and the entry point-file offset has been calculated, the file is opened and the virtual machine reads the relevant code into memory as a data stream. The length of the code is calculated from fields in the header of the file. This information is passed to the virtual program loader. The virtual program loader uses information in the file header to load the extracted code at the correct simulated offset in a virtual memory array.
[0044] A memory mapping utility maps the virtual memory map to the offset for the file type that is virtualized:
- [0045] DOS (CP/m) binary image files (.COM) offset CS: 00h
- [0046] DOS (2.0 up) Executable format files (MZ-EXE) offset CS:IP from header
- [0047] Windows NE, PE, LE offset C0000000+CS:IP from header
- [0048] Binary Image MBR, Boot sector code offset 0:7C00h
- [0049] Document COM files, HTML and VBS files no specific offset, VBA code
[0050] The Loader utility dynamically assigns physical memory to the virtual computer memory array each time a program is virtualized, and proceeds to build a new virtual machine. Each virtual machine contains a BIOS data area, a filled environment string area, DOS data area, memory control blocks, program segment prefix area, the interrupt vector table and descriptor tables. The final structure of the virtual machine depends on the type of program that is virtualized. Each virtualized program therefore runs in a fresh memory area, created when that program is loaded into the virtual PC. Previous instances, where infected programs may have been virtualized, therefore cannot affect the performance of subsequent programs. The virtual machine is shut down and its memory resources are released when the virtualized program terminates and the virtual machine completes assembly of the behavior pattern for the target, virtualized.
[0051] FIG. 5 illustrates how the virtual memory is configured for (COM) binary image files and DOS program (MZ-EXE) files. The memory map and mapper utility are adjusted depending on the file type.
[0052] The program loader simulates the loader functions of the operating system and creates system areas that represent similar system areas in the physical computer. This is particularly advantageous functionality because the code under evaluation most preferably runs in the same manner as if executed on a physical computer system. The virtualized program is executed by fetching instructions from the virtual memory array into a pre-fetch instruction queue. The instructions in the queue are decoded and their length is determined by their operational parameters.
[0053] The instruction pointer is incremented accordingly so that the instruction loader is ready to fetch the next instruction. The virtual machine determines from the r/m field of the instruction parameters where data on which the instruction operates is to be fetched. The data fetch mechanism fetches this data and presents the data to the logic unit, which then performs the operation indicated by the code. The destination of the processed data is determined from the parameters of the instruction code. The data write mechanism is used to write the processed data to emulated memory or the emulated processor register set. This process accurately reflects what takes place in a physical CPU (central processing unit).
[0054] All areas of this process are simulated, as generally illustrated in FIG. 6 . The memory exists as an array of 400 Kbyte elements into which all memory accesses are mapped by a memory mapping mechanism. The size of the memory array may be adjusted in future implementations to accommodate larger programs. The video display is simulated from a system viewpoint as 128 Kbyte of memory mapped between A000:0 and BFFF:F (inclusive) in the virtual computer's memory map. The standard IBM PC input/output area is simulated as an array of 1024 bytes representing I/O ports 0-3FFh. The CPU is simulated by performing the same low-level functions as the physical CPU, but in high-level software.
[0055] The operating system is implemented as an area in the memory array of 700h bytes containing the BIOS data fields, the DOS data area, Memory Control Blocks and DOS devices. The interrupt vector table takes up the first 1024 (400h) positions in the memory array as it would in a physical PC. The DOS interrupt structure is implemented as simulated functions that return the correct values and by filling the memory array with the correct values expected by simulating DOS functions.
[0056] The operating system is implemented as a virtual API (VAPI) that simulates the results returned by all operating system API's.
[0057] During the virtualization process, flags are set in the behavior pattern (Tstruct) field as the functions represented by those fields are virtualized. The sequence in which these functions are called is recorded in the sequencer. The behavior pattern therefore matches closely the behavior of the program under evaluation to the behavior of that program in a physical PC environment. Simulated interrupt vectors modified during the process of executing the virtualized program are called after program virtualization terminates, thus acting as applications that would call such interrupt vectors in a physical computer following modification of these vectors.
[0058] To illustrate this functionality, consider the following set of operations might be performed in operation of the analytical behavior method:
[0059]
| Search for the first EXE file in this directory | ;set FindFirst Flag (Tstruct | |
| Structure) | ||
| Is this a PE executable (examine header)? | ;set EXEcheck Flag | |
| If not, jump far | ||
| Else: Open the executable file | ;set EXEaccess Flag | |
| Write to the section table | ;set EXEwrite Flag | |
| Search for the end-of-file | ;set EXEeof Flag | |
| Write to file | ;set EXEwrite Flag | |
| Close file | ||
| Search next EXE file | ;set EXEFindNext Flag | |
| Bit + 1 | 64--- -------------- -------------------- ------------------ -------------- ---------------- --------- --------1 | |
| Returned: | 0010 0100 1010 1010 1001 0101 0010 1111 0010 1010 0010 0100 0100 1001 0000 0101 | |
| Value: | 2 4 A A 9 5 2 F 2 A 2 4 4 9 0 5 | |
| Sequencer: | 21,22, 23,24,26,29,3E,1,36,38,3B,3, 9,C,F,13,16,1A,1C,1E, 2B,2D,30,32,34, | |
| The resulting behavior pattern is: 24AA952F2A244905 | ||
[0060] The behavior pattern contains flags that indicate that the user has not had the opportunity to interact with this process through user input (the userInput flag is not set). The sequencer contains the order in which the bits were set, identifying the infection sequence shown above. Therefore this observed behavior is most likely viral.
[0061] Many viruses are encrypted, polymorphic or use ‘tricks’ to avoid detection by signature scanners. Wherever such ‘tricks’ are used, the behavior pattern points more obviously towards a virus since such tricks are not normally used in normal applications. In any case, preferred implementations of the present invention require that an infection procedure be present to trigger a virus warning to avoid false positive warnings. Encrypted viruses are no problem, because the execution of the code within the virtual machine, which generates the behavior pattern, effectively decrypts any encrypted or polymorphic virus, as it would in a physical PC environment. Because all parts of the virtual computer are virtualized in preferred embodiments, and at no time is the virtualized program allowed to interact with the physical computer, there is no chance that viral code could escape from the virtual machine and infect the physical computer.
[0062] The change detection module compares existing files at 6 levels to determine if the file was analyzed previously:
-
- [0063] The file is the same (entry point code, sample, file-name and file-size are the same).
- [0064] The file is not in the database (new file).
- [0065] The behavior pattern matches a stored pattern.
- [0066] The file's entry code is modified. The behavior pattern is binary subtracted from the previous stored pattern. The resulting bit pattern is analyzed.
- [0067] The file's entry code, CRC and header fields are the same, but the file is renamed. No other fields are modified.
- [0068] The file's behavior pattern is found in the database and matches a known viral behavior pattern.
- [0069] The file's behavior pattern is found in the database and matches a known benign behavior pattern.
[0070] The program is virtualized if the executable part of the file is modified. A file that does not contain modified executable code cannot contain a virus, unless the original file was infected. If this is the case, a previous analysis would have detected the virus. When an existing program is updated, its function remains the same, and therefore its behavior pattern closely matches its stored behavior pattern. If the altered bits indicate that an infection procedure has been added then the file is considered as infected.
[0071] Two detection mechanisms operate side-by-side, both using the behavior pattern:
Pre-Infection Detection
[0072] This is the most desirable case. In pre-infection detection, the behavior pattern is analyzed and is found to represent viral behavior for those new or modified programs introduced to the system. The program file under evaluation can be repaired by removing the virus or erased if the virus infection proves too difficult to remove or if parts of the original code were overwritten. The infected program has not yet been executed on the physical PC at this time and so nothing need be done to repair the physical PC after discovery of the virus.
Post-Infection Detection
[0073] Post-infection detection takes place in cases when initial infection is missed by pre-infection detection. A virus could be missed by pre-infection detection when it does not perform any viral function on first execution and does not modify interrupt vectors that point to an infection routine. This is the case with so-called slow infectors and similarly behaving malignant code. In post-infection detection the virus is caught the moment it attempts to infect the first executable on the PC. The file hook mechanism detects this attempted change to an executable (including documents). The ABM engine then analyzes the first executable program and finds that its behavior pattern is altered in a manner indicating that a virus is active.
Database Structure:
- [0074] File ID area: Behavior pattern, program name, file size and path.
- [0075] Repair Structures Header fields, section table and relocation tables.
- [0076] Segment tables Size and Offset of each section in the section table (Windows programs only).
[0077] Macro viruses in documents are treated as if they were executables. The original Visual Basic code is recovered by decryption (where applicable) and reverse compiling the Visual Basic document (COM) stream. The resulting source code is neither saved nor shown to protect the rights of the original publishers of legitimate Visual Basic software. After virtualization the source code is discarded.
[0078] One drawback to the described virus detection system is that the initial analysis is slower than pattern scanning. This drawback is more than offset by the advantages of the system. Using file system hooking means all new files are reported and analyzed ‘on the fly’ in background. This means that once a computer is virus-free, a complete scan is typically not required again, unless the protection system has been deactivated during a period in which new programs have been installed. In signature scanning based protection systems, the computer needs to be completely rescanned every time the virus signature database is updated. Unaltered files are not again virtualized when the user initiates subsequent disk scans, so that the process is at least as fast as pattern scanning, but with a higher degree of security. The stored information also helps to repair viral damage to files or system areas, securing complete or effectively complete recovery in most cases.
[0079] In tests of a prototype implementation ABM system, the combination of pre-infection (96%) and post-infection detection (4%) resulted in 100% detection of all known viral techniques, using a combination of new, modified and well-known viruses. Other methods detected only 100% of known viruses and scored as low as 0% for the detection of new, modified and unknown viruses. No exact figure can be quoted for tests involving signature scanner based products. The results for such products are a direct representation of the mix of known, modified and new, unknown viruses; e.g. if 30% of the virus test set is new, modified or unknown then the final score reflected close to 30% missed viruses. No such relationship exists for the implementations of preferred aspects of the present system, where the detection efficiency does not appreciably vary for alterations of the presented virus mix.
[0080] The present invention has been set forth with reference to certain particularly preferred embodiments thereof. Those of ordinary skill in the art will appreciate that the present invention need not be limited to these presently preferred embodiments and will understand that various modifications and extensions of these embodiments might be made within the general teachings of the present invention. Consequently, the present invention is not to be limited to any of the described embodiments but is instead to be defined by the claims, which follow.
Claims
I claim:
1. A method for determining that a computer program, targeted for execution in a real computer with a first operating system, is malicious, the method comprising the steps of:
a virtual PC of the real computer virtually executing the target program, the virtual PC having a second, virtual operating system that controls execution of the target program in the virtual PC, the virtual operating system simulating functionality of operating system data areas and an operating system application program interface for the virtually executing target program, virtual execution of the target program within the virtual PC generating flags representing respective functions performed by the target program;
the virtual PC tracking the flags to determine the functions performed by the target program and a sequence in which the functions are called by the target program during the virtual execution; and
upon completion of the virtual execution of the target program, the virtual PC determining that the target program is malicious based on the tracked functions performed by the target program and the sequence in which the functions are called by the target program.
2. The method of claim 1, further comprising the step of the real computer creating the virtual PC, and wherein creating the virtual PC further comprises: the real computer creating a virtual central processing unit; the real computer creating a virtual memory; the real computer creating the virtual operating system, and the real computer creating a program loader.
3. The method of claim 1, further comprising the steps of:
the real computer extracting an entry point code from the target program; and
the real computer loading the extracted entry point code into the virtual PC at a correct simulated offset.
4. The method of claim 1, wherein virtually executing a target program within a virtual PC of a real computer further comprises the virtual PC virtually executing a target program that comprises at least one of a DOS, CP/M, COM, EXE, Windows executable, OS/2 executable, 32-bit executable, OLE compound, binary image, and a driver file.
5. The method of claim 1, wherein the virtual PC simulates functionality of input/output ports.
6. The method of claim 1, wherein the virtual operating system simulates an application program interface call of the first operating system by returning a correct value to the call without completing actual performance of the call.
7. A computer program product for determining that a computer program, targeted for execution in a real computer with a first operating system is malicious, the computer program product comprising:
a computer readable storage media;
first program instructions to virtually execute the target program within a virtual PC of the real computer, the virtual PC having a second, virtual operating system that controls execution of the target program in the virtual PC, the virtual operating system simulating functionality of operating system data areas and an operating system application program interface for the virtually executing target program, virtual execution of the target program within the virtual PC generating flags representing respective functions performed by the target program;
second program instructions to track the flags within the virtual PC to determine the functions performed by the target program and a sequence in which the functions are called by the target program during the virtual execution; and
third program instructions to determine by the virtual PC, upon completion of the virtual execution of the target program, that the target program is malicious based on the tracked functions performed by the target program and the sequence in which the functions are called by the target program,
wherein the first, second, and third program instructions are stored on the computer readable storage media.
8. The computer program product of claim 7, wherein the third program instructions to determine that the target program is malicious further comprises fourth program instructions to compare the generated flags to a stored pattern representative of operations by malicious code.
9. The computer program product of claim 7, wherein the target program is virtually executed each time the target program is modified and flags generated during a virtual execution of the modified target program are compared to flags generated during a virtual execution of the target program prior to modification to determine that the modified target program is malicious.
10. A computer system for determining that a computer program, targeted for execution in the computer system, is malicious, the computer system comprising:
a CPU, a computer readable memory, a computer readable storage media, and a first operating system;
first program instructions to virtually execute the target program within a virtual PC of the computer system, the virtual PC having a second, virtual operating system that controls execution of the target program in the virtual PC, the virtual operating system simulating functionality of operating system data areas and an operating system application program interface for the virtually executing target program, virtual execution of the target program within the virtual PC generating flags representing respective functions performed by the target program;
second program instructions to track the flags within the virtual PC to determine the functions performed by the target program and a sequence in which the functions are called by the target program during the virtual execution;
third program instructions to determine by the virtual PC, upon completion of the virtual execution of the target program, that the target program is malicious based on the tracked functions performed by the target program and the sequence in which the functions are called by the target program; and
wherein the first, second, and third program instructions are stored on the computer readable storage media for execution by the CPU via the computer readable memory.
11. The computer system of claim 10, wherein the third program instructions to determine that the target program is malicious further comprises fourth program instructions to compare the generated flags to a stored pattern representative of operations by malicious code.
12. The computer system of claim 10, wherein the target program is virtually executed each time the target program is modified and flags generated during a virtual execution of the modified target program are compared to flags generated during a virtual execution of the target program prior to modification to determine that the modified target program is malicious.
13. The method of claim 1, wherein the step of determining that the target program is malicious further comprises the virtual PC comparing the generated flags to a stored pattern representative of operations by malicious code.
14. The method of claim 1, wherein the target program is virtually executed each time the target program is modified and flags generated during a virtual execution of the modified target program are compared to flags generated during a virtual execution of the target program prior to modification to determine that the modified target program is malicious.
15. The computer program product of claim 7, wherein the virtual operating system simulates an application program interface call to the first operating system by returning a correct value to the call and without completing actual performance of the call.
16. The computer system of claim 10, further comprising fourth program instructions to simulate an application program interface call to the first operating system of the computing system by returning a correct value to the call without completing actual performance of the call.
17. The method of claim 1, wherein the step of determining whether the target program is malicious further comprises the virtual PC analyzing the tracked functions to determine if the tracked functions comprise destructive content.
18. The method of claim 1, wherein the step of determining whether the target program is malicious further comprises the virtual PC determining whether the generated flags comprise an indication that a user has not had an opportunity to interact with a process of the target program.
19. The method of claim 1, wherein the step of determining whether the target program is malicious further comprises the virtual PC determining whether the generated flags comprise an indication of a detection avoidance routine.
20. The method of claim 1, further comprising the step of the virtual PC calling one or more interrupts that the target program has modified within the virtual PC and the virtual PC generating interrupt flags for each of the called interrupts,
wherein the step of determining whether the target program is malicious further comprises the virtual PC determining whether the generated interrupt flags indicate malicious code in the target program.
Patent Citations (101)
| Patent | Date | Inventor | Cited By |
|---|---|---|---|
| US4400769(A) | 1983-08-01 | Kaneda et al. | Applicant |
| US4819234(A) | 1989-04-01 | Huber | Applicant |
| US4975950(A) | 1990-12-01 | Lentz | Applicant |
| US5121345(A) | 1992-06-01 | Lentz | Applicant |
| US5278901(A) | 1994-01-01 | Shieh et al. | Applicant |
| US5345595(A) | 1994-09-01 | Johnson et al. | Applicant |
| US5398196(A) | 1995-03-01 | Chambers | Examiner |
| US5414833(A) | 1995-05-01 | Hershey et al. | Applicant |
| US5440723(A) | 1995-08-01 | Arnold et al. | Applicant |
| US5475839(A) | 1995-12-01 | Watson et al. | Applicant |
| US5586260(A) | 1996-12-01 | Hu | Applicant |
| US5590331(A) | 1996-12-01 | Lewis et al. | Applicant |
| US5606668(A) | 1997-02-01 | Shwed | Applicant |
| US5623600(A) | 1997-04-01 | Ji et al. | Applicant |
| US5623601(A) | 1997-04-01 | Vu | Applicant |
| US5630061(A) | 1997-05-01 | Richter et al. | Applicant |
| US5649095(A) | 1997-07-01 | Cozza | Applicant |
| US5675711(A) | 1997-10-01 | Kephart et al. | Applicant |
| US5696822(A) | 1997-12-01 | Nachenberg | Applicant |
| US5761504(A) | 1998-06-01 | Corrigan et al. | Applicant |
| US5764887(A) | 1998-06-01 | Kells et al. | Applicant |
| US5764890(A) | 1998-06-01 | Glasser et al. | Applicant |
| US5765030(A) | 1998-06-01 | Nachenberg et al. | Applicant |
| US5774727(A) | 1998-06-01 | Walsh et al. | Applicant |
| US5787177(A) | 1998-07-01 | Leppek | Applicant |
| US5796942(A) | 1998-08-01 | Esbensen | Applicant |
| US5798706(A) | 1998-08-01 | Kraemer et al. | Applicant |
| US5815574(A) | 1998-09-01 | Fortinsky | Applicant |
| US5826013(A) | 1998-10-01 | Nachenberg | Applicant |
| US5828833(A) | 1998-10-01 | Belville et al. | Applicant |
| US5832208(A) | 1998-11-01 | Chen et al. | Applicant |
| US5832211(A) | 1998-11-01 | Blakley, III et al. | Applicant |
| US5835726(A) | 1998-11-01 | Shwed et al. | Applicant |
| US5838903(A) | 1998-11-01 | Blakely, III et al. | Applicant |
| US5842002(A) | 1998-11-01 | Schnurer et al. | Applicant |
| US5854916(A) | 1998-12-01 | Nachenberg | Applicant |
| US5857191(A) | 1999-01-01 | Blackwell, Jr. et al. | Applicant |
| US5864665(A) | 1999-01-01 | Tran | Applicant |
| US5864803(A) | 1999-01-01 | Nussbaum | Applicant |
| US5872978(A) | 1999-02-01 | Hoskins | Applicant |
| US5875296(A) | 1999-02-01 | Shi et al. | Applicant |
| US5881236(A) | 1999-03-01 | Dickey | Applicant |
| US5884033(A) | 1999-03-01 | Duvall et al. | Applicant |
| US5907834(A) | 1999-05-01 | Kephart et al. | Applicant |
| US5950012(A) | 1999-09-01 | Shiell et al. | Applicant |
| US5964839(A) | 1999-10-01 | Johnson et al. | Applicant |
| US5964889(A) | 1999-10-01 | Nachenberg | Examiner |
| US5978917(A) | 1999-11-01 | Chi | Applicant |
| US5983270(A) | 1999-11-01 | Abraham et al. | Applicant |
| US5983348(A) | 1999-11-01 | Ji | Applicant |
| US5987606(A) | 1999-11-01 | Cirasole et al. | Applicant |
| US5987610(A) | 1999-11-01 | Franczek et al. | Applicant |
| US5987611(A) | 1999-11-01 | Freund | Applicant |
| US5999723(A) | 1999-12-01 | Nachenberg | Applicant |
| US6003132(A) | 1999-12-01 | Mann | Applicant |
| US6014645(A) | 2000-01-01 | Cunningham | Applicant |
| US6016553(A) | 2000-01-01 | Schneider et al. | Applicant |
| US6021510(A) | 2000-02-01 | Nachenberg | Applicant |
| US6026442(A) | 2000-02-01 | Lewis et al. | Applicant |
| US6029256(A) | 2000-02-01 | Kouznetsov | Applicant |
| US6035423(A) | 2000-03-01 | Hodges et al. | Applicant |
| US6041347(A) | 2000-03-01 | Harsham et al. | Applicant |
| US6061795(A) | 2000-05-01 | Dircks et al. | Applicant |
| US6067410(A) | 2000-05-01 | Nachenberg | Applicant |
| US6081894(A) | 2000-06-01 | Mann | Applicant |
| US6085224(A) | 2000-07-01 | Wagner | Applicant |
| US6088803(A) | 2000-07-01 | Tso et al. | Applicant |
| US6092194(A) | 2000-07-01 | Touboul | Applicant |
| US6094731(A) | 2000-07-01 | Waldin et al. | Applicant |
| US6098173(A) | 2000-08-01 | Elgressy et al. | Applicant |
| US6108799(A) | 2000-08-01 | Boulay et al. | Applicant |
| US6118940(A) | 2000-09-01 | Alexander, III et al. | Applicant |
| US6119165(A) | 2000-09-01 | Li et al. | Applicant |
| US6119234(A) | 2000-09-01 | Aziz et al. | Applicant |
| US6122738(A) | 2000-09-01 | Millard | Applicant |
| US6154844(A) | 2000-11-01 | Touboul et al. | Applicant |
| US6167520(A) | 2000-12-01 | Touboul | Applicant |
| US6192512(B1) | 2001-02-01 | Chess | Applicant |
| US6199181(B1) | 2001-03-01 | Rechef et al. | Applicant |
| US6266774(B1) | 2001-07-01 | Sampath et al. | Applicant |
| US6272641(B1) | 2001-08-01 | Ji | Applicant |
| US6275938(B1) | 2001-08-01 | Bond et al. | Applicant |
| US6324627(B1) | 2001-11-01 | Kricheff et al. | Applicant |
| US6338141(B1) | 2002-01-01 | Wells | Applicant |
| US6357008(B1) | 2002-03-01 | Nachenberg | Examiner |
| US6397242(B1) | 2002-05-01 | Devine et al. | Applicant |
| US6405318(B1) | 2002-06-01 | Rowland | Applicant |
| US6775780(B1) | 2004-08-01 | Muttik | Applicant |
| US6971019(B1) | 2005-11-01 | Nachenberg | Examiner |
| US6973577(B1) | 2005-12-01 | Kouznetsov | Examiner |
| US7093239(B1) | 2006-08-01 | van der Made | Examiner |
| US7185367(B2) | 2007-02-01 | Munson | Applicant |
| US7370360(B2) | 2008-05-01 | van der Made | Examiner |
| US2002/0083334(A1) | 2002-06-01 | Rogers et al. | Applicant |
| EP636977(A2) | 1995-02-01 | Applicant | |
| WO/15966 | 1999-04-01 | Applicant | |
| WO/54458 | 2000-09-01 | Applicant | |
| WO1/84285 | 2001-11-01 | Applicant | |
| WO2/06928 | 2002-01-01 | Applicant | |
| WOPCT/US01/26804 | 2002-03-01 | Applicant | |
| WO2/056152 | 2002-07-01 | Applicant |
Non-Patent Literature (21)
- Jieh-Sheng Lee, “A Generic Virus Detection Agent on the Internet”, IEEE, 1997, pp. 210-219.Examiner
- Arnold, WC Chess, DM, et al., Automated Program Analysis for Computer Virus Detection, Jul. 1991, IBM Technical Disclosure Bulletin, pp. 415-416. [n2, pp. 415-416 teaches interpreter/simulator that simulates the execution of a given “COM” or “EXE” file].Applicant
- Arnold, WC Chess, DM, et al., Employment of Virus Detection Procedures at Domain Boundaries, Dec. 1991, IBM Technical Disclosure Bulletin, pp. 199-200. [n7a pp. 199-200 teaches virus detection procedures at domain boundaries].Applicant
- Arnold, WC Chess, DM, System for Detecting Undesired Alteration of Software, Apr. 1990, IBM Technical Disclosure Bulletin, pp. 48-50. [n11 pp. 48-50 teaches detecting undesired alteration of software].Applicant
- “Advanced Virus Detection Technology for the Next Millennium,” Aug. 1999, Network Associates, A Network Associates Executive White Paper, pp. 1-14.Applicant
- “Enterprise-Grade Anti-Virus Automation in the 21st Century,” Jun. 2000, Symantec, Technology Brief, pp. 1-17.Applicant
- Jeffrey O. Kephart, et al., “Blueprint for a Computer Immune System,” [online] 1997 [accessed Oct. 6, 2003], Retrieved from Internet <URL: http://www.research.ibm.com/antivirus/SciPapers/Kephart/VB97/>, pp. 1-15.Applicant
- Baudouin Le Charlier, et al., “Dynamic Detection and Classification of Computer Viruses Using General Behavior Patterns,” 1995, Proceedings of the fifth International Virus Bulletin Conference, Boston, pp. 1-22.Applicant
- Robert Richardson, “Enterprise Antivirus Software,” [online] Feb. 2000 [accessed Oct. 6, 2003], Retrieved from Internet <URL: http//www.networkmagazine.com/article/NMG20000426S0006>, pp. 1-6.Applicant
- “Understanding and Managing Polymorphic Viruses,” 1996, Symantec, The Symantec Enterprise Papers, vol. XXX, pp. 1-13.Applicant
- Andrew P. Kosoresow, et al., “Intrusion Detection Via System Call Traces,” IEEE Software, pp. 35-42, Sep./Oct. 1997.Applicant
- Frans Veldman, “Heuristic Anti-Virus Technology,” Proceedings, 3rd International Virus Bulletin Conference, pp. 67-76, Sep. 1993.Applicant
- Symantec, Understanding Heuristics: Symatec's Bloodhound Technology, Symantec White Paper Series, vol. XXXIV, pp. 1-14, Sep. 1997.Applicant
- Carey Stover Nachenberg, “A New Technique for Detecting Polymorphic Computer Viruses,” A thesis submitted in partial satisfaction of the requirements for the degree Master of Science in Computer Science, University of California Los Angeles, pp. I-127, 1995.Applicant
- “Microsoft P-Code Technology,” http://msdn.microsoft.com/archive/default.asp?url=/archive/en-us/dnarvc/html/msdn—c7pcode2.asp, pp. 1-6, Apr. 1992.Applicant
- “DJGPP COFF Spec,” http://delorie.com/digpp/doc/coff/, pp. 1-15, Oct. 1996.Applicant
- Natvig, Kurt, “Sandbox Technology Inside AV Scanners,” Virus Bulletin Conference, Sep. 2001, pp. 475-488.Applicant
- “Norman introduces a new technique for eliminating new computer viruses,” found on Norman's website, file://C:\Documents%20and%20Settings\7489\Local%20Settings\Temporary%20Internet%20Files\OLK, pp. 1-2, published Oct. 25, 2001, printed from website Dec. 27, 2002.Applicant
- Lee et al., “A Generic Virus Detection Agent on the Internet,” System Sciences, 1997, Proceedings of the Thirtieth Hawaii International Conference on Wailea, HI, Jan. 7-10, 1997, pp. 210-219, #XP010271868.Applicant
- International Search Report for PCT/US01/19142, international filing date Jun. 14, 2001, mailing date Jan. 17, 2003.Applicant
- Kephart, “A Biologically Inspired Immune System for Computers,” Artificial Life, IV, 1994, pp. 130-139.Applicant