System and method for real-time detection of computer system files intrusion

[0001] The present invention relates to computer security in general and in particular to real-time detection of computer system files modifications.

[0002] Typically, any computer system operated by a user in a standard usage calls for a number of system files, including program files and data files, among which some are not supposed to be modified frequently which are called here vital files. This is also the case for other devices operating in an unattended way, such as the so-called, NFS and other file servers, DNS/BIND and other name servers, database servers, communication servers, gateways, web servers, and any type of server using a computing device driven by an operating system.

[0003] In a network environment, a System Administrator is the only person authorized to modify those vital programs and system files either on the computer itself or from a centralized monitoring computer (Manager/Administrator machine). More and more crackers, hostile hackers or intruders try to compromise these files in order to gain later access to the system. New system leaks are regularly discovered, leading to so-called exploits allowing almost anybodyfrom the inside or from the outside of a Companywith or without gaining the System Administrator privileges for a while. Those means may includebut not limited torace conditions, buffer overflows, stack overflows, etc.

[0004] As soon as the intruder gains the System Administrator privilege access, he/she will modify some system files and/or data files in order to enter the system more easily at a later time, should the breach he came in by be filled in.

[0005] State of the art of solutions to detect such kind of intrusion is first to compute the well-known MD5 signature of all the vital files, at computer installation time and before the computer is connected to any network, and then to store these computed MD5 signatures in a safe place (e.g. on a diskette which will be set read-only once written, on a CD/R, or by sending them securely on another computer on the network, either in a crypted form than can be cross-checked, or preferably in a write once/read many mode). Complete details on computation of MD5 signature may easily be found in the literature and particularly in The MD5 Message-Digest Algorithm (Rivest, R., and S. DusseRFC 1321MIT Laboratory for Computer Science, RSA Data Security Inc., April 1992).

[0006] U.S. Pat. N^o 5,440,723 from the Assignee discloses an automatic immune system for computers and computer networks. In the operated method, periodic monitoring of a data processing system to check for anomalous behavior that may indicate the presence of an undesirable software entity and automatic scanning for occurrences of known types of undesirable software entities are performed. Remedial action are taken if they are discovered, among which the extraction and identification of signature from the executable code portion and the adding of the signature to a signature database. The immediate use of the signature by a scanner provides protection from subsequent infections of the system, and also a network of systems, by the same or an altered form of the undesirable software entity.

[0007] MD5 signature based solutions are implemented on some known commercial products such as Tripwire, a free software in its former open-source versions and a commercial software in its present licensed ones, or such as the Enterprise Security Manager product from Symantec Corporation.

[0008] Using the MD5 signature is an interesting approach, because changing just one bit in a file totally changes its MD5 signature, but also because there is no way for the intruder to generate a new file having the same MD5 signature. So if a file is modified by an intruder, its MD5 signature is necessarily changed. However, it remains that one has to check all the files on a regular basis, by applying an auto-checking procedure on each computer and collecting the results on a security log that is to be analyzed by the System Administrator or by a software tool. The drawbacks of this approach is that if the checking is done too often, the computer resources are intensively solicited, and on the contrary if the checking is made at larger time intervals, the intruder will have more available time to experiment with the compromised system before he/she is detected.

[0009] In the majority of the client environments as illustrated on FIG. 1-a these checking are done every night, for example around 3 a.m., thereby leaving on the average twelve hours to the intruder to both cause a lot of trouble on the system. Moreover he/she gets also a lot of time to restore a sane situation before the checking begins. Even the system time stamps associated the last operations on each file can be modified when the intruder has System Administrator's access, thereby covering his/her traces. The local system logs have the same vulnerability.

[0010] Thus, there is a need for a system and method detecting offensive intrusions, that does not consume resources uselessly and that operates in real-time, by the use of a system interrupt whenever a vital file is accessed for modifications.

[0011] The subject invention herein solves the aforementioned problems in a new and unique manner that has not been part of the art previously.

[0012] It is therefore the object of the invention to provide a system and method to detect in real-time all modifications of vital files on a computer system.

[0013] In accordance with a preferred embodiment, the present invention is operated in a computer network having an administrator computer coupled to at least one user computer. The user computer is having at least one application program interface that generates a system call for any event related to the user computer's regular system files. The method consists in detecting in real-time any access to vital system files of the user computer and comprises the steps of:

[0014] a. at boot time, declaring to the at least one application program interface the vital system files as a protected set of system files;

[0015] b. detecting any system call related to the protected set of system files;

[0016] c. upon the detection of at least one system call:

[0017] simultaneously, raising an automatic interrupt I command and sending a message to the administrator computer; and

[0018] identifying which at least one vital system file of the protected set of system files is pointed by the at least one system call.

[0019] The invention may be operated on any computer system running on any operating system and coupled to an administrator computer through any kind of network link.

[0020] The method may be implemented on any computer system using a computer program product stored on computer usable medium and comprising computer readable program means for causing the computer system to perform the method as described in the appended claims.

[0021] The above and other objects, features and advantages of the invention will be better understood by reading the following more particular description of the invention in conjunction with the accompanying drawings wherein:

[0022] FIGS. 1-a and 1-b show simplified views of a checking computer environment in prior art and in the present invention.

[0023] FIG. 2 is a more detailed view of a computer network environment in which this invention applies.

[0024] FIG. 3 is a flow chart depicting the basic intrusion detection sequence according to the invention.

[0025] Before referring to the figures, it is to be noted that the inventors have made use for the security purpose of the present invention, of a feature that is already present in computer systems but for a completely different use that has no relation at all to security. The novel concept behind the invention is to take advantage of existing Application Programming Interface (API) dedicated to the creation, modification and deletion of files in order to detect in real-timethat is, exactly at the very time it occursan illegal modification of any vital file or directory of a computer system.

[0026] The creation or deletion of a file is an operation that implies modifications of another file generally called its directory. So the term file in this description is used to indicate either a simple file, i.e. a file that contains either data or programs, or a directory file, i.e. a parent file that contains both the name and location of a plurality of simple files.

[0027] The APIs used may be the well-known ChangeNotify API of the WINDOWS operating systems (WINDOWS is a trademark of Microsoft Corp.) or those equivalent System Calls of others operating systems, such as LINUX (LINUX is a trademark of Linus Torvalds)where an equivalent API, used by the graphical interface (named KDE3) is called dnotify in the 2.4 kernelor such as any UNIX (UNIX is a trademark of UNIX System Laboratories) or UNIX-like (for instance POSIX-compliant) system using the FAM (File Alteration Monitor) developed by SGI (SGI is a trademark of Silicon Graphic Inc.) for accurate real-time window contents management, or any similar APIs used to synchronize the file information displayed in computer windows with the real files on the system.

[0028] The skilled person at the lecture of the following preferred embodiment which is made in the Windows environment will easily apply all the principles to systems, open-source or not, where the ChangeNotify API or one of its aforementioned equivalents does not exist. For these systems, a simple modification of the kernel open routine by any team having access to the source code would provide the necessary interrupt command to operate the present method. The kernel open is, in any operating system, the mandatory point of passage to allow an application, whether running from superuseralso called administratormode or regular user mode to open any file on that operating system.

[0029] The ChangeNotify is thus the Windows API that monitors events related to files and directories. The ChangeNotify API informs the kernel of an active computer to send a signal to an API caller (in Windows normal operations this caller is the graphic interface itself) whenever a file or a directory that is specified for that API is modified so that any window displaying it can be warned to reflect that change. The kernel is the core of the operating system and acts between user programs and hardware devices. The kernel is started at boot time and, while running, cannot be modified on the fly. So any information stored in the kernel itself (or in a table handled within the kernel space) can be considered immune to attacks.

[0030] Referring now to the drawings, and first to FIG. 1-b, a conceptual scheme illustrates the innovative approach proposed by the inventors which highlights the main differences as regard to prior art previously discussed with FIG. 1-a. It is first to be appreciated that the user machines connected to the Manager machine may be either in an attended or in an unattended mode and thus the unattended mode required by the prior art is no more a limitation with the use of the present intrusion detection process. If an intruder either directly from a user machine or remotely through a network connection modifies a vital file, an alert is immediately and automatically raised and sent to the Manager machine to inform the System Administrator of the modification whatever the status of the user machine. It is also to be appreciated that the user machine may have various means, lately discussed to inform the user, the Administrator of the intrusion.

[0031] FIG. 2 is a general view of a Windows OS computer network environment in which this invention may apply. As already shown on FIG. 1-b, there is an Administrator machine 200 that is able to collect and store in a database (216) integrity information about a plurality of internal and/or external user computer systems (202,204,206).

[0032] The internal user computer systems may be office computers (202) connected to the administrator machine by an internal network (208) such as any Local Area Network. The external user computer systems may be home-office computers (204) or nomad computers (206 for traveling users) connected by the Internet (210) through links such as high speed link, cable modem or dial-up link.

[0033] Each user computer system may initialize its operating system (212) at boot time to allow application programs to become active and those application programs perform calls to invoke the procedures of the APIs (214). Among those calls, one may invoke the ChangeNotify API and declare to it which vital regular files or directories will have to be monitored for changes from that moment on. In the case of directories, the call may also declare whether this monitoring also applies to the whole set of files andrecursivelysub directories it refers to, which will be the general case.

[0034] Examples of such directories whose modification attempts should be monitored in real-time are WinNT (including its files and all sub directories among which WinNTCommand) in Windows NT or 2000 systems. Similarly, the /bin and /etc in Linux and other UNIX-like systems, and directories containing either vital programs or vital data in so-called proprietary systems should also be monitored in real-time.

[0035] The call may declare which interrupt should be raised when such modification occur, as it will be detailed later. These operations are performed at boot time before starting any network program to ensure that no corruption can occur from the outside at that time.

[0036] FIG. 3 is a flow chart depicting the basic intrusion detection sequence according to the invention. At initialization time of a user computer (boot time 300), a set of vital files VF is sent to a series of ChangeNotify or equivalent APIs (box 302). The list of the vital files is declared in a file or a database to be further described.

[0037] Thereafter, the user may work on its computer and operate any action. When a delete or open or modification system call is detected in real-time on one of the vital files (box 304) an automatic interrupt I command is raised (box 306). This interrupt is caught by an asynchronous routine (also known as a daemon) which raises an alert within the system in real time (box 308). Optionally, the same information may also be sent to another system in real-time for example to determine if a repair of the just changed file is necessary or not, i.e. If it is an unauthorized change.

[0038] Identification of the files to be monitored is kept in either a flat file or in a database (216), according to both the needed monitoring granularity and the ease of maintenance. Preferably, the flat file is crypted for more safeness. The whole structure is loaded into a table under control of the kernel located in a kernel space. The structure each entry in the flat file or database as well as in the table may preferably be the following one:full path of what is to be monitored, indication whether this specification is recursivethat is, applies to all sub directories, and indication whether the entry specifies that the file should be included or excluded of the monitored file list. The latter is useful for a class of programs which keeps user data and/or some customization files in one of their own sub directories instead of using the registry and the My Documents directory in Windows, or the /home directory in Linux and Unix-like systems.

[0039] Both the table or database and the code that uses it belong to the files registered at boot time as monitored files, so an intruder has no hope of fooling the system by changing them, even if the intruder knows either the key used to crypt the flat file or the database password depending on the solution chosen. This enforces security in a considerable way.

[0040] As previously said, the alert may be raised in any form and in particular such as a message displayed on the computer display or on an alternate screen or on a printer, but also by a network card or a modem, or by an audible alarm too and so on. The user being warned in real-time that the computer's vital files are being modified at a time when they should not, may take the decision to unplug its computer immediately from the network to prevent further damage. However, as such a modification can be done at any time particularly when the user is not in front of the computer, the present invention may include additional mechanism to detect which files were modified.

[0041] In normal mode the ChangeNotify or the dnotify API informs the daemon that was registered to this API that something has changed, but nothing is more specified. When these APIs are used to watch a mere directory in a window it is not important because the whole directory can then be rescanned in less than one second. Moreover, these APIs are not normally used to detect malicious modifications, but just regular operations, so they rely on the system's datetime stamp, which an intruder in some cases may manage to leave unchanged though the contents of the file did change.

[0042] The problem of file identification is enlarged in an environment where thousand of system files that contained in a complex hierarchy of directories, plus a MD5 signature to compute on every one of them are to be watched. Moreover, by the time a complete inspection of these files is made, some other files may have been modified as well, making the first report unreliable itself.

[0043] In a first embodiment, a plurality of daemons as many as there are unique directorieswithout recursionto watch, are launched at boot time in order to have as many daemons as sub directories, plus one. Whenever a daemon is awaken by interrupt I, this daemon knows which precise one-level directory it is watching, and just has to scan this one. This allows a sub-second response time at the reasonable cost of multiplying the number of instances of a daemon. The inventors have checked that a 1400 MHz AMD will typically compute 100 MD5 signatures per second on a set of 1 MB files. The cost is reasonable because the daemon instances consume no CPU time as long as they are not awaken, and share the same code and common data, only their personalized, also known as own data, using memory, a resource which is now very cheap on computers, and that will not differ much whether the list of files is kept in one daemon or splitted into many daemons.

[0044] If many daemons, they can also be programmed to monitor one another, just to ensure none of them has been killed for a while. Daemons will restart themselves automatically under control of the operating system when so defined, but any daemon uses at startup time its configuration data, which could be modified during a specific short time, the time between the moments when the daemon is killed and restarted in a particular case:when that daemon is the one that monitors the set of vital files. For that reason, the daemon names are preferably chosen at random at boot time so an intruder cannot determine which exact daemon monitors the security files.

[0045] In another embodiment, only one daemon is launched at boot time. When an interrupt I is raised, the daemon immediately rearms the interrupt for a next one to occur, and becomes in a sleep mode during which any new interrupt is catched. The set of files are scanned to determine the modifications only at the end of the sleep mode. Setting the duration of the sleep mode as twice the time needed to compute all the MD5 signatures is a preferred choice, though other can be used as well depending on the mean machine chargethe bigger the charge, the greater duration. Finally a basis for the duration of the sleep mode at low charge can be computed at boot time and its value adjusted according to that boot time value, the load factor of the system, and the maximum accepted time to get a first list of modified files.

[0046] The present description has been presented to enable one or ordinary skill in the art to make and use the invention and has been provided in the context of a patent application and its requirements. Various modifications to the preferred embodiment and the generic principles and features described herein will be readily apparent to those skilled in the art. Thus, the present invention is not intended to be limited to the embodiment shown but is to be accorded the widest scope consistent with the principles and features described herein.