Method and system for protecting a security parameter negotiation server against denial-of-service attacks

[0001] This invention relates generally to network communications, and more particularly to security threats to communication servers in a network environment.

[0002] The Internet has entered the new millenium as the most important computer network of the world. Everyday, millions of people use the Internet to communicate with each other and to gather or share information. Moreover, electronic commerce (“E-commerce”) using the World-Wide Web (WWW) of the Internet as its backbone is rapidly replacing and changing the conventional brick-and-mortar stores.

[0003] The security of communications through the Internet, however, has always been a major concern. This problem is related to the underlying network communication protocol of the Internet, the Internet Protocol (“IP”), which is responsible for delivering packets across the Internet to their destinations. The Internet Protocol was not designed to provide security features at its level of network communication operation. Moreover, the flexibility of IP allows for some creative uses of the protocol that defeat traffic auditing, access control, and many other security measures. IP-based network data is therefore wide open to tampering and eavesdropping. As a result, it substantial risks are involved in sending sensitive information across the Internet.

[0004] To address the lack of security measures of the Internet Protocol, a set of extensions called Internet Protocol Security (“IPSec”) Suite has been developed to add security services at the IP level. The IPSec Suite includes protocols for an authentication header (AH), encapsulating security protocol (ESP), and a key management and exchange protocol (IKE). A significant advantage of the IPSec Suite is that it provides a universal way to secure all IP-based network communications for all applications and users in a transparent way. Moreover, as the IPSec Suite is designed to work with existing and future IP standards, regular IP networks can still be used to carry communication data between the sender and recipient. The IPSec Suite is also scalable and can therefore be used in networks ranging from local-area networks (LANS) to global networks such as the Internet.

[0005] Even though the IPSec standard provides a comprehensive and robust way to secure network communications against tampering and eavesdropping, the components implementing the IPSec Suite themselves may be subjected to various security threats in the network environment. For instance, the IPSec layer includes a component called an “Internet Key Exchange” (“IKE”) server, which is responsible for negotiating with another IKE for security parameters, collectively called a “Security Association” (“SA”), of security operations for securing a given network communication stream. For each secured communication stream, a separate SA has to be negotiated and maintained. Because of the system resources required for handling each communication requests, it is possible for an attacker to construct and send a large number of false communication requests, forcing the IKE server to consume large amounts of system resources. Such an attack potentially can burden the server to the extent that it is no longer able to serve legitimate users.

[0006] In view of the foregoing, the present invention provides a method and system for protecting a network security server for negotiating network security parameters, such as an Internet Key Exchange (“IKE”) server of the IPSec suite, from denial-of-service attacks that flood the server with false connection requests. The vulnerability of the security server to such attacks comes from the need for the server to maintain state data for on-going negotiations in response to requests from unknown clients. In accordance with the invention, the resilience of the negotiation server to such attacks is significantly enhanced by dynamically detecting the degradation of the performance of the system, and dynamically adjusting relevant performance variables, such as negotiation delay, retransmission delay, and packet drop percentage, etc., to reduce the states maintained by the negotiation server when performance degradation is detected. A useful indicator of the system health may be the interval between consecutive runs of a reaper for removing states that are no longer useful. To further enhance the effectiveness of the protection against denial-of-service attacks, the maximum number of states pending negotiation responses for outstanding new negotiation requests from a client may be limited, and the total number of stored states pending negotiation responses may also be limited.

[0007] Additional features and advantages of the invention will be made apparent from the following detailed description of illustrative embodiments, which proceeds with reference to the accompanying figures.

[0012] Turning to the drawings, wherein like reference numerals refer to like elements, the invention is illustrated as being implemented in a suitable computing environment. Although not required, the invention will be described in the general context of computer-executable instructions, such as program modules, being executed by a personal computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Moreover, those skilled in the art will appreciate that the invention may be practiced with other computer system configurations, including hand-held devices, multi-processor systems, microprocessor based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, and the like. The invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

[0013] With reference to FIG. 1, an exemplary system for implementing the invention includes a general purpose computing device in the form of a conventional personal computer 20, including a processing unit 21, a system memory 22, and a system bus 23 that couples various system components including the system memory to the processing unit 21. The system bus 23 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. The system memory includes read only memory (ROM) 24 and random access memory (RAM) 25. A basic input/output system (BIOS) 26, containing the basic routines that help to transfer information between elements within the personal computer 20, such as during start-up, is stored in ROM 24. The personal computer 20 further includes a hard disk drive 27 for reading from and writing to a hard disk 60, a magnetic disk drive 28 for reading from or writing to a removable magnetic disk 29, and an optical disk drive 30 for reading from or writing to a removable optical disk 31 such as a CD ROM or other optical media.

[0014] The hard disk drive 27, magnetic disk drive 28, and optical disk drive 30 are connected to the system bus 23 by a hard disk drive interface 32, a magnetic disk drive interface 33, and an optical disk drive interface 34, respectively. The drives and their associated computer-readable media provide nonvolatile storage of computer readable instructions, data structures, program modules and other data for the personal computer 20. Although the exemplary environment described herein employs a hard disk 60, a removable magnetic disk 29, and a removable optical disk 31, it will be appreciated by those skilled in the art that other types of computer readable media which can store data that is accessible by a computer, such as magnetic cassettes, flash memory cards, digital video disks, Bernoulli cartridges, random access memories, read only memories, and the like may also be used in the exemplary operating environment.

[0015] A number of program modules may be stored on the hard disk 60, magnetic disk 29, optical disk 31, ROM 24 or RAM 25, including an operating system 35, one or more applications programs 36, other program modules 37, and program data 38. A user may enter commands and information into the personal computer 20 through input devices such as a keyboard 40 and a pointing device 42. Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, scanner, or the like. These and other input devices are often connected to the processing unit 21 through a serial port interface 46 that is coupled to the system bus, but may be connected by other interfaces, such as a parallel port, game port or a universal serial bus (USB). A monitor 47 or other type of display device is also connected to the system bus 23 via an interface, such as a video adapter 48. In addition to the monitor, personal computers typically include other peripheral output devices, not shown, such as speakers and printers.

[0016] The personal computer 20 may operate in a networked environment using logical connections to one or more remote computers, such as a remote computer 49. The remote computer 49 may be another personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to the personal computer 20, although only a memory storage device 50 has been illustrated in FIG. 1. The logical connections depicted in FIG. 1 include a local area network (LAN) 51 and a wide area network (WAN) 52. Such networking environments are commonplace in offices, enterprise-wide computer networks, intranets and the Internet.

[0017] When used in a LAN networking environment, the personal computer 20 is connected to the local network 51 through a network interface or adapter 53. When used in a WAN networking environment, the person computer 20 typically includes a modem 54 or other means for establishing communications over the WAN 52. The modem 54, which may be internal or external, is connected to the system bus 23 via the serial port interface 46. In a networked environment, program modules depicted relative to the personal computer 20, or portions thereof, may be stored in the remote memory storage device. It will be appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers may be used.

[0018] In the description that follows, the invention will be described with reference to acts and symbolic representations of operations that are performed by one or more computers, unless indicated otherwise. As such, it will be understood that such acts and operations, which are at times referred to as being computer-executed, include the manipulation by the processing unit of the computer of electrical signals representing data in a structured form. This manipulation transforms the data or maintains it at locations in the memory system of the computer, which reconfigures or otherwise alters the operation of the computer in a manner well understood by those skilled in the art. The data structures where data is maintained are physical locations of the memory that have particular properties defined by the format of the data. However, while the invention is being described in the foregoing context, it is not meant to be limiting as those of skill in the art will appreciate that various of the acts and operation described hereinafter may also be implemented in hardware.

[0019] Referring now to FIG. 2, the present invention is directed to a way to protect a security parameter negotiation server, such as an IKE server of the IPSec suite, from malicious denial-of-service attacks that attempt to flood the server with false connection requests. For illustration purposes, the invention will be described below in connection with a preferred embodiment that implements the IPSec Suite protocols for secured delivery of network communications. It will be appreciated, however, that the system and method of the invention for providing protection against denial-of-service attacks can also be effectively used with other network security protocols that require negotiations of security parameters for securing network communications.

[0020] In the embodiment shown in FIG. 2, a computer 70 implements the IPSec Suite protocols for secured delivery of IP-based packets. The components supporting the IPSec protocols include a policy agent 72, an IPSec driver 74, and an Internet Key Exchange (“IKE”) server 76. The security policies assigned to the host computer 70 by the administrator of the system determine the levels of security for various types of communications. The security policies are picked up by the policy agent 72 and passed the IKE server 76 and the IPSec driver 74. The IKE server 76 uses the negotiation policies associated with the assigned security policies to conduct negotiations with a peer (i.e., the IKE component of another computer on the network) to establish security parameters for communications with the host of the peer. The negotiated security parameters include, for example, the parameters for authentication and encryption methods and the keys, and are collectively referred to in the IPSec Suite protocols as a Security Association (“SA”). The results of the negotiation by the IKE server 76 is passed to the IPSec driver 74, which performs security operations, such as data encryption, on packets of a communication stream using the negotiated SA for that stream.

[0021] In this illustrated embodiment, the host computer 70 on which the IPSec components reside is part of an internal network 80 such as a local-area network (“LAN”). The host computer 70 is also connected to an external network 82, such as the Internet, and communicates with other computers on the external network by sending and receiving packets based on the Internet Protocol. The host computer 70 in this arrangement functions as a firewall or gateway for computers on the internal network 80 to communicate with computers on the external network 82. For example, a computer 84 on the internal network may communicate with a computer 86 on the external network 82 by transmitting IP-based communication packets 92 through the host computer 70, whose IPSec components will handle the task of securing the communication stream. When the host computer is used in this capacity of a gateway for the internal network, the IPSec components, such as the IKE server 76, especially have to be resilient to attacks mounted by malicious attackers on the external network while providing services to legitimate users.

[0022] In accordance with an aspect of the invention, the IKE server 76 may be vulnerable to denial-of-service attacks that flood it with false connection requests if no special protection measure is taken. The vulnerability of the IKE server to such flooding attacks comes from the need for the IKE server to store states not only for successful negotiations but also for on-going negotiations. As shown in FIG. 2, when an initial communication request 90 comes from a computer 86 on the external network 82, the IKE server 76 initiates a negotiation process with the peer IKE 88 of the requesting computer 86 to establish the security parameters for the communication. Under the IKE protocol of the IPSec suite, this negotiation involves two phases. In the first phase, the two IKE peers 76 and 88 establish a secure channel for conducting the IKE negotiation (called the IKE SA). In the second phase, the two IKE peers negotiate general purpose SAs over the secure channel established in the first phase. The first phase is typically accomplished in a “main mode” that involves three two-way exchanges between the SA initiator and the recipient. The second phase is accomplished in a “quick mode” that is less complicated than the main mode since the negotiation is already inside a secure channel. As these phases and modes of the IKE negotiation process are defined in the IKE protocol and well known to those skilled in the art, it is not necessary to describe them in greater detail here.

[0023] It is important, however, for purposes of the invention to understand that to support the negotiation process the IKE server has to store a “state” associated with the negotiation. Specifically, when a request 90 for a new negotiation with a peer arrives, the IKE server allocates system resources to create a state for the negotiation. In this context, the state is a proposed SA to be established by negotiation. As shown in FIG. 2, the IKE server 76 maintains a state table 120 that is a list of all SAs pending negotiation as well as SAs that have been successfully negotiated. The resources allocated for the SA state include dynamically allocated memory and a critical section for synchronization. The SA state record starts with data including the peer address, an indication of whether the server is the initiator or responder of the negotiation, the current state of the negotiation (e.g., an OAK_MM_SETUP state as will be described below). As the negotiation proceeds, more information is filled into the SA state, such as the key generation data, the negotiation attributes, the authentication material, etc. At the end of the main mode phase of the negotiation, the SA is fully filled out.

[0024] It is important to note that at the time the SA state is created in response to a negotiation request, the IKE does not know whether the request is really from the peer computer identified in the request. After creating the state for the new negotiation request, the IKE server processes the request to see whether it is valid, and then responds if the request is valid. The rest of negotiation then follows, with four more round trip packet exchanges. Part of this exchange authenticates the peer, and at that time the IKE can determine if it should allow access to that peer. This peer authentication does not occur until the third round trip, however. In short, the IKE server has to create a state immediately in response to a request from any unknown source, and the peer authentication takes place later. Thus, a malicious user of the peer computer can send in a large number of requests to force the IKE sever to create a large number of states. This consumes system resources, making the system to run slower. As the system runs slower, it cannot reclaim resources quickly, causing the system to run even slower and finally coming to a grinding halt.

[0025] In accordance with the invention, the resilience of the IKE server to the denial-of-service attack is significantly improved by dynamically detecting when the performance of the system begins to degrade, and adjusting performance variables to actively remove unneeded states and reduce the workload of the IKE server. The effectiveness of the protection against denial-of-service attacks is further enhanced and the ability of the server to serve legitimate requests is improved by limiting the number of states for pending negotiations for each client and the total number of such states. These protective measures are described in greater detail in the following description.

[0026] In accordance with a feature of the embodiment, the activation periodicity of a reaper component 122 of the system is used as a primary barometer for the system health. The function of the reaper 122 is to remove unwanted states from the state table 120 of the IKE server 76. The reaper is scheduled to run at fixed intervals, although the actual intervals between consecutive runs of the reaper would depend on the system workload. For instance, in a multi-threading system, the reaper thread may be scheduled to run every 45 seconds. On a lightly loaded system, the reaper thread will be activated at or close to the scheduled time. When the system is under a heavy workload, however, the operating system may fall behind its schedule and activate the reaper later than the scheduled time. As a result, the interval between two consecutive runs of the reaper becomes longer than 45 seconds. As the workload of the system becomes heavier, the intervals between consecutive runs of the reaper are likely to increase. The delay in the activation of the reaper thus serves as a reliable indicator of whether the system is being overloaded.

[0027] When the reaper activation intervals become longer than the scheduled interval, dynamic adjustments of relevant performance variables are made to reduce the load on the system. In a preferred embodiment, the performance variables that affect the operation of the IKE server include negotiation delay, retransmission delay, and packet drop percentage. The negotiation delay controls how long a negotiation process is allowed to last. By reducing the negotiation delay, pending negotiations are timed out sooner, and their associated SA s are removed from the state table 120. The retransmission delay is the time the IKE server 76 will wait for a response from the requesting computer before retransmitting a packet in the negotiation process. Increasing the retransmission delay makes retransmissions further apart. As a result, the system makes fewer retransmissions per unit time, thereby reducing the workload on the system. The packet drop percentage is the percentage at which the incoming negotiation packets are randomly dropped. Increasing the packet drop percentage means that the system handles fewer requests and therefore does less work. Adjusting these performance variables to reduce the system workload allows the system to heal itself from a burst of attack and to withstand short loads that far exceed its normal capacity. It will be appreciated that other performance variables that have direct or indirect impacts on the workload of the system may also be adjusted to lessen the overloading of the system.

[0028] The adjustments of the performance variables are preferably made in a progressive manner such that they are gradual at first and become more drastic as the system overloading becomes more severe. By way of example, referring to FIG. 3, each time the reaper thread is activated (step 140), the interval between the present run and the previous run is determined (step 142). In this example, the reaper 122 is scheduled to run at an interval of 45 seconds. If the reaper actually runs at an interval equal to or greater than 60 seconds (step 144), the performance variables are modified to reduce the workload of the system (146). After the reaper run interval returns to the range between 60 seconds and 45 seconds, the performance variables may be modified in the opposite direction to allow them to move back toward their initial values (step 148). For instance, in one implementation, the negotiation delay has a minimum value of 30 seconds and a maximum value of 60 seconds. Each time the reaper run interval exceeds 60 seconds, the negotiation delay is decreased by 5 seconds. After the reaper run interval returns to below 60 seconds, the negotiation delay is increased by one second for each reaper run. Similarly, the transmission delay may have a minimum of 0 second and a maximum of 15 seconds, with an increment step of 3 seconds and a decrement step of 1 second. The drop packet percentage has a minimum of 0 and a maximum of 100, with increment and decrement steps of 3 and 5, respectively. Also, the packet drop percentage starts to be incremented only if the negotiation delay is already at its minimum. In this way, the IKE server avoids dropping packets until it becomes necessary to do so.

[0029] Adjusting performance variables to reduce the system workload as described above is effective in preventing the IKE server from being paralyzed by a flooding attack. Nevertheless, since those performance variables are global, the workload reduction does not discern valid clients from potentially malicious attackers. As a result, the server will equally deny service to valid users and malicious attackers. In this regard, the protection against denial-of-service attacks in a preferred embodiment is made more focused on potential attackers by limiting the number of states pending negotiation for each client. By way of example, the number of outstanding requests for each client may be limited to a selected number, such as 6. When it is detected that a client already has 6 or more states pending negotiation (step 150), the thread that processes negotiation requests simply drops any subsequent new request from the same client as identified by the IP address of the request (step 152). This eliminates the possibility of being flooded by false requests generated by an attacker on a single machine.

[0030] It is, however, possible for an attacker to put fake source IP addresses in the false requests (which is commonly called “spoofing”). To deal with that possible scenario, in a preferred embodiment a configurable threshold (e.g., 1000) is also set for the total number of states pending negotiation. Specifically, when the IKE server 76 sends a response to a new request for negotiation, the state it allocates is marked to indicate that the negotiation for this state is not completed, such as by setting a flag named “OAK_MM_SETUP”. This setting is not changed unless the IKE server receives a valid response from the client that sent the request. In the case of spoofing, since source IP addresses of the false requests are fake, it is unlikely that the attacker would receive the negotiation packets from the IKE server and respond accordingly. Thus, the total number of states with pending negotiations is an indicator of the possibility that the IKE server is under a flooding attack.

[0031] When the reaper runs, it checks the total number of states with the negotiation-pending flag set. If the total number exceeds the pre-configured threshold (step 154), the IKE server is told not to accept any new connection request (step 156) and to more aggressively time out those negotiation-pending states. In this way, valid connections are given the opportunity to progress (i.e., to complete the negotiations), and once the negotiations are successfully completed, they are not affected by the attack. In a preferred embodiment, when it is determined that the total number of states pending negotiation has been reduced to below the threshold (step 154), the IKE server does not immediately accept all new requests but rather gradually allows new requests in. In other words, a portion of the new requests may still be dropped. For example, the new request drop rate may be set to 100%, 25%, or 0%. If the total number of negotiation-pending states is below the threshold but the new request drop rate is not 0 (step 158), the IKE server may reduce the drop rate to the next lower level, such as from 100% to 25 or from 25% to 0 (step 160).

[0032] In view of the many possible embodiments to which the principles of this invention may be applied, it should be recognized that the embodiment described herein with respect to the drawing figures is meant to be illustrative only and should not be taken as limiting the scope of invention. For example, those of skill in the art will recognize that the elements of the illustrated embodiment shown in software may be implemented in hardware and vice versa or that the illustrated embodiment can be modified in arrangement and detail without departing from the spirit of the invention. Therefore, the invention as described herein contemplates all such embodiments as may come within the scope of the following claims and equivalents thereof.