Systems and methods for network performance measurement using packet signature collection

[0001] The instant application claims priority from provisional application No. 60/317,662, filed Sep. 6, 2001, and provisional application No. 60/337,749, filed Nov. 8, 2001, the disclosures of which is incorporated by reference herein in their entirety.

[0002] The present application relates to co-pending application Ser. No. 10/044,073, entitled “Systems and Methods for Point of Ingress Traceback of a Network Attack,” filed on Jan. 11, 2002; co-pending application Ser. No. 09/881,145, entitled “Method and Apparatus for Identifying a Packet,” filed on Jun. 14, 2001; and co-pending application Ser. No. 09/881,074, entitled “Method and Apparatus for Tracing Packets,” filed on Jun. 14, 2001, each of which is hereby incorporated by reference in its entirety.

[0003] The present invention relates generally to communications networks and, more particularly, to systems and methods for passively measuring performance characteristics of communications networks.

[0004] Conventional network performance monitoring techniques fall into two categories: passive and active. Passive techniques involve examination of existing network traffic and include packet and byte counts for packets matching various criteria, and packet length histograms and stateful measurements such as average packet and data rate measurements. Active techniques involve modifying existing packet traffic, or injecting test traffic from one measurement device addressed to another measurement device, and including enough information in the test packets to extract useful measurements. A common example of an active technique is the “ping” program, which sends an ICMP Echo Request to a remote system, and processes the reply. Such packets typically include the sender's time of transmission, and allow the measurement of loss rates, average end-to-end network transit delay, and delay variance (“jitter”).

[0005] A major drawback to active measurement techniques is that such techniques consume network bandwidth—often considerable fractions of the link bandwidth, if highly-accurate results are desired over a relatively short timescale. The bandwidth consumed is, thus, not available to actual user traffic—i.e., it is often seen as wasted, except when actual network performance problems are present. Also, congestion is a common source of network performance problems (too much traffic at one or more points in the network), and sending active measurement packets at such times only worsens the congestion. In addition, there is no guarantee that test traffic will be treated by the network like normal user traffic. This might be due to either normal network packet-classification behavior, or deliberate attempts by the network operator to bias performance tests in its favor.

[0006] Passive network measurements techniques, thus, may be more desirable since they do not impact the network as a whole and do not consume otherwise usable bandwidth. Passive network measurement techniques also operate on actual user traffic and, thus, give a more accurate picture of the user's experience on the network. Passive techniques, however, are usually limited to measurements taken at a single point, since there has been no easy way to correlate appearances of the same packet at different places in the network. A conventional technique of this sort involves the collection of “packet traces” at multiple points in the network. These “packet traces” include logs of every packet header seen at that point, with an associated time stamp taken from a global clock source (e.g., a GPS receiver). Packet traces require storing about 100–200 bits from every packet, depending on the intended use. On a high bandwidth link, packet traces typically require huge amounts of storage (e.g., gigabyte disk drives), often with high bandwidth interfaces. For example, a 1 Gb/s interface will typically require a 500 Mb/s trace-collection storage device that, in turn, usually requires a special, high performance disk. Also, trace collection storage usually fills up rapidly, and takes a long time to transfer to a central repository (usually over the network itself, thus using a lot of network bandwidth). Therefore, packet traces generally cover only a few seconds to a few minutes of time, and are rarely taken more than a few times per day.

[0007] Therefore, there exists a need for systems and methods that can passively monitor network performance characteristics without requiring large amounts of storage and without using excessive amounts of network bandwidth.

[0008] Systems and methods consistent with the present invention address this and other needs by providing mechanisms for calculating signatures of packets at selected nodes of a network. The calculated signatures may include substantially fewer bits than the packet or packet header and, thus, require relatively small amounts of storage capacity. The substantially fewer bits of the calculated signatures further require relatively small amounts of network bandwidth when sent across the network to one or more collection agents. By correlating the packet signatures collected from the nodes of the network, the collection agent(s) can determine network performance parameters, such as, for example, end-to-end delay, delay variance, or packet loss rates.

[0009] In accordance with the purpose of the invention as embodied and broadly described herein, a method of measuring network performance parameters includes calculating signature values for packets received at one or more nodes in the network, each of the signature values comprising an identifier for a corresponding packet. The method further includes collecting the signature values from the one or more nodes to obtain collected signature values and determining one or more network performance parameters based on the collected signature values and network topology information.

[0010] In a further implementation consistent with the present invention, a method of measuring a packet loss rate across a network includes determining first packet signatures of packets entering the network and determining second packet signatures of packets leaving the network. The method further includes comparing the determined first and second packet signatures to identify packets entering the network that do not leave the network and determining the packet loss rate based on the comparison.

[0011] In another implementation consistent with the present invention, a method of measuring network performance parameters includes receiving packet signatures of packets received at a group of nodes in a network, where the packet signatures are calculated at the group of nodes. The method further includes correlating appearances of identical ones of the packets among the group of nodes using the calculated packet signatures to obtain correlated appearances and determining temporal behavior of packet traffic between the group of nodes based on the correlated appearances to obtain determined temporal behavior. The method additionally includes determining network performance parameters based on the determined temporal behavior.

[0012] In a further implementation consistent with the present invention, a method of logging packet signatures at a node in network includes calculating signature values for packets received at the node, each of the signature values comprising an identifier for a corresponding packet. The method further includes timestamping each of the received packets to produce a timestamp value associated with each packet and logging the calculated signature values and the timestamp values. The method additionally includes sending a group of the calculated signature values and associated time stamps to a collection agent across the network.

[0024] The following detailed description of the invention refers to the accompanying drawings. The same reference numbers in different drawings identify the same or similar elements. Also, the following detailed description does not limit the invention. Instead, the scope of the invention is defined by the appended claims.

[0025] Systems and methods consistent with the present invention provide mechanisms for determining network performance parameters, such as, for example, end-to-end delay, delay variance, or packet loss rates. Multiple nodes of a network can be equipped with packet signature recorders that calculate and store packet signatures for the packets received at each node. The algorithms used to produce the packet signatures may include hashing algorithms (e.g., MD5 message digest algorithm, secure hash algorithm (SHS), RIPEMD-160), message authentication codes (MACs), and Cyclical Redundancy Checking (CRC) algorithms, such as CRC-32. The signature values of each received packet may be stored in memory and forwarded to a collection agent for network performance analysis. The collection agent may correlate appearances of packets at the nodes of the network, using the collected packet signatures, to determine the network performance parameters.

[0026] FIG. 1 illustrates an exemplary network 100 in which systems and methods, consistent with the present invention, may operate to passively measure network performance parameters using packet signatures calculated at specific network nodes. Network 100 may include a sub-network 105 interconnected with other sub-networks 110 and 115 via respective gateways 120. Network 100 may further include one or more collection agents 130 that interconnect with sub-network 105 and monitor traffic between network nodes 125-1 through 125-n and gateways 120-1 through 120-n. Collection agents 130 may connect with sub-network 105 via wired, wireless or optical connection links (not shown). In some exemplary embodiments, the functions associated with collection agents 130 may be distributed among nodes 125. Sub-networks 105, 110 and 115 can include one or more networks of any type, including a local area network (LAN), metropolitan area network (MAN), wide area network (WAN), Internet, or Intranet. Network nodes 125-1 through 125-n may include routers, bridges or the like, for routing data through sub-network 105. Gateways 120 route data from sub-network 105 to sub-networks 110 and 115.

[0027] FIG. 2 illustrates exemplary components of a network node 125 consistent with the present invention. Gateways 120 may be similarly configured. In general, network node 125 receives incoming packets, determines the next destination (the next “hop” in sub-network 105) for the packets, and outputs the packets as outbound packets on links that lead to the next destination. In this manner, packets “hop” from node to node in sub-network 105 until reaching their final destination.

[0028] As illustrated, node 125 may include multiple input interfaces 205-1 through 205-R, a switch fabric 210, multiple output interfaces 215-1–215-S, and a packet signature recorder 220 (where R may not be equal to S). Each input interface 205 of node 125 may further include routing tables and forwarding tables (not shown). Through the routing tables, each input interface 205 may consolidate routing information learned from the routing protocols of the network. From this routing information, the routing protocol process may determine the active route to network destinations, and install these routes in the forwarding tables. Each input interface may consult a respective forwarding table when determining a next destination for incoming packets.

[0029] In response to consulting a respective forwarding table, each input interface 205 may either set up switch fabric 210 to deliver a packet to its appropriate output interface 215, or attach information to the packet (e.g., output interface number) to allow switch fabric 210 to deliver the packet to the appropriate output interface 215. Each output interface 215 may queue packets received from switch fabric 210 and transmit the packets on to a “next hop.”

[0030] Packet signature recorder 220 may include mechanisms for computing a signature of each packet received at an input interface 205, or output interface 215, and storing each computed signature in a memory (not shown). Packet signature recorder 220 may use any technique for computing the signature of each incoming packet that produces a value that is likely to be unique across a very large number of packets. Such techniques may include hashing algorithms (e.g., MD5 message digest algorithm, secure hash algorithm (SHS), RIPEMD-160), message authentication codes (MACs), or Cyclical Redundancy Checking (CRC) algorithms, such as CRC-32. The computed signatures for each incoming packet, thus, represent “hashes” that should be fairly pseudo-random relative to the incoming packet.

[0031] Packet signature recorder 220 may be internal or external to node 125. The internal packet signature recorder 220 may be implemented as an interface card plug-in to a conventional switching background bus (not shown). The external packet signature recorder 220 may be implemented as a separate auxiliary device connected to the router through an auxiliary interface. The external packet signature recorder 220 may, thus, act as a passive tap on the node's input or output links.

[0032] FIG. 3 illustrates exemplary components of packet signature recorder 220 consistent with the present invention. Packet signature recorder 220 may include signature taps 310a-310n, first-in-first-out (FIFO) queues 305a-305n, a multiplexer (MUX) 315, and a memory 320.

[0033] Each signature tap 310a–310n may produce a signature of each packet received by a respective input interface 205-1–205-R (or, alternatively, a respective output interface 215-1–215-S). The signature typically comprises k bits, where each packet may include a number of p bits and k<p. FIFO queues 305a–605n may store packet signatures received from signature taps 310a–310n. MUX 315 may selectively retrieve packet signatures from FIFO queues 305a–305n and store the packet signatures, along with a time stamp, in memory 320. Memory 320 may include, for example, a small high-speed random access device, such as an SRAM.

[0034] The number of packet signature bits k needed for each packet may be dictated by the requirement that the packet signature value be highly likely to be unique among all packet signatures recorded in the network, over the average lifetime of a packet in the network (e.g., approximately on the order of 100 ms). While the probability of two packet signatures of length k bits colliding is 1 in 2^k, each packet signature may be compared to N other packet signatures, where N may be assumed approximately equal to 2^k. With N independent trials, each with a collision probability of 1/(2^k), the number of packet signature bits k should be set approximately equal to 2*log 2(N) to avoid significant packet signature collision (where “*” is a multiplication symbol). For a network such as the Internet, the number of packets N in the network at any instant can be estimated to be on the order of 10 Gb/s (e.g., approximate cross-section of the network's total bandwidth) divided by 1000 bits (e.g., the average size of an IP packet) multiplied by 100 ms (e.g., average lifetime of a packet in the network). This estimation equals 1 million. The number of packet signature bits per packet (k) must, therefore, be at least 2*log 2(10⁶) (i.e., about 40 bits). This number of bits is substantially less than the number of bits in a typical packet header trace record.

[0035] FIG. 4 illustrates an exemplary packet signature database 400 that may be stored in memory 320 of packet signature recorder 220 consistent with the invention. Database 400 may include packet signature records 405 that further include calculated signature values for packets received at a corresponding gateway 120 or network node 125.

[0036] FIG. 5 illustrates exemplary signature record entries 500 of packet signature records 405. Each signature record entry 500 may include a packet signature value 505, a time stamp 510 and an optional destination Internet Protocol (IP) prefix 515. Packet signature value 505 may include a signature of a corresponding packet computed by packet signature recorder 220. Time stamp 510 may include a global time stamp that corresponds to the arrival of the corresponding packet at the packet signature recorder 220. Time stamp 510 may, for example, be derived from a Global Positioning System (GPS) signal or from the Network Time Protocol (NTP). Destination IP prefix 515 may indicate the IP prefix of the destination node of the corresponding packet. The value recorded in destination IP prefix 515 may include a routing entry index for the IP prefix or a signature of the IP prefix and its mask value. Each signature record entry 500 may further be time-indexed, with each computed packet signature being placed in the entry 500 corresponding to a current time.

[0037] FIG. 6 illustrates exemplary components of a collection agent 130 (e.g., 130-1–130-N of FIG. 1) consistent with the present invention. Collection agent 130 may include a processing unit 605, a memory 610, an input device 615, an output device 620, network interface(s) 625 and a bus 630.

[0038] Processing unit 605 may perform all data processing functions for inputting, outputting, and processing of data. Memory 610 may include Random Access Memory (RAM) that provides temporary working storage of data and instructions for use by processing unit 605 in performing processing functions. Memory 610 may additionally include Read Only Memory (ROM) that provides permanent or semi-permanent storage of data and instructions for use by processing unit 605. Memory 610 can also include large-capacity storage devices, such as a magnetic and/or optical recording medium and its corresponding drive.

[0039] Input device 615 permits entry of user data into collection agent 130 and may include a user interface (not shown). Output device 620 permits the output of data in video, audio, or hard copy format. Network interface(s) 625 interconnect collection agent 130 with sub-network 105. Packets received from nodes 125 may be received via network interface(s) 625. Bus 630 interconnects the various components of collection agent 130 to permit the components to communicate with one another.

[0040] FIG. 7 illustrates an exemplary collected packet signature database 700 that may be stored in memory 610 of collection agent 130 consistent with the invention. Database 700 may include collected packet signature records 705 that further include calculated signature values for packets received at a corresponding gateway 120 or network node 125.

[0041] FIG. 8 illustrates exemplary signature record entries 800 of collected packet signature records 705 consistent with the invention. Each signature record entry 800 may include a network node identifier 805, a packet signature value 505, a time stamp 510, and an optional destination Internet Protocol (IP) prefix 515. Network node identifier 805 may include, for example, a network address of the node 125 from which the packet signature value stored in packet signature value 505 was received.

[0042] FIG. 9 is a flowchart that illustrates an exemplary process, consistent with the present invention, for computing and logging packet signature values at network nodes 125 (or gateways 120) of sub-network 105 that are equipped with packet signature recorders 220. The process may begin with the reception of a packet at a packet signature recorder 220 [act 905]. A signature algorithm may then be applied to a copy of the received packet to produce a packet signature value [act 910]. For example, a hashing algorithm (e.g., MD5 message digest algorithm, secure hash algorithm (SHS), RIPEMD-160), a message authentication code (MACs), or a Cyclical Redundancy Checking (CRC) algorithm, such as CRC-32, may be applied to a copy of the received packet. Optionally, an identifier indicating the packet's destination IP prefix may be generated and appended to the packet signature value [act 915]. As another optional alternative, m additional signature bits may be generated for the packet [act 920]. To generate the m additional signature bits for each packet, an identical signature algorithm may be applied at all packet signature recorders 220 in sub-network 105. A determination may then be made whether the additional signature bits meet a specified criteria [act 925]. For example, a determination may be made whether the additional signature bits fall within a specified range. As another example, a determination may be made whether the additional signature bits equal a particular value (e.g., all zeros or all ones). If the additional signature bits do not meet the specified criteria, then the packet signature value may not be time stamped or logged, and the process may continue at act 937 below. If, however, the additional signature bits do meet the specified criteria, then the process may continue at act 930 below.

[0043] The packet signature value (along with optional destination IP prefix identifier) may be time stamped [act 930] and the packet signature value, timestamp, and optional destination IP prefix identifier may be logged as a signature record entry 500 in database 400 [act 935]. A determination may then be made whether it is time to send the data to a collection agent [act 937]. If it is not time to send data to the collection agent, the process returns to act 905 to receive another packet. In contrast, if it is determined that it is time to send data to a collection agent, then the process moves to act 940. A block that may include multiple signature record entries 500 may then be sent to one or more collection agents 130 [act 940]. The block of signature record entries 500 may be, for example, sent periodically to the collection agent(s) 130. The block of signature record entries 800 may include a time stamp for the whole block, with time stamp 510 of each record entry 800 including low order bits derived from a global time stamp (as described above) and high order bits derived from the time stamp for the whole block. “Old” signature record entries, as indicated by each timestamp value 510, may be cleaned from packet signature records 405 [act 945]. The “old” signature record entries may be, for example, cleaned from packet signature records 405 on a periodic basis.

[0044] Though, in some exemplary embodiments, packet signatures may be computed for every packet that arrives at gateways 120 or nodes 125, in other exemplary embodiments, only some fraction of arriving packets may have packet signatures computed. In such exemplary embodiments, the packet signatures may be computed uniformly across the gateways 120 and/or nodes 125 such that each gateway or node computes packet signatures for the same packets, or according to a same criteria. The exemplary process of FIG. 9, described above, may typically apply to unicast packets.

[0045] FIGS. 10–12 are flowcharts that illustrate an exemplary process, consistent with the present invention, for determining network performance parameters using packet signatures and associated time stamps collected from multiple nodes in the network. The process may begin with the receipt, at a collection agent 130, of a block of logged packet signature values/time stamps from a network node 125 (or gateway 120) of sub-network 105 [act 1005]. Each of the received packet signature values and associated time stamps may be stored as signature record entries 800 in collected packet signature records 705 [act 1010]. The ingress and egress links of sub-network 105 may then be identified [act 1015] using network topology information. For example, collection agent 130 may identify the link connecting gateway 120-1 to sub-network 110 as an ingress link, and the link connecting gateway 120-n to sub-network 115 as an egress link. A determination may then be made whether all network ingress and egress links are monitored by packet signature recorders 220 [act 1020]. If all network ingress and egress links are monitored by packet signature recorders 220, then the process may continue at act 1025 below.

[0046] If all network ingress and egress links are not monitored by packet signature recorders 220, then a link may be predicted on which any given packet should exit the network using the packet's IP destination prefix, network topology information, and routing information that was current when the packet transited the network [act 1105]. A determination may then be made whether the predicted link is monitored by a packet signature recorder 220 [act 1110]. If not, the packet may be excluded from traffic temporal behavior analysis and packet loss rate calculations [act 1130]. If the predicted link is monitored by a packet signature recorder 220, a determination may be made whether the packet appeared on the predicted egress link [act 1115]. If not, the packet may be assumed to be lost, and can be excluded from the traffic temporal behavior analysis, but included in packet loss rate calculations [act 1120]. If the packet does appear on the predicted egress link, the packet may be included in traffic temporal behavior analysis and packet loss calculations and the process may continue at act 1225 below.

[0047] Returning to FIG. 10, if all network ingress/egress links are monitored, a determination may be made whether two or more packets with the same packet signature values have been recorded both entering and leaving the network [act 1025]. If so, the two or more packets may be excluded from traffic temporal behavior analysis (see acts 1225–1235; FIG. 12), but included in packet loss rate calculations (see act 1235) [act 1210]. If not, a determination may be made whether any packet signature value has one ingress record and two or more egress records [act 1030]. If so, the packet corresponding to the packet signature value may be excluded from traffic temporal behavior analysis, but included in packet loss rate calculations. If no packet signature values have one ingress record and two or more egress records, a determination may further be made whether any packet signature value has two or more ingress records and one egress record [act 1205]. If so, the packet corresponding to the packet signature value may be excluded from traffic temporal behavior analysis, but included in packet loss rate calculations [act 1210].

[0048] If no packet signature values have two or more ingress records and one egress record, an additional determination may be made whether there is one ingress and one egress record with identical packet signature values, but the packet signature values correspond to different packets [act 1215]. This determination cannot be made from the packet signature values alone, but can be determined by temporal statistics derived from time stamp data. For example, the time stamp data may indicate that a given packet arrived at an egress link with a shorter delay than is possible on the shortest path between the ingress and egress links. The collection agent 130 may, thus, conclude that the packet signature values correspond to different packets. If this is the case, both packets may be excluded from traffic temporal behavior analysis and packet loss rate calculations [act 1220]. If not, the appearances of identical packet signature values may be correlated among all network nodes [act 1225]. The temporal behavior of traffic between signature recording network nodes may then be determined using stored knowledge of the network topology [act 1230]. For example, using packet signature matches, associated time stamp values, and network topology information, collection agent 130 may determine quality of service (QOS) parameters related to sub-network 105, such as, for example, end-to-end delay and delay variance (“jitter”). Packets arriving on an ingress link that have no corresponding appearance on any egress link may be identified to further determine a packet loss rate for sub-network 105 [act 1235].

[0049] Systems and methods consistent with the present invention, therefore, provide mechanisms that permit the determination of network performance parameters, such as, for example, end-to-end delay, delay variance, or packet loss rates. By equipping multiple nodes of a network with packet signature recorders, signatures of packets received at each node can be calculated and stored for forwarding to a collection agent for analysis. The collection agent may correlate appearances of packets at the nodes of the network, using the collected packet signatures, to determine the network performance parameters.

[0050] The foregoing description of exemplary embodiments of the present invention provides illustration and description, but is not intended to be exhaustive or to limit the invention to the precise form disclosed. Modifications and variations are possible in light of the above teachings or may be acquired from practice of the invention. For example, while certain components of the invention have been described as implemented in hardware and others in software, other configurations may be possible. Also, while series of acts have been described with regard to FIGS. 9–12, the order of the acts may vary in other implementations consistent with the present invention. Also, non-dependent acts may be performed in parallel. The scope of the invention is defined by the following claims and their equivalents.