Using ternary and binary content addressable memory stages to classify information such as packets

[0001] This is a continuation of nonprovisional application Ser. No. 09/862,018, filed May 21, 2001, now U.S. Pat. No. 7,002,965 which is hereby incorporated by reference in its entirety.

[0002] This invention relates to communications and computer systems, and in particular routers and packet switching systems; and more particularly, the invention relates to using ternary and binary content-addressable memory stages to classify information such as packets.

[0003] The communications industry is rapidly changing to adjust to emerging technologies and ever increasing customer demand. This customer demand for new applications and increased performance of existing applications is driving communications network and system providers to employ networks and systems having greater speed and capacity (e.g., greater bandwidth). In trying to achieve these goals, a common approach taken by many communications providers is to use packet switching technology. Increasingly, public and private communications networks are being built and expanded using various packet technologies, such as Internet Protocol (IP).

[0004] A network device, such as a switch or router, typically receives, processes, and forwards or discards a packet based on one or more criteria, including the type of protocol used by the packet, addresses of the packet (e.g., source, destination, group), and type or quality of service requested. Additionally, one or more security operations are typically performed on each packet. But before these operations can be performed, a packet classification operation must typically be performed on the packet.

[0005] Packet classification as required for access control lists (ACLs) and forwarding decisions is a demanding part of switch and router design. This packet classification of a received packet is increasingly becoming more difficult due to ever increasing packet rates and number of packet classifications. For example, ACLs require matching packets on a subset of fields of the packet flow label, with the semantics of a sequential search through the ACL rules. IP forwarding requires a longest prefix match.

[0006] One known approach uses a bank of ternary content-addressable memories (TCAMs) to perform packet classification. TCAMs allow the use of wildcards in performing their matching, and thus are more flexible than binary content-addressable memories (CAMs). When this bank of TCAMs is properly programmed, the TCAMs are able to perform such a match in a single lookup. However, TCAMs consume a lot of power, are of limited size and cost more than conventional memory. Needed are new approaches and systems for providing packet classification.

[0007] Systems and methods are disclosed for using ternary and binary content-addressable memory stages to classify information such as packets. One embodiment includes a ternary content-addressable memory to receive the set of information and to produce a first result. A binary content-addressable memory receives the first result or a second result derived from the first result, and generates one or more classification indications.

[0014] Methods and apparatus are disclosed using ternary and binary content-addressable memory stages to classify information such as packets. Embodiments described herein include various elements and limitations, with no one element or limitation contemplated as being a critical element or limitation. Each of the claims individually recite an aspect of the invention in its entirety. Moreover, some embodiments described may include, but are not limited to, inter alia, systems, networks, integrated circuit chips, embedded processors, ASICs, methods, and computer-readable medium containing instructions. The embodiments described hereinafter embody various aspects and configurations within the scope and spirit of the invention.

[0015] As used herein, the term “packet” refers to packets of all types, including, but not limited to, fixed length cells and variable length packets, each of which may or may not be divisible into smaller packets or cells. Moreover, these packets may contain one or more types of information, including, but not limited to, voice, data, video, and audio information. Furthermore, the term “system” is used generically herein to describe any number of components, elements, sub-systems, devices, packet switch elements, packet switches, routers, networks, computer and/or communication devices or mechanisms, or combinations of components thereof. The term “computer” is used generically herein to describe any number of computers, including, but not limited to personal computers, embedded processors and systems, control logic, ASICs, chips, workstations, mainframes, etc. The term “device” is used generically herein to describe any type of mechanism, including a computer or system or component thereof. The terms “task” and “process” are used generically herein to describe any type of running program, including, but not limited to a computer process, task, thread, executing application, operating system, user process, device driver, native code, machine or other language, etc., and can be interactive and/or non-interactive, executing locally and/or remotely, executing in foreground and/or background, executing in the user and/or operating system address spaces, a routine of a library and/or standalone application, and is not limited to any particular memory partitioning technique. The terms “network” and “communications mechanism” are used generically herein to describe one or more networks, communications mediums or communications systems, including, but not limited to the Internet, private or public telephone, cellular, wireless, satellite, cable, local area, metropolitan area and/or wide area networks, a cable, electrical connection, bus, etc., and internal communications mechanisms such as message passing, interprocess communications, shared memory, etc. The term “one embodiment” is used herein to reference a particular embodiment, wherein each reference to “one embodiment” may refer to a different embodiment, and the use of the term repeatedly herein in describing associated features, elements and/or limitations does not establish a cumulative set of associated features, elements and/or limitations that each and every embodiment must include, although an embodiment typically may include all these features, elements and/or limitations. In addition, the phrase “means for xxx” typically includes computer-readable medium or media containing computer-executable instructions for performing xxx. In addition, the terms “first,” “second,” etc. are typically used herein to denote different units (e.g., a first element, a second element). The use of these terms herein does not necessarily connote an ordering such as one unit or event occurring or coming before another, but rather provides a mechanism to distinguish between particular units. Additionally, the use of a singular form of a noun is non-limiting, with its use typically including one or more of the particular thing rather than just one (e.g., the use of the word “memory” typically refers to one or more memories without having to specify “memory or memories,” or “one or more memories” or “at least one memory,” etc.). Moreover, the phrases “based on x” and “in response to x” are used to indicate a minimum set of items x from which something is derived or caused, wherein “x” is extensible and does not necessarily describe a complete list of items on which the operation is performed, etc. Additionally, the phrase “coupled to” is used to indicate some level of direct or indirect connection between two elements or devices, with the coupling device or devices modifying or not modifying the coupled signal or communicated information. Moreover, the term “or” is used herein to identify a selection of one or more, including all, of the conjunctive items. Additionally, the transitional term “comprising,” which is synonymous with “including,” “containing,” or “characterized by,” is inclusive or open-ended and does not exclude additional, unrecited elements or method steps.

[0016] Methods and apparatus are disclosed herein that use ternary and binary content-addressable memory stages to classify information such as packets. One embodiment uses a stage of one or more TCAMS followed by a second stage one or more CAMS (or alternatively some other binary associative memories such as hash tables or TRIEs) to classify a packet. One exemplary system includes TCAMs for handling input and output classification and a forwarding CAM to classify packets for Internet Protocol (IP) forwarding decisions on a flow label. This input and output classification may include, but is not limited to routing, access control lists (ACLs), quality of service (QoS), network address translation (NAT), encryption, etc. These IP forwarding decisions may include, but are not limited to IP source and destination addresses, protocol type, flags and layer 4 source and destination ports, a virtual local area network (VLAN) id and/or other fields. One embodiment may include a hash directory of flow labels implemented in a large static random access memory (SRAM) suitable for flow classification for netflow statistics, microflow policing and redirection. In one embodiment, upon packet reception, the packet flow label is extracted from the received packet and passed to the classification TCAMs. The index returned from the TCAMs is used to map to a mask to use for lookup in the hash directory. This mapping, for example, can also specify other attributes such as whether to monitor, police or redirect the flow. The flow label is then looked up in the hash directory using this mask. If a matching hash directory entry is found, the handling specified by this directory entry is used, overriding the relevant handling that may have been determined by earlier packet classification mechanisms.

[0017] One embodiment also provides a netflow capability to serve double duty in a system, providing both netflow monitoring capability as well as providing extra capacity to the TCAMs for packet classification. In one embodiment, upon packet reception, the packet flow label is extracted from the received packet and passed to the classification TCAMs. The index returned from the TCAMs is used to map to a mask to use for lookup in the hash directory. This mapping, for example, can also specify other attributes such as whether to monitor, police or redirect the flow. The flow label is then looked up in the hash directory using this mask. If a matching hash directory entry is found, the handling specified by this directory entry is used, overriding the relevant handling that may have been determined by earlier packet classification mechanisms.

[0018] A mechanism, such as software, typically configures the packet classification mechanism to place rules that allow exact match in the hash directory as much as possible, keeping the TCAM space for those that require the flexibility of the TCAM.

[0019] In one embodiment, the forwarding TCAMs contain an entry that matches all IP multicast-addressed packets, causing them to be forwarded to process level (i.e., off the fast path). The flow label for a packet is received by the TCAMs, which then classify the packet as belonging to a multicast group. An indication is passed to the hash directory on each received IP multicast packet to look up the packet based on IP source and destination address. For example, an update mechanism (e.g., software, firmware, specialized circuitry etc.) receives a specification of a source and multicast group pair “(S,G)” and the associated forwarding action before receiving any packets with source address and destination address as (S,G). On receipt of such information, this mechanism adds an (S,G) entry to the hash directory and maps that entry to forwarding state for this as a multicast packet, corresponding to the forwarding action it determined from the multicast routing protocol. In this manner, a system can accommodate a large number of multicast groups in the hash directory without impacting the space available in a limited TCAM used for forwarding decisions.

[0020] Additionally, in one embodiment, the update mechanism may, on discovering a hash conflict between current entries and a new (S,G) entry it needs to add, choose to add this new (S,G) entry to the forwarding TCAM. By adding this entry earlier in the TCAM to the entry matching all IP multicast, this entry takes precedent. By further arranging for this entry to map to the desired multicast forwarding behavior, the software is further able to use the TCAM as a overflow assist to the hash table, eliminating one of the key disadvantages of implementing an associative memory as a hash table as compared to a true binary CAM.

[0021] A similar approach is used in one embodiment to handle unicast addressing of locally attached hosts. For example, in a large multi-layer enterprise system, a significant demand on the size of the routing table can be the routes to individual locally attached hosts, where the router needs to map to the specific MAC address of each host. One embodiment uses a separate entry in the forwarding TCAM for each such host which can use up many TCAM entries. In a more efficient manner, one embodiment uses for each prefix representing a set of directly connected hosts, a TCAM classification to match this prefix and specify a lookup in the hash directory matching just on destination address. An update mechanism adds a hash table entry for each such host H into the hash directory with an associated forwarding action that rewrites the MAC address and forwards it out the port associated with that host.

[0022] One embodiment is configured with broad security ACLs (say as a firewall), denying, for example, all incoming multicast on a specified interface. Such a system may participate in protocols that allow restrictive opening of “holes” in the firewall for specified flows. For example, a designated multicast flow may be allowed through. In this scenario, software can add these flow-specific permitting rules to the hash directory, with the associated “deny” entries in the TCAMs specifying a lookup in the hash directory by the appropriate key and mask combination. In this approach, the input ACL TCAM is not only relieved of providing the space for these specialized rules, but can avoid the churn of having rules added and removed as flows come and go, especially because the cost of modifying the TCAM contents may be quite expensive for the update mechanism on many platforms. In one embodiment, subfields of the flow label are looked up in binary associative memories to reduce the requirements for TCAMs.

[0023] In one embodiment, further user configuration may complicate the update mechanism required in using binary and ternary CAMs. For example, a network administrator may configure a system to monitor all traffic on a particular locally attached subnet using a netflow statistics capability. In this case, the update mechanism typically would need to revise the mask used for lookups in the hash directory for packets destined to this subnet to handle the full flow label and specify the appropriate forwarding action be applied to each such flow. In one embodiment, the microflow entries are automatically created from a flow parameter table, and the flow parameter table specifies the forwarding/redirection action for each flow that it automatically creates. One embodiment also specifies how to handle a packet for which it is unable to create a new microflow entry, ensuring the packets are not dropped or misdirected just because the traffic is being monitored.

[0024] Although this invention has been described using a hash table as the binary associative memory, it could be also implemented as a conventional binary CAM. It could also be implemented as a lookup structure, such as a TRIE.

[0025] Also, this invention could also be applied to other fields of a packet besides the conventional layer 2/3/4 flow label.

[0026] Further, other means of packet classification can be used to determine the binary associative memory lookup behavior and handling besides the results of a TCAM. For example, a packet header parser may determine a packet matches some characteristic that determines the binary associative memory handling, without relying on a TCAM classification.

[0027] FIG. 1 illustrates one embodiment of a system 100 using a TCAM/CAM classifier to classify information, such as received packets. For example, system 100 may be part of a communications system, packet switching system, router, computer system, etc. As shown, system 100 comprises a processor 101, memory 102, storage devices 103, subsystem with TCAM/CAM classifier 104, and network interface with TCAM/CAM classifier 105, which are electrically coupled via one or more communications mechanisms 109 (shown as a bus for illustrative purposes). Various embodiments of system 100 may include more or less elements. For example, two components, subsystem with TCAM/CAM classifier 104 and network interface with TCAM/CAM classifier 105, use packet classification methods and apparatus disclosed herein are shown in FIG. 1 as part of system 100 to emphasize the expansive nature and numerous embodiments of the invention. For example, system 100 may part of a packet switching system or router which uses packet classification methods and apparatus as part of a network interface or line card, and/or in another component or on separate chips, cards, boards, subsystems, etc.

[0028] The operation of system 100 is typically controlled by processor 101 using memory 102 and storage devices 103 to perform one or more tasks or processes. In one embodiment, system 100 programs or loads subsystem with TCAM/CAM classifier 104 and/or network interface with TCAM/CAM classifier 105 with control data entries for performing the packet classification according to the invention. Memory 102 is one type of computer-readable medium, and typically comprises random access memory (RAM), read only memory (ROM), integrated circuits, and/or other memory components. Memory 102 typically stores computer-executable instructions to be executed by processor 101 and/or data which is manipulated by processor 101 for implementing functionality in accordance with the invention. Storage devices 103 are another type of computer-readable medium, and typically comprise disk drives, diskettes, networked services, tape drives, and other storage devices. Storage devices 103 typically store computer-executable instructions to be executed by processor 101 and/or data which is manipulated by processor 101 for implementing functionality in accordance with the invention. “Computer-readable medium” is an extensible term including any memory, storage device, and/or other storage mechanism.

[0029] FIG. 2A illustrates a block diagram of one embodiment of a packet classifier using a ternary content-addressable memory stage 203 and a binary addressable memory stage (e.g., CAM, hash table, TRIEs) 204 to classify packets. In the exemplary embodiment shown, packets or packet headers 201 and zero or more other operands 202 (e.g., QoS parameters, ACLs, routing information, etc.) are provided to one or more TCAMs 203. TCAMs 203, based on their programming, generate a result which is provided, either directly or through one or more components (not shown in FIG. 2) to one or more binary addressable memories 204, which in turn generate one or more classification indications. Optional selection logic 205, including but not limited to discrete logic, an encoder, a priority encoder, a multiplexer, or other selection mechanism or combination thereof, may be used to select between multiple classification indications generated by one or more binary addressable memories 204 to generate a packet classification signal 206.

[0030] The flow diagram shown in FIG. 2B illustrates one process for using a ternary content-addressable memory stage and a binary addressable memory stage to classify information or packets. Processing begins with process block 220, and proceeds to process block 222, wherein a TCAM receives a set of information to classify. Next, in process block 224, the TCAM generates a first result based on the received set of information. Next, in process block 226, optionally, one or more components receive the first result and generate a second result. Next, in process block 228, a CAM receives the first or second result, and, in process block 230, the CAM generates one or more classification indications. Then, optionally in process block 232, selection logic (e.g., a multiplexer, encoder, priority encoder, ASIC, discrete logic, etc.) selects a classification from the one or more classification indications. Processing is then complete as indicated by process block 234.

[0031] FIG. 3 illustrates a block diagram of one exemplary embodiment including one or more TCAMs 301 for handling input and output classification (e.g., for ACLs, QoS, NAT, encryption, etc.) for a packet or packet header stored in packet buffer 300. This illustrated embodiment also includes one or more forwarding CAMs 305 to classify packets for IP forwarding decisions.

[0032] On packet reception, the packet flow label is extracted from the received packet and passed to the classification TCAMs 301. In one embodiment, TCAMs 301 typically accept a full flow label including IP source and destination addresses, protocol type, flags and layer 4 source and destination ports, a virtual local area network (VLAN) id and/or other fields. The system includes a hash directory of flow labels implemented in a large SRAM memory 302, suitable for flow classification for netflow statistics, microflow policing and redirection. Using on-chip SRAM, this hash directory may be implemented, for example, as a 16-set associative hash table, effectively implementing a binary associative memory. In one embodiment, TCAMs 301 produce an index which is used by classification memory 302 to return an entry that provides indications on forwarding, policing and/or other packet classification indications.

[0033] In one embodiment, the full flow label of the packet is then masked off by mask generator 303. A mask is typically selected based on other classifications applied to the packet, including classification produced by TCAM 301. The masked flow label is then used as the hashing key, and a lookup operation is performed in the hash directory using this mask in hash function 304. In one embodiment, a CAM or other lookup structure (e.g., TRIE) is used in place or, or in addition to the hash table. If a matching hash directory entry is found by hash function 304, the handling specified by this directory entry is used by one or more CAMs 305 to generate one or more classification indications and overriding the relevant handling that may have been determined by earlier packet classification mechanisms. If the one or more CAMs 305 may generate multiple classification indications, selection logic 306 is used to select between the classification indications, and to generate the packet classification 307. Additionally, selection logic 306 may further receive and use the result of generated by TCAMs 301 in making its selection. For example, in one embodiment, TCAMs 301 may generate a forwarding and/or policing decision, and CAMs 305 may be used to override this generated forwarding and/or policing decision.

[0034] FIG. 4 illustrates one embodiment and its operation in which the one or more forwarding TCAMs 401 can contain an entry that matches all IP multicast-addressed packets, causing them to be forwarded to process level (i.e., off the fast path). The flow label for a packet is received by TCAMs 401 from packet buffer 400. TCAMs 401 classifies the packet as belonging to a multicast group, and a lookup is performed in flow class table 402, typically based on the source and group address of the multicast packet. In one embodiment, TCAMs 401 produce an index which is used by classification memory 402 to return an entry that provides indications on forwarding, policing and/or other packet classification indications.

[0035] In one embodiment, the result of the lookup performed in memory 402 is then masked by masked generator 403, with the resulting indication passed to the hash function 404 on each received IP multicast packet to look up the packet based on IP source and destination address. The hashed value of the flow label is then used by one or more CAMs 405 to perform the final classification stage. One or more CAMs 405 typically generate one or more selection indications, which are then selected between by selection logic 406 to generate the packet classification 407. In a system where one or more CAMs 405 generate a single packet classification (i.e., packet classification 407), selection logic 406 could be eliminated.

[0036] In view of the many possible embodiments to which the principles of our invention may be applied, it will be appreciated that the embodiments and aspects thereof described herein with respect to the drawings/figures are only illustrative and should not be taken as limiting the scope of the invention. For example and as would be apparent to one skilled in the art, many of the process block operations can be re-ordered to be performed before, after, or substantially concurrent with other operations. Also, many different forms of data structures could be used in various embodiments. The invention as described herein contemplates all such embodiments as may come within the scope of the following claims and equivalents thereof.