Quality of service control mechanism and apparatus

[0002] Not Applicable

[0003] Not Applicable

[0004] The present invention is generally related to telecommunications networks, and more particularly to implementing Quality of Service (“QoS”) processing control within such networks.

[0005] Devices such as bridges and routers are employed to move data units between end-stations in a telecommunications network. The data units may represent various types of traffic including voice, video and computer data. Because some traffic may be associated with real time communication, processing of certain data units may be particularly time critical. For example, processing delay or loss of data units associated with a real time voice or video transmission could significantly degrade the quality of such communication. However, delays in exchange of some types of computer data may not significantly affect the quality of such communication. It may also be desirable to prioritize traffic within a given type of communication.

[0006] One method for prioritizing data units in a telecommunications network is with QoS parameters. Each data unit within the telecommunications network includes a header portion and a data portion. The header portion includes information for handling the data unit such as Source Address and Destination Address information. The header may also include a QoS priority indicator. It is known to insert QoS information in the transport layer of the header. Devices which receive and retransmit data units within the telecommunications network then examine the QoS portion of the header of such data units and assign priority accordingly. However, QoS has not yet been implemented in an effective manner.

[0007] In accordance with the present invention, Quality of Service (“QoS”) variables for data units of at least one predetermined protocol Type are stored in a cache memory. For data units in a one way connection (“flow”) with identified Source Address and Destination Address, thirteen bytes selected out of the Internet Protocol (“IP”) header are employed as at least a portion of a key to perform a cache lookup to obtain at least one QoS variable. The QoS variable is employed to prioritize the data unit for processing.

[0008] Enhanced performance is provided for data units of a specified protocol type that are associated with a flow. Both routing and QoS information can be loaded into the cache. Such routing and QoS information can then be retrieved by executing a single cache lookup.

[0009] Token buckets may be employed to facilitate QoS implementation. A token bucket is associated with at least one flow. Tokens are then added to the token bucket at specified time intervals. A counter is employed to track token count, and the rate at which tokens are added indicates a rate limit for frames that can be scheduled through the token bucket. The token count indicates the number of bytes that can be sent through the bucket, thereby defining a transmission rate limit.

[0010] In order to further facilitate smooth operation the bridge/router can implement a Random Early Drop (“RED”) procedure. Logic circuits generate a “congested ports” bitmask to indicate which output ports are deemed to be close to overflowing. For each output port which is indicated to be congested, the logic circuits configure a “RED shift value.” The RED shift value is the length in bits of a bitmask that is ANDed with a random number to yield a result, N, that indicates a number of frames to be queued. The Nth frame is intentionally dropped. Hence, a relatively small number of packets are dropped such that end-stations recognize the situation and respond by transmitting at a lower rate, thereby relieving congestion and reducing the chance that output queues will overflow.

[0011] The invention will be more fully understood in view of the following Detailed Description of the Invention, in conjunction with the Drawing, of which:

[0012] FIG. 1 is a simplified block diagram of a telecommunications bridge/router;

[0013] FIG. 2 is a block diagram which illustrates the motherboard and network interface modules of FIG. 1;

[0014] FIG. 3 is a flow diagram which illustrates implementation of QoS control;

[0015] FIG. 4 illustrates memory tables employed by the bridge/router for QoS control; and

[0016] FIGS. 5-7 illustrates queue prioritization.

[0017] Referring to FIGS. 1 and 2, a bridge/router for use in a telecommunications network includes a motherboard 12 and at least one network interface module 14 which are interconnected through a backplane 16. Separate interface modules support Ethernet, Fiber Distributed Data Interface (“FDDI”) and Asynchronous Transfer Mode (“ATM”) traffic. In one embodiment each 10/100 Mb Ethernet interface module has six ports, each gigabit Ethernet interface module has one port, each FDDI interface module has six ports and each ATM interface module has two OC3 ports or one OC12 port. The ports provide connections to other devices in the network, through which data units can be received and transmitted. Incoming data units may be bridged, routed, translationally bridged and filtered by the bridge/router. Logic circuits in the motherboard and interface modules are responsible for data unit reception and transmission, parsing Data Link and Network Layer headers, looking up source and destination Media Access Control (“MAC”) and Network Layer addresses and making forwarding decisions.

[0018] The motherboard 12 includes an Address Cache ASIC (“ACA”) 26 with associated address cache memory 28, a Frame Processor (“FP”) 30 and a Master Buffer ASIC (“MBA”) 32. The Address Cache ASIC 26 is responsible for performing cache 28 lookups on Destination Addresses (“DAs”) and Source Addresses (“SAs”). The Address Cache ASIC is employed to obtain MAC addresses for bridging support and Network Layer addresses for routing support. The Master Buffer ASIC 32 is responsible for data buffer management in buffer RAM 22.

[0019] Each network interface module includes buffer RAM 22, a Transmit ASIC (“TA”) 25 and a Receive ASIC (“RA”) 27. The Transmit ASIC and Receive ASIC are specific to the type of data traffic which the network interface device is designed to support (such as Ethernet, ATM and FDDI). The Receive ASIC 27 functions to perform a preliminary analysis on incoming data units. The Transmit ASIC 25 functions to transmit data units.

[0020] The Receive ASIC 27 includes a Receive State Machine (“RXSM”) 40, a Receive Header Processor (“RHP”) 46 and a Receive Frame Processor (“RFP”) 48. The Receive State Machine is responsible for receiving data units through one or more ports from an associated communications link. After receiving a data unit, the Receive State Machine 40 generates data unit status information. The status information, which contains error information and byte and frame count data on a per port basis, is stored in registers 44. The Receive Header Processor 46 is responsible for identifying data units to be bridged or routed, determining inbound data unit encapsulation type, and performing protocol specific processing for routed data units. The Receive Header Processor also determines which VLAN, if any, each incoming frames is received on. There are different versions of Receive Header Processors 46 for different network interface types, e.g., Ethernet, FDDI and ATM. The Receive Header Processor 46 is primarily implemented in microcode. A Receive Segmentation DMA controller (“RSEG”) 50 controls storage of received data units within appropriate buffer RAM 22 locations and forwards status information to the Receive Frame Processor 48. Information in a VLAN forwarding table 49 is employed by the Receive Frame Processor 48 to verify if the data unit is allowed to be forwarded through the outbound interface. In particular, the Receive Frame Processor 48 is responsible for making forwarding decisions based on information supplied by the Receive Header Processor 46, Address Cache ASIC 26, Receive State Machine 40, the RSEG 50 and VLAN configuration information contained in configuration tables associated with the Receive Frame Processor. The Receive Frame Processor 48 also generates Transmit Vectors for data units being processed in hardware over a fast processing path and Receive Vectors for data units being passed to the Frame Processor 30 software for further processing over a slower path. The Receive Frame Processor 48 is implemented partially in microcode.

[0021] The Transmit ASIC 25 includes a Transmit State Machine (“TXSM”) 58, a plurality of Transmit State Machine FIFOs 59, and a Transmit Header Processor (“THP”) 60. A Transmit Segmentation Controller (“TSEG”) 61 serves to move data unit segments from locations within the Buffer RAM 22 into an input FIFO designated as the TSEG FIFO 63, which comprises an input FIFO to the Transmit Header Processor 60. The Transmit Header Processor 60 performs any necessary header translations and, upon completion of such translations, moves the translated header to the Transmit State Machine FIFO 59.

[0022] The Transmit Header Processor 60 also inserts VLAN tags into frames as necessary. Data units are forwarded from the Transmit State Machine FIFO 59 over the respective output port 20 of the network interface module 14. The Transmit State Machine 58 is responsible for controlling transmission of data units from the respective output port 20. Following transmission, the Transmit State Machine generates data unit transmit status information which is stored in registers 66. The status information includes error information and transmit byte and frame count information on a per port basis. Different versions of the Transmit State Machine 58 are provided for different network interface module types, e.g., Ethernet, FDDI and ATM.

[0023] FIGS. 3 and 4 illustrate data unit processing in conjunction with a network interface module configured for Ethernet type traffic. When an Ethernet frame enters the Receive Header Processor 46 as shown in step 70, the header portion of the frame is examined as shown in step 72. Given 20 a header with the format: Destination Address (“DA”)/Source Address (“SA”)/Type (e.g., IPv4, Appletalk etc.)/Internet Protocol (“IP”) header, the Organization Unique Identifier (“OUI”) is compared to the upper three bytes of the DA. If the OUI does not match the upper three bytes of the DA, then the frame is forwarded to the Frame Processor 30 for further processing by software as shown in step 74. If the OUI matches the upper three bytes of the DA, then the lower three bytes of the DA are examined to determine whether the value indicated by the lower three bytes of the DA falls within a predefined range as shown in step 76. The upper and lower bounds of the predefined range are maintained in first and second respective registers. The lower three bytes of the DA are compared with each of the first and second registers to determine whether the lower three bytes of the DA fall within the predefined range. If the lower three bytes of the DA do not fall within the range, then the frame is forwarded to the Frame Processor 30 for further processing by software as shown in step 74. If the lower three bytes of the DA fall within the predefined range, then the DA matches the address specified for the bridge/router and the location of the Type field is examined as shown in step 78. In particular, the value of the field following the SA field is compared with a predetermined value (1500) below which the field is assumed to be a Type field.

[0024] The specific protocol Type is determined as shown in step 80, where predefined values correspond to specific Types, e.g., 0800 corresponds to IPv4. The frame is then examined to determine if it is associated with a flow as shown in step 83. A flow is a one way connection of frames with positively identified Source Address and Destination Address. Step 83 may be implemented as follows:

[0025] If the frame is associated with a flow (flow=TRUE), a 13-byte IPv4 flow ID is employed an indicated in step 90. The 13-byte flow ID includes IP protocol Type, IP SA, IP DA, and UDP Source and Destination ports. The Address Cache ASIC 26 is then employed, and QoS variables are returned as shown in step 92. In particular, the Address Cache ASIC 26 returns a token bucket depth indicator, a transmit queue designator, a loss flag, an excess transmit queue designator, an excess loss flag, a designator for excess action to be taken on non- conforming frames, a retag enable indicator and a retag class designator. The excess transmit queue designator designates the transmit queue for non-conforming frames. If retag enable is set then the frame will be retagged and requeued rather than dropped in the event that bridge/router traffic prevents initial queueing and transmission. Since both routing and QoS information are stored in the address cache, frame forwarding can be commenced after a single lookup operation.

[0026] The Address Cache ASIC also returns a QoS enable bit.

[0027] When set, the QoS enable bit indicates that the QoS information in the Cache is valid. When not set the QoS enable bit indicates that the frame is not actually associated with a flow, even though the result flow=TRUE was reached in step 83. When flow=TRUE and the QoS enable bit indicates that the frame is not actually associated with a flow, the frame is processed as a non-flow frame.

[0028] If the frame is not associated with a flow (flow=FALSE), the Receive Header Processor forwards the MAC DA, IP DST or IP SRC and DST, and lookup key length indicator to the ACA. The protocol Type and Class of Service (“CoS”) are employed as indices into the QCR Selector 81 as shown in step 84 to obtain a 4-bit value. The 4-bit value is then employed as an index into the QoS Control Register (“QCR”) Table 82 as shown in step 86. The QCR Table 82 provides a priority designation in one of eight priority levels.

[0029] Each entry in the QCR Table 82 includes eight fields totalling 32-bits. 20-bits are employed to designate token bucket depth. 2-bits are employed to designate which of the four queues associated with each port to transmit on. 1-bit is employed as a loss flag. The loss flag indicates the loss bit setting of the transmit vector. 2-bits are employed to designate the excess transmit queue, which designates the transmit queue for non-conforming frames. 1-bit is employed as an excess loss flag. 2-bits are employed to designate excess action to be taken on non-conforming frames. 1-bit is employed for retag enable, and 3-bits are employed to designate retag class. If retag enable is set then the frame will be retagged and requeued rather than dropped if bridge/router traffic prevents initial queueing and transmission.

[0030] There are sixteen class-based QCRs which the Receive Frame Processor employs to assign QoS variables on a per-traffic class basis. The QCRs may be assigned as follows:

[0031] The QCR selector 81 has separate sections 85 for unicast, multicast and broadcast traffic. Within each section, the Receive Header Processor protocol number is employed as an index to obtain a word. TCP/IP is handled as a separate protocol to allow UDP and TCP frames to be given different QoS control. The 802.1p traffic class is used to select the correct nibble within the word. The QCR selector may be implemented as:

[0032] Token buckets are employed in SRAM to implement QoS parameters with traffic control algorithms. The token bucket algorithm is implemented in a packet scheduler. Each token bucket is initially empty at a time zero. Tokens are added to a token counter at specified intervals of time to indicate a rate limit for frames that can be scheduled through the token bucket. The token count reflects a credit for how many bytes can be sent through the bucket without violating a specified rate limit. The bucket has a maximum depth which reflects buffering available for the bucket and also governs the first level of traffic scheduled through the bucket. The token count never exceeds the bucket depth.

[0033] There are 128 token buckets and each token bucket is specified by a 20-bit token counter and a 12-bit token bucket rate (128*32=4 Kbits). The twelve bit token bucket depth is expressed in units of 256 bytes, so bucket depths range from 256 bytes to 212 times that or 1 Mbyte. The token bucket rate is expressed in units of 16 bytes per millisecond. Each bucket may control up to the aggregate incoming data rate on all ports associated with an interface module.

[0034] FIG. 5 illustrates queue prioritization on a per port basis. Four queues, designated 0, 1, 2 and 3 are associated with each port, and each frame is associated with at least one such queue for transmission. The Master Buffer ASIC is configured with a queue drainage schedule that governs the relative rate at which each of the four queues is drained to the Transmit Header Processor. Frames are sequentially loaded in the queues and selected for transmission based upon the illustrated prioritization ordering scheme. Frames associated with queue 0 are always given the highest priority and frames associated with queue 3 are always given lowest priority. Queue 1 is a lower priority than queue 0, and higher priority than queues 2 or 3. However, queues 1 and 2 implement cross over sharing at a predetermined depth (six frames deep is shown) into each such queue. In the illustrated example when five frames associated with queue 1 are transmitted, two frames associated with queue 2 are then transmitted, assuming such queues are loaded with frames.

[0035] Traffic associated with queue 0 is preferably of a protocol Type that will not utilize all available bandwidth and starve queues 1, 2 and 3. For example, RSVP Type traffic can be limited by other mechanisms within the bridge/router and hence may be assigned to queue one. Further, administrative overhead data transmitted in frames between devices in the network may be assigned to queue 0 in order to facilitate smooth network operation.

[0036] In the alternative embodiment illustrated in FIG. 6 equal weighting is given to each queue. As a result, each queue is presented with drainage equal opportunities. FIG. 7 illustrates a drainage schedule with relative weighting of 8-4-2-1. Multiple drainage schedules may be employed, and each port can be drained according to any of the drainage schedules.

[0037] In order to further facilitate smooth operation the bridge/router implements a Random Early Drop (“RED”) procedure. The Frame Processor 30 configures a congested ports bitmask to indicate to the Receive Frame Processor 48 which output ports are deemed to be nearly full of queued frames. The Frame Processor determines which output ports are congested based upon Master Buffer ASIC 32 statistics and Master Buffer ASIC queue lengths. For each output port which is congested, the Frame Processor configures a “RED shift value.” The RED shift value is a length, in bits, of a bitmask that is ANDed with a random number generated by the Receive Frame Processor. The resulting number, N, is the number of frames to be queued, and the Nth frame is intentionally dropped. When the Nth frame is dropped, a new random number is generated and the countdown to the next dropped frame begins. The drop rate resulting from the RED procedure is a function of the RED shift value and the generated random numbers. In particular, Drop Rate=

[0038] TotalDrops/TotalPackets=2/(N+1).

[0039] The Frame Processor dynamically adjusts the RED shift value for congested output ports based on feedback from the Master Buffer ASIC. Hence, small numbers of packets are dropped before output queues are overflowing, and a percentage of the end-stations generating the congesting traffic should respond with lower transmission rates.

[0040] Having described the preferred embodiments, other embodiments that incorporate concepts of the presently disclosed invention will become apparent to those of skill in the art. Therefore, the invention should not be viewed as limited to the disclosed embodiments but rather should be viewed as limited only by the spirit and scope of the appended claims.