Apparatus and method for just-in-time transfer of transmit commands to a network interface

[0001] The invention relates generally to networking and, more particularly, to an apparatus and method for providing transmit commands to a network interface when the number of transmit commands at the network interface falls below a threshold.

[0002] A network interface typically couples a computer system with a network, the network interface comprising on-board circuitry or an attached peripheral card. Functions performed by the network interface include receiving packets from the network and transmitting packets from the computer system out onto the network. When packets are received, the network interface may assert one or more interrupts to indicate receipt of the packets to system components—e.g., an operating system or application programs resident on the computer system—that need the packet data. Similarly, when packets are transmitted from the computer system, the network interface may assert one or more interrupts to indicate transmission of a packet or to signal that a transmit queue is empty and ready to receive additional packets for transmission. Control of the network interface is provided by a network driver resident in a memory of the computer system, and the network driver includes an interrupt service routine (ISR) for processing the interrupts generated by the network interface.

[0003] When a packet is to be transmitted, the network driver will provide the network interface with a transmit command, and the transmit command indicates to the network interface that the identified packet is ready for transmission. One type of transmit command is a “packet descriptor” (or simply a “descriptor”). A descriptor identifies a memory location of a packet, and the network driver will assign one or more packet descriptors to each packet that is to be transmitted from the computer system.

[0004] Typically, as the network driver receives packets from the computer system's operating system and assigns descriptors thereto, the network driver immediately provides the descriptors to the network interface, regardless of the state of the network interface. The descriptors are sent to the network interface over a bus (e.g., a system bus, a peripheral bus, and the like), and each group of descriptors transmitted over the bus requires a bus transaction. Immediately sending descriptors to the network interface is an inefficient use of the bus, however, as only a relatively small number of descriptors will be transmitted to the network interface during a single bus transaction. Use of these “small” bus transactions will necessitate a large number of such transactions, thereby inhibiting use of burst data transactions and preventing the bus from achieving its maximum throughput.

[0005] Immediate transfer of descriptors to the network interface also results in packet latency. Each transmission of new descriptors to the network interface may interrupt or otherwise delay operation of the network interface, as the network interface will not be able to transmit packets while descriptors are being transferred to the network interface. Multiple transfers of a small number of new descriptors to the network interface will repeatedly interrupt operation—e.g., fetching packets identified by descriptors previously transferred to the network interface and transmitting those packets to the network—of the network interface. Such delays in transmitting packets increase packet latency and can result in unutilized bandwidth.

[0006] Interrupts generated by the network interface during packet transmission to the network may also cause packet latency. One conventional interrupt is commonly referred to as the “Transmit Complete” interrupt. The Transmit Complete interrupt is asserted each time the network interface completes transmission of a packet or packets. A function of the Transmit Complete interrupt is to initiate resource reclamation, which is carried out by the network driver's ISR or similar function. During resource reclamation, processed descriptors are recovered (there is a finite number of available descriptors) for subsequent use by the network driver. Asserting the Transmit Complete interrupt at completion of each transmit operation, however, leads to the generation of a large number of interrupts for high-speed devices. These interrupts must be processed by the network driver, which increases the overhead (e.g., processing resources) associated with interrupt handling, thereby decreasing processor efficiency and increasing packet latency.

[0007] Another conventional interrupt is the “Transmit Queue Empty” interrupt. This interrupt signals to the network driver that the network interface's transmit queue is empty and may receive additional packets for transmission. However, by the time the network driver is able to respond to this interrupt (e.g., by providing additional transmit commands), the network interface has sat idle for a period of time, which is an inefficient use of resources and introduces latencies. Algorithms have been developed to predict when the transmit queue at the network interface would be “almost” out of packets, but these algorithms have proven inaccurate and unable to improve efficiency.

[0013] Embodiments of an apparatus and method for the just-in-time transfer of transmit commands to a network interface are disclosed in FIGS. 1 though 5 and the accompanying text. For clarity and ease of understanding, these embodiments are presented below in the context of a specific type of transmit command—i.e., a descriptor. It should be understood, however, that the embodiments disclosed herein are not limited to the use of descriptors and, further, that these embodiments may be practiced with any type of transmit command or other command. It should be further understood that, although the present invention is described in the context of a network interface, the present invention is generally applicable to any type of I/O controller.

[0014] Referring to FIG. 1, a system 100 includes a bus 110 having a processor 120 coupled therewith. The processor 120 may comprise any microprocessor, ASIC (application specific integrated circuit), or other suitable processing device. A read-only memory (ROM) 130, or other suitable memory, may also be coupled with the bus 110, and the ROM 130 may have a system BIOS (basis input/output system) 192 resident thereon. In addition, one or more input devices 140, as well as one or more output devices 145, may be coupled with the bus 110. Common input devices 140 include keyboards, pointing devices such as a mouse, and scanners or other data entry devices, while typical output devices 145 include video monitors, printers, and audio output devices (e.g., a sound card and/or speakers).

[0015] A main memory 150, or other suitable memory, is coupled with the bus 110, the main memory 150 comprising, for example, dynamic random access memory (DRAM). An operating system (O/S) 194 and one or more application programs 196 may be resident in the main memory 150 during operation of the system 1. One or more drivers, such as a network driver 198, may also be resident in main memory 150. In addition, one or more packets 10—at least some of the packets 10 awaiting transmission to the network 5—may be stored in the main memory 150.

[0016] The operating system 194, applications programs 196, and network driver 198 may be stored in a storage device 160, the storage device 160 comprising, for example, a hard disk drive or other suitable non-volatile memory. The storage device 160 may be coupled with the bus 110 via a Small Computer System Interface (SCSI) bus 112 (see, e.g., the SCSI-3 family of specifications). Further, the system 100 may include one or more removable memory devices. For example, a CD-ROM drive 170 may be coupled with the bus 10 via SCSI bus 112, and a floppy disk drive 175 may also be coupled with the bus 110.

[0017] The system 100 is coupled with a network 5 by a network interface 180. The network interface 180 may be coupled with any type of network 5—including the Internet, a wide area network (WAN), a metropolitan area network (MAN), or a local area network (LAN)—exhibiting any suitable network architecture. The network interface 180 may be integrated directly into the system 100 (e.g., provided by instructions and/or circuitry disposed on a motherboard). Alternatively, the network interface 180 may comprise a separately attached peripheral card, such as a network interface card (NIC). For example, the network interface 180 may comprise a PCMCIA (Personal Computer Memory Card International Association) compatible peripheral card—see, e.g., PC Card Standard, March 1997 Release—or a PCI (Peripheral Component Interconnect) compatible peripheral card—see, e.g., PCI Local Bus Specification, Revision 2.2. Further, the network interface 180 may be used with any type of media, including copper wire or other cabling, fiber optic cable, or a wireless media.

[0018] The network interface 180 provides an interface between the network 5 and the system 100. For example, the network interface 180 may receive packets 10 from main memory 150 (or other memory of system 100) and transmit the packets 10 to the network 5. Also, the network interface 180 may receive packets from the network 5 and indicate receipt—such as by asserting an interrupt—of the packet(s) to the network driver 198, as well as to the operating system 194 and applications programs 196. The network interface 180—the operation of which will be explained in detail below—may include a controller 182, a memory 184, and a transmit queue or packet FIFO (first-in-first-out) buffer 186.

[0019] The system 100 may further include a packet descriptor queue 155. The packet descriptor queue 155 may be provided in the main memory 150 or other suitable memory and may be cached in the network interface 180. When the operating system 194 provides packets 10 that are to be forwarded to the network interface 180 for subsequent transmission, the network driver 198 assigns one or more descriptors to each packet 10. The descriptors identify memory locations of the packets 10. As will be explained in greater detail below, the packet descriptor queue 155 may store descriptors (or, more generally, any suitable queue may store transmit commands) for later transfer to the network interface 180.

[0020] Illustrated in FIG. 2 is a method 200 of allocating descriptors, as may be implemented by the network driver 198. The network driver 198 maintains a sum variable indicating the total number of descriptors at the network interface 180, this sum variable maintained by the network driver 198 being referred to herein as the “apparent” descriptor total (DTOT_ND). The network driver 198 also has a pre-defined threshold, which will be referred to herein as the “first” threshold (DTHLD_ND). If the apparent descriptor total is greater than the first threshold, the network interface 180 has a sufficient number of descriptors in-process and additional descriptors should not be provided thereto. If the number of descriptors at the network interface 180, as indicated by the apparent descriptor total, is less than the first threshold, the network interface 180 can receive additional descriptors.

[0021] Referring to reference numeral 210 in FIG. 2, the network driver 198 receives one or more packets 10—or, more generally, the network driver 198 receives data—from the operating system 194 (or from another component of system 100). As denoted at 220, the network driver 198 assigns descriptors (DNEW) to the packets 10, each of the packets 10 having one or more descriptors identifying a memory location. The apparent descriptor total (DTOT_ND) is then compared to the first threshold (DTHLD_ND)—see reference numeral 230—and, if the apparent descriptor total is greater than the first threshold, the network driver 198 places the descriptors (DNEW) in the packet descriptor queue 155, as denoted at 240, for later transmission to the network interface 180. The number of descriptors at the packet descriptor queue 155 (DQ) is then incremented by the number of descriptors (DNEW) placed therein.

[0022] Referring again to reference numeral 230, if the apparent descriptor total (DTOT_ND) is less than the first threshold (DTHLD_ND), the network driver 198 provides the descriptors (DNEW) to the network interface 180 for processing (e.g., fetching of packets 10 from main memory 150 and transmission of packets 10). The apparent descriptor total (DTOT_ND) is then incremented by the number of descriptors (DNEW) transmitted thereto from the network driver 198, as shown at 255. All other packets 10 received from the operating system 194 may be processed by the network driver 198 in an identical fashion.

[0023] Shown in FIG. 3 is a method 300 of processing descriptors, as may be implemented at the network interface 180. The network interface 180 maintains a sum variable indicating the total number of descriptors at the network interface 180, and this sum variable will be referred to herein as the “actual” descriptor total (DTOT_NI). The network interface 180 also has a pre-defined threshold, which will be referred to herein as the “second” threshold (DTHLD_NI). If the actual descriptor total is less than the second threshold, the network interface 180 can receive additional descriptors. As will be explained below, the network interface 180 will indicate to the network driver 198 that it is ready to receive additional descriptors by asserting an interrupt. If the actual descriptor total is greater than the second threshold, the network interface 180 has a sufficient number of descriptors in-process and additional descriptors should not be received.

[0024] As shown at reference numeral 310 in FIG. 3, the network interface 180 may receive descriptors from the network driver 198—such as may occur when the apparent descriptor total maintained by the network driver 198 is less than the first threshold or as may occur when the actual descriptor total maintained by the network interface 180 is less than the second threshold, as will be explained below—and these descriptors may be stored in the memory 184 of network interface 180. The received descriptors (either DNEW or DQSENT) are added to the actual descriptor total (DTOT_NI), as denoted at 315.

[0025] Based upon the memory locations identified by the descriptors, the network interface 180 fetches packets 10 from the main memory 155—see reference numeral 320—and places the packets 10 in the packet FIFO 186, as shown at 330. The network interface 180 may then transmit packets 10 to the network 5, as shown at reference numeral 340. Referring to reference numeral 350, the network interface 180 may provide an indication of processed descriptors (DOLD)—i.e., those descriptors corresponding to transmitted packets—and the actual descriptor total (DTOT_NI) is decremented by the number of processed descriptors (DOLD), as denoted by 355.

[0026] The actual descriptor total (DTOT_NI) is compared to the second threshold (DTHLD_NI)—see reference numeral 360—and, if the actual descriptor total is, at any time, less than the second threshold, the controller 182 of network interface 180 will assert an interrupt—see reference numeral 370—to signal the network driver 198 that additional descriptors may be provided to the network interface 180. This interrupt, which will be referred to herein as the “Network Interface Nearly Empty” interrupt, may be asserted at the same pin or status bit as the conventional interrupts (such as, e.g., the Transmit Complete and Transmit Queue Empty interrupts). However, in contrast to the conventional interrupts, the Network Interface Nearly Empty interrupt is asserted when the number of unprocessed descriptors—i.e., those descriptors corresponding to packets awaiting transmission—at the network interface 180 falls below the second threshold. The Network Interface Nearly Empty interrupt will cause the network driver 198 to execute its ISR, as will be explained below.

[0027] The process proceeds for any remaining packets 10. If any packets 10 remain in the packet FIFO 186—see reference numeral 380—those packets are transmitted to the network 5 (see reference numeral 340) and the descriptors associated with the transmitted packets are decremented from the actual descriptor total (see reference numerals 350, 355). Likewise, if any packets 10 (for which the network interface 180 already has the descriptors) remain in main memory 150—see reference numeral 385—those packets are fetched from main memory 150 (see reference numeral 320) and placed in packet FIFO 186 (see reference numeral 330) for subsequent transmission to the network 5 (see reference numerals 340, 350, 355). If no packets 10 remain in packet FIFO 186 and, further, if no packets 10 (for which the network interface 180 already has descriptors) remain in main memory 150, the network interface 180 is idle—as shown at reference numeral 390—and will await receipt of additional descriptors.

[0028] Illustrated in FIG. 4 is a method 400 of processing the Network Interface Nearly Empty interrupt asserted by the network driver 180. As shown at reference numeral 410, the network driver 198 receives (or otherwise recognizes) the interrupt asserted by the network interface 180 (see FIG. 3, reference numeral 370). In response to receipt of this interrupt, the ISR of network driver 198 is executed—by, e.g., the processor 120—as denoted at reference numeral 420. During execution of the ISR, the network driver 198 recovers processed descriptors (DOLD) from the network interface 180, as shown at 422. Referring to reference numeral 424, the network driver 198 then provides some (DQSENT) of the descriptors queued (DQ) in packet descriptor queue 155 to the network interface 180. The number of queued descriptors provided to the network interface (DQSENT) in response to assertion of the Network Interface Nearly Empty interrupt may comprise all of the descriptors (DQ) currently stored in the packet descriptor queue 155 or, alternatively, a selected portion thereof. It should be understood, however, that there may not be any queued descriptors at packet descriptor queue 155 when the ISR is executed.

[0029] The apparent descriptor total (DTOT_ND) maintained by the network driver 198 is incremented by the number of queued descriptors sent (DQSENT) to the network interface 180 and is decremented by the number of processed descriptors (DOLD), as shown at 426. Also, referring to reference numeral 428, the total number of queued descriptors (DQ) in packet descriptor queue 155 is decremented by the number of descriptors sent (DQSENT) to the network interface 180. The method 400 of processing the Network Interface Nearly Empty interrupt asserted by the network interface 180 may be repeated upon each assertion of this interrupt.

[0030] The above-described methods 200, 300, 400 may be better understood with reference to the combined diagram of FIG. 5, which illustrates each of these methods as well as the interaction between the network driver 198 and network interface 180. It should be noted that, in FIG. 5, those actions performed by the network driver 198 are shown is solid line, whereas those actions performed by the network interface 180 are shown in dashed line. Also, the reference numerals of FIGS. 2, 3, and 4, respectively, are incorporated into FIG. 5.

[0031] Referring to FIG. 5, the network driver 198 may receive one or more packets 10—or, more generally, the network driver 198 may receive data—from the operating system 194 (or other component of system 100)—see reference numeral 210—and the network driver 198 assigns one or more descriptors (DNEW) to each of the packets 10, as shown at 220. Referring to reference numeral 230, if the apparent descriptor total (DTOT_ND) is greater than the first threshold (DTHLD_ND), the descriptors (DNEW) are queued in the packet descriptor queue 155, as illustrated by reference numerals 240, 245.

[0032] Referring again to reference numeral 230, if the apparent descriptor total (DTOT_ND) is less than the first threshold, the network driver 198 will provide the descriptors (DNEW) to the network interface 180, as shown at 250, 255. The network interface 180 subsequently receives the descriptors (DNEW)—see reference numerals 310, 315—and the network interface 180 will fetch the packets 10 identified by these descriptors (DNEW) from main memory 150, as shown at 320. Once a packet 10 is fetched from main memory 150, the packet 10 may be placed in the packet FIFO 186—see reference numeral 330—for transmission to the network 5 (see reference numerals 340, 350, 355). The transmission of packets 10 continues so long as packets remain in the packet FIFO 186 (see reference numeral 380). Further, the fetching of packets 10 from main memory 150, the storing these packets in packet FIFO 186, and the subsequent transmission of these packets to the network 5, respectively, continues so long as packets 10 (for which the network interface 180 has already received descriptors) remain in main memory 150 (see reference numeral 385).

[0033] If, at any time during the operation of the network interface 180, the actual descriptor total (DTOT_NI) falls below the second threshold (DTHLD_NI)—see reference numeral 360—the network interface 180 will assert the Network Interface Nearly Empty interrupt—see reference numeral 370—and, in response, the network driver 198 will execute its ISR (see reference numerals 410, 420). When the ISR is executed (see reference numerals 422, 424, 426, 428), the network driver 198 will provide all or a selected portion (DQSENT) of the descriptors queued (DQ) in packet descriptor queue 155, if any, to the network interface 180 (see reference numerals 424, 310). The network interface 180 processes these descriptors as set forth above (see reference numerals 315, 320, 330, 340, 350, 355).

[0034] The network driver 198 maintains a running total of the number of descriptors at the network interface 180, this running total being referred to as the apparent descriptor total (DTOT_ND), as noted above. The network interface 180 itself also maintains a running total of the number of descriptors that it has, this running total being referred to as the actual descriptor total (DTOT_NI), also as noted above. It should be noted that, although the apparent and actual descriptor totals provide an indication of the same property—i.e., the total number of descriptors at the network interface 180—these two variables may not necessarily be equal at a given instant in time. For example, when additional descriptors are provided to the network interface, the apparent descriptor total increments first (see FIG. 5, reference numerals 255, 426, 315), whereas the actual descriptor total decrements first (see FIG. 5, reference numerals 355, 426) when processing of descriptors is complete.

[0035] As set forth above, the network driver 198 has a threshold number of descriptors referred to as the first threshold, and the network interface 180 has a threshold number of descriptors referred to as the second threshold. In practice, the first and second thresholds will generally be substantially equivalent. However, it should be understood that the disclosed embodiments are not limited to use of identical thresholds by the network driver 198 and the network interface 180, respectively. It may be desirable, for example, to have different first and second thresholds to compensate for the fact that, at any instant in time, the actual and apparent descriptor totals may not be the same, as noted above.

[0036] During periods of low network traffic, wherein the system 100 is sending a limited number of packets (or no packets) out onto the network 5, the network interface 180 may never achieve a state where the actual descriptor total exceeds the second threshold and/or where the apparent descriptor total exceeds the first threshold. In such a circumstance, the network interface 180 and network driver 198 operate in a conventional manner. For example, each time a transmit function is complete, an interrupt will be asserted (see FIG. 5, reference numeral 370) because the actual descriptor total remains below the second threshold and, in this instance, the interrupt asserted would be analogous to the conventional Transmit Complete interrupt. Thus, during low traffic periods, the network driver 198 will immediately dump descriptors to the network interface 180—resulting in repeated interruptions of the network interface's operation—and, further, the network interface 180 will assert multiple interrupts, thereby increasing the overhead associated with interrupt processing. However, these characteristics (i.e., repeated interruptions of network interface 180 and the high overhead for interrupt processing) minimally impact the performance of system 100, as the system 100 has ample resources during low traffic periods.

[0037] During periods of high network traffic, however, the network interface 180 does not receive additional work—i.e., additional descriptors—unless the amount of work currently at the network interface 180 falls below a specified level, as measured by the first and second thresholds. Accordingly, operation of the network interface 180 is not interrupted every time the network driver 198 receives a packet from the operating system 194. Further, by asserting the Network Interface Nearly Empty interrupt any time the number of descriptors at the network interface 180 falls below this specified level (i.e., the second threshold), the network driver 198 is “warned” in advance that the network interface 180 is “nearly” out of descriptors and, therefore, the network driver 198 can provide additional descriptors (from the packet descriptor queue 155) to the network interface 180 before the network interface 180 runs out of descriptors. Thus, descriptors are provided to the network interface 180 on a “just-in-time” basis, thereby preventing unnecessary interruption of the network interface 180 but also preventing periods of delay while the network interface 180 remains idle awaiting receipt of additional descriptors from the network driver 198.

[0038] Sending descriptors—or, more generally, transmit commands—to the network interface 180 on a just-in-time basis provides a number of advantages. The network driver 198 does not repeatedly interrupt operation of the network interface 180—by immediately dumping descriptors to the network interface 180, irrespective of its state—and the use of multiple “small” bus transactions is minimized. Accordingly, a large number of descriptors may be sent to the network interface 180 in a single burst data transaction across the bus 110, thereby improving bus efficiency and minimizing packet latency. In addition, the number of interrupts asserted by the network interface 180 is reduced, as the Network Interface Nearly Empty interrupt will, during high traffic periods, be asserted only when the actual descriptor total falls below the second threshold. Thus, the overhead associated with interrupt processing in system 100 is reduced and packet latency minimized.

[0039] The foregoing detailed description and accompanying drawings are only illustrative and not restrictive. They have been provided primarily for a clear and comprehensive understanding of the present invention and no unnecessary limitations are to be understood therefrom. Numerous additions, deletions, and modifications to the embodiments described herein, as well as alternative arrangements, may be devised by those skilled in the art without departing from the spirit of the present invention and the scope of the appended claims.