Software interface between a parallel bus and a packet network

[0002] This application claims the benefit of U.S. Provisional Patent Application 60/152,849, filed Sep. 8, 1999, and of U.S. Provisional Patent Application 60/175,339, filed Jan. 10, 2000, which are incorporated herein by reference.

[0003] The present invention relates generally to computing systems, and specifically to systems that use packet-switching fabrics to connect a computer host to peripheral devices.

[0004] In current-generation computers, the central processing unit (CPU) is connected to the system memory and to peripheral devices by a parallel bus, such as the ubiquitous Peripheral Component Interface (PCI) bus. PCI bus protocols and rules are described in detail in the PCI Local Bus Specification, Revision 2.2 (1998), published by the PCI Special Interest Group (Hillsboro, Oreg.), which is incorporated herein by reference. Briefly, the PCI specification defines a set of bus commands, or cycles, each identified by a 4-bit code on the C/BE (Bus Command and Byte Enable) lines of the bus. A bus master, such as a CPU, sends commands to a target device on the bus by specifying a 32- or 64-bit bus address of the target device. Table I below lists the PCI bus command codes for reference. The different types of read and write commands in the table are based on the division of the overall PCI address space into Memory, I/O and Configuration address spaces.

[0005] As data path-widths grow, and clock speeds become faster, the parallel bus is becoming too costly and complex to keep up with system demands. In response, the computer industry is moving toward fast, packetized, serial input/output (I/O) bus architectures, in which computing hosts and peripheral are linked by a switching network, commonly referred to as a switching fabric. A number of architectures of this type have been proposed, including “Next Generation I/O” (NGIO) and “Future I/O” (FIO), culminating in the “InfiniBand” architecture, which has been advanced by a consortium led by a group of industry leaders (including Intel, Sun, Hewlett Packard, IBM, Compaq, Dell and Microsoft). Storage Area Networks (SAN) provide a similar, packetized, serial approach to high-speed storage access, which can also be implemented using an InfiniBand fabric.

[0006] Communications between a parallel bus and a packet network generally require a communications interface, to convert bus cycles into appropriate packets and vice versa. For example, a host channel adapter or target channel adapter can be used to link a parallel bus, such as the PCI bus, to the InfiniBand fabric. When the adapter receives data from a device on the PCI bus, it inserts the data in the payload of an InfiniBand packet, and then adds an appropriate header and error checking code, such as a cyclic redundancy check (CRC) code, as required for network transmission. The InfiniBand packet header includes a routing header and a transport header. The routing header contains information at the data link protocol level, including fields required for routing the packet within and between fabric subnets. The transport header contains higher-level, end-to-end transport protocol information. Similar headers are used in other types of packet networks known in the art, such as Internet Protocol (IP) networks.

[0007] Similarly, when the channel adapter receives a packet over the fabric for delivery to a device on the bus, it strips off the header and then generates an appropriate bus cycle, to convey the packet payload to the bus device. As a rule, the adapter is designed so that applications running on a CPU on the bus do not have to deal with network protocol considerations, such as generating data link or routing information and computing headers, and likewise so that applications running on a network host do not have to deal with bus cycle generation.

[0008] In contrast to interface device and methods known in the art, it is an object of some aspects of the present invention to provide a network interface that enables an application running on a bus device to access and program all parts of the header, as well as the payload, of a packet for transmission over a packet network.

[0009] It is a further object of some aspects of the present invention to provide a network interface that enables an application running on a network device to directly generate cycles on a parallel bus.

[0010] In some preferred embodiments of the present invention, a bridge device links a parallel bus, such as a PCI bus, to a packet network, such as an InfiniBand fabric. The bridge device comprises inbound and outbound packet registers, having addresses in an address space of the parallel bus, so that a master device on the bus, such as a CPU, can read and write to the registers. A software application running on the master device writes the entire substantive contents of a packet to the outbound packet register, including both the payload and the header, addressed to a target device on the network. The application is programmed so that the packet will conform with all of the relevant network protocols. The master device notifies the bridge device when it has finished writing the packet. The bridge device then transmits the packet over the network as is, preferably after having added a suitable error checking code. After the packet is sent, the outbound packet register is cleared.

[0011] The target device on the network can similarly send an inbound packet to the master device on the bus. The packet is preferably addressed to a management agent running on the bridge device, which places the packet data in the inbound packet register. The bridge device then generates an interrupt to the master device, notifying it that a packet has arrived. The application running on the master device reads the inbound packet register, which is then cleared and prepared to receive further packets.

[0012] In one of these preferred embodiments, the inbound and outbound packet registers are used to create a “virtual component” on the bus, whose effective bus address is the address of the registers. When the master device writes to this bus address, the bridge device hardware generates a packet to the specified target device. If this target device is an embedded processor, in another bridge device on the network, for example, the packet may elicit certain behaviors at the destination, such as assertion of interrupts or other external signals.

[0013] In further preferred embodiments of the present invention, the bridge device comprises a bus cycle register, additionally or alternatively to the inbound and outbound packet registers mentioned above. The bus cycle register is accessible for writing by devices on the network at a virtual network address that is associated with the bridge device. A software application running on a host device on the network sends a command packet to the bridge device at the network address, instructing the bridge device to write the bus address of a target device on the bus, as well as data to be delivered to the target device, into the bus cycle register. The application also writes an appropriate command code, such as a PCI C/BE code, to a designated field of the bus cycle register. The host device notifies the bridge device when it has finished writing to the bus cycle register, whereupon the bridge device generates a bus cycle with the command code, address and data specified by the host device.

[0014] Preferred embodiments of the present invention thus differ fundamentally from bridges known in the art, which attempt to make the bus/network interface transparent to devices on either side of the Interface insofar as possible. Transparent bridge implementations of this sort are described, for example, in two U.S. patent applications, by Kagan et al., entitled “Bridge between Parallel Buses over a Packet-Switched Network” and “Parallel Bus Communications over a Packet-Switching Fabric.” Both of these applications are assigned to the assignee of the present patent application and are incorporated herein by reference. In contrast to the present invention, these applications describe bridge devices that translate bus cycles into complete network packets, and vice versa, so that devices on the bus are relieved entirely of the need to deal with network protocols, and network devices are relieved of having to deal with bus cycles. Although this transparent translation model is useful in most software applications that use the bridge, there are some applications, such as network management applications, in which it is desirable for a processor on the bus to be able to read and write packets to the network directly, without the channel adapter as intermediary. Bus and system management applications running on the network may similarly need to generate certain bus cycles. The preferred embodiments described herein provide a novel solution to these needs.

[0015] There is therefore provided, in accordance with a preferred embodiment of the present invention, a bridge device, for coupling a parallel bus to a packet network, the device including:

[0016] a bus interface adapter, coupled to the parallel bus so as to receive bus cycles from a master device on the bus;

[0017] an outbound packet register, having a bus address in an address space of the bus and adapted to store an outbound network address header and payload data written to the bus address of the register by the master device in one or more of the bus cycles received by the bus interface adapter; and

[0018] a network interface adapter, coupled to the network so as to transmit over the network an outbound packet containing the outbound network address header and payload data from the register, to a target device on the network specified by the network address header.

[0019] Preferably, the network address header includes a network routing header, wherein the network interface adapter is adapted to transmit the packet substantially without computation of network routing information for the packet beyond that contained in the network address header written by the master device.

[0020] Additionally or alternatively, the network interface adapter is adapted to compute and append an error checking code to the packet.

[0021] Preferably, the device includes a register status indicator, which is set to indicate that the entire network address header and payload data have been written to the outbound packet register, wherein the network interface adapter transmits the packet responsive to setting the status indicator.

[0022] Further preferably, the bus address of the outbound packet register includes a first bus address, and the device includes an inbound packet register, having a second bus address in the address space of the bus and adapted to receive and store an inbound network address header and payload data from an inbound packet received over the network by the network interface adapter, wherein the inbound address header and payload data are read by the master device from the inbound packet register in a further one or more of the bus cycles received by the bus interface adapter. Most preferably, the outbound network address header specifies a local identifier associated with the network interface adapter as a sending address of the outbound packet, and wherein responsive to the sending address, the target device addresses the inbound packet to the local identifier.

[0023] In a preferred embodiment, the bus includes a Peripheral Component Interface (PCI) bus, and the network includes an InfiniBand fabric. Preferably, the bus interface adapter includes a channel adapter, and the network interface adapter includes a switch.

[0024] There is also provided, in accordance with a preferred embodiment of the present invention, a bridge device for coupling a packet network to a parallel bus, the device including:

[0025] a network interface adapter, coupled to the network so as to receive over the network an inbound packet containing bus cycle information, including a bus address and command code;

[0026] a bus cycle register, adapted to receive and store the bus cycle information; and

[0027] a bus interface adapter, coupled to the parallel bus so as to generate a bus cycle of a type specified by the command code, and addressed to the bus address, as specified in the information stored in the bus cycle register.

[0028] Preferably, the bus cycle register is associated with a virtual address on the network, to which a host device on the network writes the bus cycle information by means of the inbound packet.

[0029] Additionally or alternatively, the bus cycle information further includes data to be written to the bus address.

[0030] Preferably, the device includes a register status indicator, which is set to indicate that all of the bus cycle information has been written to the bus cycle register, wherein the bus interface adapter drives the bus cycle responsive to setting the status indicator.

[0031] There is additionally provided, in accordance with a preferred embodiment of the present invention, a method for coupling a parallel bus to a packet network, including:

[0032] receiving one or more bus cycles from a master device on the bus, addressed to a bus address of an outbound packet register in an address space of the bus, the cycles conveying over the bus an outbound network address header and payload data;

[0033] incorporating the outbound network address header and payload data in an outbound packet; and

[0034] transmitting the outbound packet over the network to a target device on the network specified by the network address header.

[0035] There is further provided, in accordance with a preferred embodiment of the present invention, a method for coupling a packet network to a parallel bus, including:

[0036] receiving over the network an inbound packet containing bus cycle information, including a bus address and command code; and

[0037] generating a bus cycle of a type specified by the command code and addressed to the bus address, as specified by the bus cycle information.

[0038] The present invention will be more fully understood from the following detailed description of the preferred embodiments thereof, taken together with the drawings in which:

[0039] FIG. 1 is a block diagram that schematically illustrates a system for bridging between a parallel bus and a packet-switched network, in accordance with a preferred embodiment of the present invention;

[0040] FIG. 2 is a block diagram that schematically illustrates details of a bridging device in the system of FIG. 1, in accordance with a preferred embodiment of the present invention;

[0041] FIG. 3 is a flow chart that schematically illustrates a method for transmission of packets over a network by a device on a parallel bus, in accordance with a preferred embodiment of the present invention;

[0042] FIG. 4 is a flow chart that schematically illustrates a method for receiving packets over a network for delivery to a device on a parallel bus, in accordance with a preferred embodiment of the present invention; and

[0043] FIG. 5 is a flow chart that schematically illustrates a method for generating cycles on a parallel bus by a device linked to the bus by a packet network, in accordance with a preferred embodiment of the present invention.

[0044] FIG. 1 is a block diagram that schematically illustrates a system 20 for bridging between a parallel bus 28, such as a PCI bus, and a network 34, such as an InfiniBand (IB) switch fabric, in accordance with a preferred embodiment of the present invention. In this embodiment, a master device, such as a central processing unit (CPU) 22, is coupled to communicate via a system bus with a system controller 24 and a system memory 26, as is known in the art. CPU 22 is coupled through the system controller to bus 28, over which it communicates with peripheral devices 30. A bridge device 32 on bus 28 also enables the CPU to communicate over network 34 with a network device 36. The network device typically comprises a host or I/O device, but may alternatively comprise substantially any element in or connected to network 34.

[0045] Further details of the design and operation of system 20, and particularly of bridge device 32, are described in the above-mentioned U.S. patent applications. While preferred embodiments are described hereinbelow with reference to the PCI bus and the IB fabric, it will be understood that the principles of the present invention may be applied, mutatis mutandis, in communication between parallel buses and networks of other types and obeying different standards, such as Internet Protocol (IP) networks and Asynchronous Transfer Mode (ATM) networks.

[0046] In order to communicate with network device 36, CPU 24 issues an appropriate PCI write cycle to an address in the address space of PCI bus 28 that is assigned to bridge device 32. The bridge device transmits an IB packet, based on the data written by CPU 24, over network 30 to device 36. The network device typically returns an appropriate response back to bridge 32, which is read over bus 28 by the CPU, using an appropriate PCI read cycle. Alternatively or additionally, network device 36 may instruct bridge device 32 directly to generate certain cycles on PCI bus 28, addressed either to CPU 22 or to one of devices 30, for example. The structure of bridge device 32 and the mechanisms used in these communication functions are described in detail hereinbelow.

[0047] FIG. 2 is a block diagram that schematically illustrates details of bridge device 32, in accordance with a preferred embodiment of the present invention. Device 32 comprises a channel adapter 40, which is coupled to PCI bus 22, and a switch 42, coupled to network 30. In the present example, the channel adapter preferably comprises a host channel adapter (HCA), used in communication between host devices, such as CPU 22 and network 34. Alternatively, the channel adapter may comprise a target channel adapter (TCA). A PCI cycle register 54 is associated with adapter 40. The functionality of this register, which is accessible to devices on network 34 for read and write operations, is described in detail hereinbelow with reference to FIG. 5.

[0048] The structure and function of switch 42 is preferably as described in the above-mentioned provisional patent applications 60/152,849 and 60/175,339. The design of switch 42 is beyond the scope of the present patent application, however, and substantially any packet switch meeting the requirements of network 34 may be used in the context. In fact, for the purposes of the present invention, switch 42 serves essentially as a network interface, and may be replaced by any other suitable type of network interface element known in the art.

[0049] A subnet management agent (SMA) 44 (also referred to as a fabric service agent—FSA) performs management and control functions within bridge device 32. Preferably, the SMA is a software process running on a suitable processing unit or controller in device 32, or on a controller external to device 32. Alternatively or additionally, the SMA may comprise dedicated logic circuits.

[0050] SMA 44 communicates with dedicated memory registers 46 in bridge device 32, including a cause register 48, an inbound packet register 50 and an outbound packet register 52. Registers 48, 50 and 52 have addresses in the address space of bus 28, so that they can be written to and read from directly by software applications running on the bus. Each of registers 50 and 52 preferably comprises a buffer corresponding in size to a maximum size of a packet that can practically be sent over network 34, for example, 512 bytes. These registers are described further hereinbelow with reference to FIGS. 3 and 4.

[0051] The structure and function of cause register 48 are described in detail in a U.S. patent application by Kagan et al., entitled “Remote Event Handling in a Packet Network,” which is assigned to the assignee of the present patent application, and whose disclosure is incorporated herein by reference. In brief, the cause register comprises multiple fields, which correspond to different types of events that may occur in or in connection with bridge device 32. When one of the events occurs, a bit or bits in the corresponding cause register field are set. SMA 44 is programmed to respond in appropriate ways to certain events appearing in the cause register, for example by transmitting a packet through switch 42 over network 34, or by initiating an interrupt through adapter 40 to bus 28.

[0052] FIG. 3 is a flow chart that schematically illustrates a method by which CPU 22 assembles and transmits a packet over network 34 to network device 36, in accordance with a preferred embodiment of the present invention. In order to send a data packet to network device 36, a software application running on CPU 22 loads the packet into outbound packet register 52, using appropriate PCI write cycles addressed to the register. The packet as loaded includes the entire header required by the InfiniBand fabric, containing both the routing header and the transport header, along with the data payload. The software application computes the packet header based on a knowledge of the necessary network protocol and topology information, so that the packet can be routed successfully through the network using the header information, without involvement by bridge device 32 in setting header parameters. CPU 22 may also calculate the packet CRC code and load the code into register 52 along with the header and payload. Preferably, however, this function is left to dedicated hardware in switch 42, which carries out CRC calculation and checking for all packets conveyed through the switch, as is known in the art.

[0053] After it has loaded the entire packet into outbound packet register 52, the software application on CPU 22 sets a “sysout full” bit in cause register 48. Setting this bit causes SMA 44 to initiate a packet send procedure. The SMA passes the packet contents from register 52 to switch 42, which preferably calculates and appends the CRC code, as described above. The switch sends the packet on through network 34 to device 36. Meanwhile, SMA 44 clears the sysout full bit and, preferably, writes a “sysout done” indication to the cause register. This step tells CPU 22 that it can now write another packet to register 52.

[0054] FIG. 4 is a flow chart that schematically illustrates a method by which CPU 22 receives packets sent to it by network device 36, in accordance with a preferred embodiment of the present invention. This method is typically used when the network device has to respond to the outbound packet sent to it by the CPU using the method of FIG. 3. The routing header of the outbound packet assembled by the CPU in register 52 preferably specified a local identifier (LID) of SMA 44 as the originating network address of the packet. Therefore, in response, network device 36 sends an inbound packet to this same LID.

[0055] When SMA 44 receives the packet, it first checks to determine what type of packet it is. As the management agent for bridge device 32, SMA 44 also receives management packets from network 34 and is programmed to process and handle these packets itself. When the inbound packet from network device 36 is received at device 32, however, the SMA reads its contents and determines that it is not capable of processing them. Therefore, the SMA stores the packet in inbound packet register 50 and sets a “sysin full” bit in cause register 48.

[0056] Setting the sysin full bit causes the software application on CPU 22 to perform a bus read cycle, in order to read the contents of inbound packet register 50. Preferably, setting the sysin full bit causes an interrupt to the CPU to be invoked, so that the CPU reads register 50. Alternatively, the CPU may poll this bit periodically to determine when a new packet arrives. In either case, after the CPU has read register 50, the software clears sysin full, so that the next packet from device 36 can be written to the register. Any packets arriving at SMA 44 while sysin full is set are dropped, and an overrun event is preferably generated in the bridge device.

[0057] FIG. 5 is a flow chart that schematically illustrates a method used by network device 36 to directly generate cycles on bus 28, in accordance with a preferred embodiment of the present invention. The method uses PCI cycle register 54, whose structure is preferably as summarized in Table I below:

[0058] In order to generate a cycle on bus 28, network device 36 writes PCI address, data, command and byte enable information to the appropriate fields in register 54. Preferably, the network device writes the information as the payload of a remote direct memory access (RDMA) write packet that it sends to a virtual network address of register 54 in bridge device 32, which is known to software running on the network device. When the packet is received in the register, the “go” bit is set by the bridge device.

[0059] Setting the go bit causes channel adapter 40 to drive the specified cycle to PCI bus 28, by placing the data in the fields of register 54 onto the bus at the appropriate cycle times. Upon completion of the cycle, the adapter clears the go bit and updates the status field to indicate whether the cycle was successfully received at the target device (CPU 22 or one of the other bus devices 30), or whether an abnormal condition occurred. Typically, the abnormal condition could be a retry or disconnect response from the target device, or a master- or target-abort due to an addressing error, for example. If the cycle generated by network device 36 was a PCI read cycle and was completed successfully, the network device may subsequently read the data returned by the target device on the bus from the PCI data field of register 54.

[0060] It will be appreciated that the preferred embodiments described above are cited by way of example, and that the present invention is not limited to what has been particularly shown and described hereinabove. Rather, the scope of the present invention includes both combinations and subcombinations of the various features described hereinabove, as well as variations and modifications thereof which would occur to persons skilled in the art upon reading the foregoing description and which are not disclosed in the prior art.