Fault-tolerant maintenance bus architecture

[0002] 1. Field of the Invention

[0003] This invention relates to fault-tolerant computer systems and more particularto a dedicated maintenance bus for use with such computer systems.

[0004] 2. Background Information

[0005] Fault-tolerant computer systems are employed in situations and environments that demand high reliability and minimal downtime. Such computer systems may be employed in the tracking of financial markets, the control and routing of telecommunications and in other mission-critical functions such as air traffic control.

[0006] A common technique for incorporating fault-tolerance into a computer system is to provide a degree of redundancy to various components. In other words, important components are often paired with one or more backup components of the same type. As such, two or more components may operate in a so-called lockstep mode in which each component performs the same task at the same time, while only one is typically called upon for delivery of information. Where data collisions, race conditions and other complications may limit the use of lockstep architecture, redundant components may be employed in failover mode. In failover mode, one component is selected as a primary component that operates under normal circumstances. If a failure in the primary component is detected, then the primary component is bypassed and the secondary (or tertiary) redundant component is brought on line. A variety of initialization and switchover techniques are employed to make a transition from one component to another during runtime of the computer system. A primary goal of these techniques is to minimize downtime and corresponding loss of function and/or data.

[0007] Fault-tolerant computer systems are often costly to implement since many commercially available components are not specifically designed for use in redundant systems. It is desirable to adapt conventional components and their built-in architecture whenever possible. All modem computer systems have particular capabilities directed to control and monitoring of functions. For example, large microprocessor chips such as the Pentium III™, available from Intel Corporation of Santa Clara, Calif., are designed to operate within a specific temperature range that is monitored by a commercially availble environmental/temperature-sensing chip. One technique for interconnecting such an environmental monitor or other monitoring and control devices is to utilize a dedicated maintenance bus. The maintenance bus is typically separate system's main data and control bus structure. The maintenance bus generally connects to a single, centralized point of control, often implemented as a peripheral component interconnect (PCI) device.

[0008] However, as discussed above, conventional maintenance bus architecture is not specifically designed for redundant operation. Accordingly, prior fault-tolerant systems have utilized a customized architecture for transmitting monitor and control signals over the system's main buses (or dedicated proprietary buses) using, for example, a series of application specific integrated circuits (ASICs) mounted on each circuit board being monitored. To take advantage of current, commercially available maintenance bus architecture in a fault tolerant computing environment, a more comprehensive and costeffective approach is needed.

[0009] Accordingly, it is an object of this invention to provide maintenance bus architecture having a high degree of fault-tolerance. This maintenance bus architecture should be interoperable with commercially available components and should allow a fairly high degree of versatility in terms of monitoring and control of important computer system components.

[0010] This invention overcomes the disadvantages of the prior art by providing a fault-tolerant maintenance bus architecture that includes two maintenance buses interconnecting each of a plurality of printed circuit boards, termed “parent” circuit boards. The two maintenance buses are each connected to a pair of system management modules (SMMs) that are configured to perform a variety of maintenance bus activities. The SMM can comprise any acceptable device for driving commands on the maintenance bus arrangement. Within each parent board are a pair of redundant bridges both having a unique address. One bridge is connected to the first maintenance bus while a second bridge is connected to the second maintenance bus of the pair. A child maintenance bus interconnects the two bridges through a “child” printed circuit board. The introduction of a separate board to implement the child maintenance bus can be useful, but is not essential according to this invention. The child maintenance bus is itself interconnected with a variety of monitor and control functions on maintenance bus-compatible subsystem components. The SMMs can address components on each child printed circuit board individually and receive appropriate responses therefrom. In the event of a bus or bridge failure, the SMM can still communicate with the child subsystem components via the redundant bus and bridge.

[0011] The bridge can include an interconnection to a further bridge. This remote bridge can, itself, be interconnected to additional microprocessors and associated memory. The remote bridge is addressed through one of the parent board's bridges so the communication to and from the SMM can occur. The SMM can be interconnected with a variety of other computer system peripherals and components, and can be accessed over a local network or through an Internet-based communication network.

[0012] The foregoing and other objects and advantages of the invention will become more clear with reference to the following detailed description as illustrated by the drawings in which:

[0013] FIG. 1 is a block diagram showing an overview of a fault-tolerant maintenance bus architecture according to this invention;

[0014] FIG. 2 is a more detailed block diagram showing one parent and child printed circuit board implementing a fault-tolerant maintenance bus according to this invention; and

[0015] FIG. 3 is the board of FIG. 2 including a bridge for accessing a remote microprocessor board according to an alternate embodiment.

[0016] FIG. 1 details a fault-tolerant maintenance bus architecture according to a preferred embodiment of this invention. A pair of parent maintenance buses MBA and MBB are shown. These maintenance buses are identical in architecture and can be implemented as a combination of cables, circuitry and circuit board traces. The buses MBA and MBB interconnect with a plurality of input/output (I/O) slots and pin locations within a cabinet that may contain a plurality of circuit boards. The parent maintenance buses MBA and MBB can also jump between cabinets in a larger computer system. It is generally contemplated that the buses are implemented in a multi-cabinet fault-tolerant server system, but the architecture according to this invention can be utilized in a variety of fault-tolerant computing configurations. According to a preferred embodiment, the buses MBA and MBB are each two-wire buses designed to take advantage of integrated circuit components utilizing the I²C bus standard. The I²C bus is a proprietary design of Philips Semiconductors of the Netherlands. This standard has become widely adopted for consumer electronics and various circuit applications, and is now supported by a large number of commercially available monitoring and control devices. Details on the use of the I²C bus can be found in the I²C-bus and how to use it(including specifications), April 1995 update, Chapter 3 by Philips Semiconductors. Typically, the bus is clocked at a speed of approximately 10 Kbytes/Sec. While I²C is employed as the bus standard according to a preferred embodiment of this invention, it is expressly contemplated that other maintenance bus standards can be utilized according to the teachings of this invention with appropriate modifications.

[0017] To avoid signal loss over long distances, the parent buses MBA and MBB are amplified to generate a signal at 5V×30 mA. A variety of bus amplification circuits can be used. In particular, commercially available hardware bus extenders are employed. The amplified bus operates at a gain that is ten times the normal operating range for an I²C bus (5V×3 mA). As described further below, this difference in operating level between the parent bus and various circuit components is compensated-for (on both sides) using the extender hardware.

[0018] In FIG. 1, three circuit board assemblies 102, 104 and 106 are shown. Each of these circuit board assemblies is interconnected with the bus pair (MBA and MBB). These board assemblies can represent a variety of computer system components. For example, the boards can together comprise a set of redundant identical boards or a set of separate functions including a central processing unit (CPU) board, “front panel” board and input/output (I/O) board. Each board assembly 102, 104 and 106 is defined functionally as a parent printed circuit board 112, 114 and 116 and an associated child printed circuit board 122, 124 and 126. As discussed above, while a separate board to implement the child maintenance bus can be useful according to an embodiment of this invention, it is not required. In general, the division between the parent board and child board is is somewhat arbitrary, and the actual physical structure for one or more boards can be implemented as a single plug-in printed circuit card residing in a connector socket or slot on a larger cabinet-based motherboard.

[0019] Also interconnected with the parent bus pair MBA and MBB are a pair of redundant system management modules (SMMs) identified herein as SMMA 128 and SMMB 130. Each SMM is a microprocessor-based component. The SMMs 128 and 130 each reside on a PCI bus 132 and 134. The SMM performs a variety of functions and includes both Ethernet and modem capabilities allowing it to interconnect with the computer operating system and other network communication structures (block 136). The SMM may also include other unrelated system components such as a video driver chip. The SMM is particularly based around a Motorola Power PC™ 860T microprocessor utilizing the VxWorks real-time operating system available from Wind River Systems, Inc. of Alameda, Calif. The term “system management module” (or SMM) is defined broadly to include any acceptable device for driving commands on the maintenance bus arrangement. While a microcontroller described above is used in a preferred embodiment, the SMM can be an application specific integrated circuit (ASIC), a programmable logic array, a microprocessor unit or any other command originator interconnected with the maintenance bus arrangement. The term “command module” is also used to define the SMM in its various possible embodiments.

[0020] Each SMM includes a pair of I²C buses 138 and 140. Each pair is connected with a respective bus from the parent bus pair MBA and MBB. The SMMs are configured to operate in failover mode. In other words, SMMA operates under normal circumstances. In this mode SMMB monitors and communicates with SMMA over the shared I²C bus, ready to take over for SMMA if it fails. If a failure is detected, then SMMB takes over operation but is otherwise largely idle, during normal run time. The function of the SMMs is described in further detail below. In summary, the SMMs carry information about components on each of the board assemblies 102, 104 and 106. The SMMs use this information to monitor and generally control the board assemblies. This information may be transferred to other parts of the computer system and over a network via the PCI bus.

[0021] Each parent board 112, 114 and 116 includes various data processing, display and communication capabilities in accordance with its purpose. Each board 112, 114 and 116 also includes a respective CPU (CPU1, CPU2 and CPU3) 152, 154 and 156, respectively.

[0022] Each CPU can comprise an Intel Pentium III™, Xeon™ or any other acceptable microprocessor having I²C or equivalent maintenance bus architecture. Each board 112, 114 and 116 is interconnected with the parent buses MBA and MBB at various interconnection points, where appropriate. Since the parent bus is amplified by a gain by approximately ten times the normal I²C operating level, interconnections with the parent buses can be made via bus extenders to be described further below.

[0023] Each parent board 112, 114 and 116 also includes a pair of interconnections 160 and 162 with each of the respective parent maintenance buses MBA and MBB. The interconnections 160 and 162 link to respective bridges 192 and 194. These bridges, in turn interconnect with respective child maintenance buses CB1, CB2 and CB3 to interconnect child board components. On each child board 122, 124 and 126 resides various control and monitoring subsystem components 172, 174 and 176, respectively. The subsystem components are described in further detail below. These components are each in communication with the maintenance bus using the preferred I²C standard. The interconnection between each parent board and child board occurs via a pair of bridges 192 and 194. Each bridge is essentially identical in architecture, each pair of bridges 192 and 194 has the same different address for communication with the SMMs. The address of the bridge pair on each board, however differs so that the SMM can uniquely address a specific board. Addresses are established based upon the pin and socket arrangement for the respective bridge. It is useful to assign the same address to both bridges 192, 194 in the pair since they each reside on a different bus (MBA or MBB). For a given transaction, the SMM utilizes only one of the two bridges on a parent board to accomplish a task. The other, unused bridge in the pair can be used if the SMM cannot complete the transaction with the original bridge. As noted above, in order provide an amplified parent bus signal and an appropriate signal level to the subsystem bridges, bus extender hardware is in employed. The bus extender hardware is available from Phillip Semiconductors under part number 82B715. Using amplified parent buses, approximately thirty or more loads can be carried. The extender acts as a buffer for signals traversing the extender hardware providing the necessary amplification and deamplification. Extender components 196 are provided between the parent buses MBA and MBB and corresponding bridge interconnections 160 and 162. While not shown, interconnections 160 and 162 can also include appropriate series resistors and FET triggers in line with the extender components 196 in accordance with the 82B715 hardware manufacturer's data sheet.

[0024] The bridges 192 and 194 each act as store-forward devices in the transfer of I²C signals into and out of the child board subsystem. In other words, the bridges receive packetized signals from the SMMs and transfer them to appropriate I²C-compatable maintenance bus ports on subsystem components. Likewise, the bridges receive signal from subsystem components and transfers them back to the SMMs. In order to provide desired fault-tolerance, two bridges 192 and 194 are employed, each communicating with one of the dual parent buses MBA and MBB. In this manner, the failure of a single bridge or parent bus does not cause a lost of connection between the subsystem components and SMMs. This is because each child bus CB1, CB2 and CB3 is interconnected with both bridges simultaneously. The subsystem components are accessed via the child bus. A reset connection (R1, R2 and R3) and power connection (P1, P2 AND P3) extend from each bridge in the pair. A reset and/or power command from an SMM to the active bridge in the pair is used to power-up or reset the underlying board assembly. The SMMs are configured to provide independent reset and power commands to the bridges 192 and 194, to allow powering and reset of each underlying board through the maintenance bus arrangement. In general, the active bridge performs power-up. However, the bridges are configured to handshake, or otherwise communicate, to ensure that the board hardware is functioning properly before power-up occurs generally within the board.

[0025] According to a preferred embodiment, each bridge 192, 194 comprises a commercially available Intel 87C54 microcontroller. This circuit package includes a built-in programmable storage device (an erasable programmable read-only memory EPROM) and 256 bytes of random access memory (RAM). This package is relatively low-cost and complete. Data traveling over the I²C bus is buffered in the RAM while basic routing and power control functions are preprogrammed into the bridge microcontroller EPROM. Though the 87C54 is the preferred embodiment, any microcontroller with sufficient I/O ports to drive both parent and child maintenance buses could instantiate the bridges 192 and 194.

[0026] With further reference to FIG. 2, the exemplary parent board assembly 102 is shown in further detail. Particularly, the subsystem components 172, interconnected with the I²C bus are illustrated. The electrically erasable programmable read only memory (EEPROM) carrying the board identification (ID), generally termed the IDEEPROM 202, is provided on the bus. In addition, a light-emitting diode (LED) monitor 204 is provided. This LED provides a visible indication of the status of the board for an operator of the board. In addition, an environmental monitor chip 206 having I²C compatibility is provided. This chip typically monitors temperature and other important functions and transmits appropriate data and/or alarms regarding environment. Microprocessor information from CPU1 (152) is also interconnected with the bus CB1 via an I²C interconnection. The CPU support information 208 is transmitted over the I²C bus, as well as other important status data. I²C interconnections with the dual inline memory module sockets (DIMMS) 210 of the board assembly are also provided by the child bus CB1. In addition, other I/O ports 212 with I²C capabilities may be serviced by the child bus CB1.

[0027] It is generally contemplated that the SMMs communicate with the bridges 192 and 194 via data packets, sent over the I²C bus with appropriate destination addresses. In general, each bridge and subsystem component contains its own unique address on the maintenance bus that make it identifiable by the SMMs. The SMMs have knowledge of the subsystem components on each board. Packets sent to and from the SMM have the bridge identification and the data within the packet is used to identify the particular subsystem device. A variety of protocols and communication techniques can be used according to this invention. Heretofore, I²C connections have operated using a highly simplified communication scheme without the benefit of addressing and protocol techniques.

[0028] Because of the fault-tolerant nature of the bus arrangement and bridge system, addressing of control and monitor functions between the SMMs and the appropriate bridge are highly desirable. One such protocol is taught in the co-pending, commonly owned U.S. patent application Ser. No. 09/548,202, entitled Fault-Tolerant Maintenance Bus Protocolby A. Charles Suffin, which is expressly incorporated herein by reference. Briefly, the protocol is controlled by the active SMM and includes a variety of data and response byte code proceeded by an appropriate address. The byte codes activate and/or poll the various subsystem components which return corresponding responses addressed to the active SMM. Certain functions such as power-up and reset are executed by the bridge itself, while others are past through the bridge onto the child bus CB1. Packets can include checksum features to enhance fault-tolerance. Both the bridge and SMM count the number of received bytes and compare that result to the checksum.

[0029] If the numbers do not match, then the packet is discarded. Other features of the exemplary protocol are described further in the incorporated patent application.

[0030] Reference is now made to FIG. 3, which again illustrates the exemplary board assembly 102. The subsystem 172 of this board includes the set subsystem components described above with reference to FIG. 2. In addition, another bridge 302 is interconnected the child bus CB1. This bridge is similar in configuration to the bridges 192 and 194, and can be constructed from the same type of microcontroller circuit. The bridge 302 includes another discrete address that is recognized by the SMM so that data is transferred via the bridges 192, 194 to the subordinate bridge 302 as if it were any travelling to any other subsystem component. Note that the bridge addressing arrangement according to this embodiment enables components to be accessed with relatively small addresses (typically one byte). Once received, the bridge 302 stores and forwards the message to the I²C-compatable ports on further computer circuitry 304. Note that the CPU information block 208 is connected through the subordinate bridge 302 according to FIG. 3. According to this embodiment, the processor information is located behind the child bridge, accounting for the depicted arrangement. In this example, the circuitry 304 includes another microprocessor (such as an Intel Xeon™) and/or associated memory and other peripherals. The above-described protocol enables messages to be transferred from the child bus through bridges to additional, subordinate bridges (such as bridge 302). Further components, such as circuitry 304, can be accessed through these subordinate bridges.

[0031] It should now be clear that the foregoing architecture enables an effective and low-cost technique for implementing a fault-tolerant maintenance bus within a number of separate computer components.

[0032] The foregoing has been a detailed description of a preferred embodiment. Various modifications and additions can be made without departing from the spirit and scope of the invention. For example, while the maintenance bus is implemented as an I²C standard, it can be implemented in any other acceptable standard and the number of lines in the bus can be varied from the two lines shown. While a serial maintenance bus is utilized, it is contemplated that a parallel bus can be employed according to an alternate embodiment. Various components such as bridges and SMMs can be implemented using a variety of commercially available and customized circuits. Accordingly, this description is meant to be taken only by way of example, and not to otherwise limit the scope of the invention.