Fault Tolerant Networking Architecture

1. Field Of The Invention

This invention relates generally to computer networks, and in particular to a fault tolerant network having a semaphore box for controlling access to shared peripherals by a plurality of computers.

2. Description of Related Art

Multiprocessor systems typically include some method of providing interprocessor communication. For example, interprocessor communication through a shared main memory is typically referred to as a "loosely coupled" computer system. Interprocessor communication through shared registers is typically referred to as a "tightly coupled" computer system. Prior patent of the Assignee of the present invention, Cray Research, Inc., disclose various forms of interprocessor communication.

One such prior patent is U.S. Pat. No. 4,636,942, issued Jan. 13, 1987, to Chen et al., which patent is incorporated herein by reference. This patent discloses a computer vector multiprocessing control wherein a pair of processors are provided and each are connected to a central memory through a plurality of memory reference ports. Processors are further connected to a plurality of shared registers, including registers for holding scalar and address information, and registers for holding information to be used in coordinating the transfer of information through the shared registers.

Another prior patent is U.S. Pat. No. 4,661,900, issued Apr. 28, 1987, to Chen et al., which patent is incorporated herein by reference. This patent discloses a flexible chaining method and apparatus wherein a part of processors are connected to a central memory through a plurality of memory reference ports. The processors are connected to a plurality of shared registers that may be directly addressed by either processor, and which hold scalar and address information in registers for holding information to be used in coordinating the transfer of information through the shared registers.

Still another prior patent is U.S. Pat. No. 4,754,398, issued Jun. 28, 1988, to Pribnow, which patent is incorporated herein by reference. This patent discloses an interprocessor communication system for a multiprocessor system that includes a plurality of clusters having a plurality of semaphore registers and information registers.

Whatever the merits of these prior patents for controlling interprocessor communication, they do not achieve the benefits of the present invention.

The present invention discloses a fault tolerant network for computers that includes a semaphore box for controlling access to shared peripherals. The semaphore box is comprised of two major sections: an I/O section and a semaphore section. The semaphore section contains two sets of semaphores: a first set comprising reservation semaphores for the shared peripherals and a second set comprising heartbeat semaphores for the sharing computers. The first set is used to reserve a particular peripheral for a particular computer; the second set provides a "heartbeat" to prevent reservation semaphores from being set indefinitely in the event communication with a particular computer is lost.

The heartbeat semaphores are arranged in an array. Each time a semaphore command is received, a row of heartbeat semaphores is set. When the command completes, a column of heartbeat semaphores is returned to the requesting computer and then cleared. If the heartbeat semaphore in a particular position of the column is set, then the associated computer has accessed the semaphore box sometime between the current access of the requesting computer and its prior access. Conversely, if the particular heartbeat semaphore is cleared, then the associated computer has not accessed the semaphore box during the period. If the particular heartbeat semaphore for the associated computer remains cleared for some number of consecutive accesses, the requesting computer should conclude that the associated computer has lost communication with the semaphore box.

Referring now to the drawings, in which like reference numbers represent like elements throughout the several views:

FIG. 1 is a block diagram of a system configuration according to the preferred embodiment;

FIG. 2 is a block diagram of the semaphore box in the preferred embodiment;

FIG. 3 is an illustration of an array of heartbeat semaphores in the preferred embodiment;

FIG. 4 illustrates a channel command word in the preferred embodiment;

FIG. 5 illustrates four possible function codes of the channel co word in the preferred embodiment; and

FIG. 6 illustrates the format of the channel status word in the preferred embodiment.

In the following description of the preferred embodiment, reference is made to the accompanying drawings which form a part hereof, and in which is shown by way of illustration a specific embodiment in which the invention may be practiced. It is understood that other embodiments may be used and structural changes may be made without departing from the scope of the present invention.

In the following description, the term "semaphore" is often used. In the preferred embodiment, this term refers to a memory cell that is shared by a plurality of processors to provide a form of communication between the processors by indicating when significant events have taken place.

FIG. 1 is a block diagram of a system configuration according to the preferred embodiment of the present invention. The present invention discloses a fault tolerant network for computers 10 that includes a semaphore box 16 for controlling access to shared peripherals 12.

The computers 10 may be YMP/16 computers of the type manufactured by Cray Research, Inc., the Assignee of the present invention. The peripherals 12 may be Solid-state Storage Devices (SSDs) of the type described in either U.S. Pat. No. 4,630,230, issued Dec. 16, 1986, to Sundet (which patent is incorporated herein by reference), or in U.S. Pat. No. 4,951,246, issued Aug. 21, 1990, to Fromm et al. (which patent is incorporated herein by reference). Those skilled in the art, however, will recognize that other computers 10 and other peripherals 12 could substituted therefor.

Each peripheral 12 connects to the computers 10 via a channel 15. Each computer 10 connects to a semaphore box 16 via a channel 14. Access to the shared peripherals 12 is coordinated among the computers 10 by means of communication through the semaphore box 16.

A block diagram of the semaphore box 16 is shown in FIG. 2. The semaphore box 16 is comprised of two major sections: a semaphore section and an I/O section.

The semaphore section contains two sets of semaphores: a first set comprising reservation semaphores for the shared peripherals 12; and a second set comprising heartbeat semaphores for the sharing computers 10. The first set is used to reserve a particular peripheral 12 for a particular computer 10; the second set provides a "heartbeat" to prevent reservation semaphores from being set indefinitely in the event communication with a particular computer 10 is lost. In the preferred embodiment, there are three identical copies, or groups 18, 20, and 22, of the two sets arranged in a triple module redundant configuration.

The reservation semaphores can be individually set cleared, or tested by commands received by the semaphore box 16 from the computers 10. In the preferred embodiment, each reservation semaphore is four bits long. One bit contains reservation information associated with one of the peripherals 12; the other three bits are a port identifier field containing a number of the input port 24 which most recently set the reservation semaphore. The port identifier field thus identifies a particular computer 10 connected to the associated input port 24. The port identifier field is updated each time the reservation semaphore is set.

FIG. 3 is an illustration of an array 30 of heartbeat semaphores, wherein each heartbeat semaphore is identified by a concatenated row-column value. The array 30 measures eight by eight, but those skilled in the art will recognize that any size array 30 may be used. The array 30 of heartbeat semaphores is used to prevent the reservation semaphores from remaining in a "set" condition after communication with a computer 10 has been lost. A reservation semaphore that remains "set" prevents access to the associated peripheral 12 by the other computers 10.

Each time a semaphore command is received by the semaphore box 16, a row of heartbeat semaphores is set. Preferably, the row number is the same as the number of the input port 24 which received the command, although other means of associating rows with computers 10 could be used. When the command completes, a column of heartbeat semaphores is returned to the requesting computer 10 by the output port 26 and then cleared. The column becomes part of the channel status word. Like the row number, the column number is preferably the same as the number of the input port 24 which received the command, although other means of associating columns with computers 10 could be used.

The heartbeat semaphores provide a means of determining whether the computers 10 connected to the semaphore box 16 are still active. This is best illustrated by the following example. Assume computer A wants to determine the running status of computer B. Each time computer B sends a command to the semaphore box 16, all heartbeat semaphores in row B are set. Each time computer A sends a command to the semaphore box 16, the heartbeat semaphores in column A are returned in the channel status word and then cleared. Since the rows and columns intersect, the Bth heartbeat semaphore in column A indicates whether computer B has accessed the semaphore box 16 in the period since the previous computer A access. If the heartbeat semaphore is set, computer B has accessed the semaphore box 16 sometime between the current access of computer A and the prior access of computer A. Conversely, if the heartbeat semaphore is clear, computer B has not accessed the semaphore box 16 during the period. If the heartbeat semaphore for computer B is clear for some number of consecutive accesses or some predetermined interval, then computer A concludes that computer B has lost communication with the semaphore box 16 and takes appropriate action. Those skilled in the art will recognize that the number of consecutive accesses or the predetermined interval is programmable. Thus, any computer 10 connected to the semaphore box 16 must periodically access the semaphore box 16 to keep its "heartbeat" alive.

Referring again to FIG. 2, the I/O section of the semaphore box 16 consists of input ports, voting circuits, and output ports. Each attached computer 10 communicates with the semaphore box 16 across a channel 14 which attaches to the semaphore box 16 at an input port 24 and an output port 26. Preferably, the computer 10 is attached to an output port 26 identically numbered or otherwise associated with an input port 24. Each input port 24 and output port 26 of the semaphore box 16 is logically independent from the others. Operations on the semaphore box 16 are accomplished by transmitting commands to the input port 24. The input port 24 then transmits the commands to all three of the semaphore groups 18, 20, and 22. (Note that in FIG. 2 only a portion of the connections between the input ports 24 and the semaphore groups 18, 20, and 22 are illustrated.) The semaphore groups 18, 20, and 22 transmit the results of the operation to a voting circuit 28. The three copies of the results are inspected by voting circuits 28. The voting circuits 28 detect an error if there is a difference in the execution results. If a single semaphore group 18, 20, or 22 is in error, its results are ignored by the voting circuits 28. If two semaphore groups 18, 20, or 22 fail, all comparisons will fail and the results of group 20 are used (although there is no assurance that the results of group 20 are correct). The output port 26 then transmits the results to the computer 10.

A port conflict occurs when more than one command is received by the semaphore section from the I/O section in the same clock period. Conflicts are resolved on a priority basis in the preferred embodiment, although other methods of resolving conflicts could be substituted therefor. In the preferred embodiment, lower numbered input ports 24 have priority over higher numbered input ports 24. Thus, requests from higher numbered input ports 24 are held until the semaphore response for the lower numbered input port 24 is returned to the I/O section. The requests are not queued, although alternative embodiments could easily implement such a scheme. Instead, in the preferred embodiment, if a lower numbered request is received while a higher number request is being held, the lower numbered request will again be honored first.

FIG. 4 illustrates a channel command word 32 in the preferred embodiment. The channel command word 32 consists of a single 64-bit word. It is transmitted over the channel 14 as four 16-bit parcels followed by a channel disconnect pulse. After an input port 24 receives the channel command word 32, it performs error checks to detect possible parity errors, command compare errors, or channel protocol errors. Detection of a channel error aborts the operation.

Each parcel of the channel command word 32 contains the same information, but the individual bit assignments differ in each parcel. Rearranging the bit assignments prevents a defective wire, connector pin, or associated circuits from causing a catastrophic error.

Eight bits of each parcel are used to hold a command. The remaining eight bits are unused, except for parity, and may contain any value. Of the eight command bits, two bits contain a semaphore function code and six bits contain a semaphore select code. The four possible function codes are described in FIG. 5. The semaphore select code determines which of the reservation semaphores are affected by the semaphore function code.

Each parcel is sent to a different semaphore group for execution: parcel 0 is sent to semaphore group 18; parcel 1 is sent to semaphore group 20; parcel 2 is sent to semaphore group 22; parcel 3 is discarded. Upon completion, the appropriate output port 26 receives results from each of the three semaphore groups 18, 20, and 22. Both reservation semaphore and heartbeat semaphore information is contained in the results. The three copies of the results are inspected by voting circuits 28 and an appropriate channel status word is transmitted by the output port 26 to the originating computer 10. If the command received from the input port 24 had been in error, the voting circuits 28 detect the error and prevent corruption of any data on the shared peripherals 12.

FIG. 6 illustrates the format of the channel status word 36, which consists of two 16-bit parcels followed by a channel disconnect pulse. The format is further described below.

The uppermost bit (T0) of the channel status word 36 is used only for a test and set operation. It reflects the state of the selected reservation semaphore at the time the function code is received by the semaphore section. Bit T0 is set to the value 1 if the reservation semaphore is initially set. Otherwise, a value of 0 is returned in this position if the reservation semaphore is initially clear, and the set operation has been performed.

Bit S0 reflects the condition of the selected reservation semaphore at the start of the current command.

Bits I0-I2 indicate the number of the input port 24 (having a value of 0-7) which most recently changed the state if the selected reservation semaphore.

The heartbeat status byte provides each computer 10 with information about the availability of the other computers 10 connected to the semaphore box 16. The bit position (H0-H7) of the heartbeat status corresponds to the input port 24 (0-7) which controls the setting of the particular heartbeat semaphore.

Bit A0 is 1 if an error is detected by the input port 24 or output port 26. Bit A0 is a summation of bits C2-C0, P5, P4, p3-p0, and P3-P0 further described below.

Bits C0-C2 (semaphore compare error) are set if there is an error in the execution results. The comparison is performed on 13 bits: the selected 4-bit semaphore, a semaphore test flag, and the 8-bit heartbeat semaphore column. A semaphore compare error could be caused by any of several conditions; a hardware malfunction; a command error; residue from a previous command error; or a semaphore group 18, 20, or 22 whose contents have not been fully restored following a power loss or maintenance action. If a single semaphore group 18, 20, or 22 is in error, its results are ignored by the voting circuits 28. If two semaphore groups 18, 20, or 22 fail, all comparisons will fail and the results of group 20 are used (although, there is no assurance that the results of group 20 are correct).

Bit C0 is set if information from semaphore group 18 fails to compare with either group 20 or 22. Similarly, bit C1 is set if information from semaphore group 20 fails to compare with either group 18 or 22. Bit C2 is set if information from semaphore group 22 fails to compare with either group 18 or 20.

Bits P5-P0 indicate an error was detected during receipt of the command by the input port 24. If P5-P0 are set, the command is aborted and parcel 0 of the channel status word 36 is invalid.

Bit P5 is set if the function codes do not match in all four parcels of the channel command word 32. This could be the result of a programming error or a data error in the channel command word 32. The requested operation is aborted and the command is not sent to the semaphore groups 18, 20, and 22. In the channel status word 36, the P5 bit (command compare error) is then set and parcel 0 of the channel status word 36 is invalid.

Bit P4 indicates that a channel protocol error was detected by the input port 24 or output port 26. Normal protocol is defined as four ready pulses each with its accompanying parcel of data followed by a disconnect pulse. A resume pulse must be returned before a subsequent ready pulse is received. If more than four ready pulses are received before a disconnect pulse, then a channel protocol error has occurred. If a channel protocol error is detected, then all command data is assumed to have been corrupted and is ignored. The stat of the heartbeat semaphores is not changed and information in parcel 0 of the channel status word 36 is invalid. Bit P4 in the channel status word 36 is set. However, no channel status word 36 is transmitted until a disconnect pulse is received. This ensures that any extra command parcels are flushed.

Additionally, the appropriate parity bits p3-p0 and P3-P0 are set in the event of a data error. Four parity bits accompany each parcel of the channel command word 32. Bits P3-P0 perform a parity check using true logic levels, while bits p3-p0 perform a parity check using false logic levels. Thus, independent redundant checks are performed. Additionally, the P5 bit (command error) is set if a data bit rather than a parity bit caused the parity error.

Bit TM is set to 1 when the output port 26 has been placed in test mode.

The semaphore box 16 was designed for fault tolerance. For example, the semaphore box 16 may remain online and operational during repairs without impacting the integrity of the semaphores. Further, even if two of the three semaphore groups 18, 20, and 22 fail, the semaphore box 16 continues operating using the remaining group. As mentioned above, if no group compares correctly with any other, group 20 is used.

All logic modules are supplied with a common clock from a master clock module and operate synchronously with one another. The master clock module is duplicated on a second module for backup purposes, but only one clock module may be powered on at any given time. Each logic module is provided with two clock inputs, one for each clock module. Selection of the active master clock module is by means of a manual switch which controls power to the clock module and provides the logical clock enable signals. During the process of switching from one clock module to another, however, the system clock is not valid and the system requires re-initialization.

The semaphore box 16 resides in a standalone cabinet with its own cooling and internal power. Cooling is accomplished with forced room air. Multiple fans provide redundancy so that if a single fan malfunctions the equipment still remains operational.

Power is provided by four set s of identical power supplies to enhance fault tolerance. Each set of supplies is sized so that it is able to supply all the necessary power needs independently of the other. Power load shifting from the loss of one supply is automatic.

Each input port 24, output port 26, semaphore group 18, 20, and 22, and the master clock are tied to a common power bus, so that they may be individually disconnected from the power bus. Thus, no more than one module is affected by the loss of a single power supply breaker. Further, if power is removed from a single module, all other modules remain operational. Except for the master clock module, the process of applying or removing power to a single module does not affect the operation of the other modules. In addition, if a single module is removed from the system, all other modules remain operational. Thus, the process of inserting or removing modules does not affect the operation of the other modules.

In summary, a fault tolerant network has been described which includes a semaphore box 16 for controlling access to shared peripherals 12. The semaphore box 16 is comprised of two major sections: an I/O section and a semaphore section. The semaphore section contains reservation semaphores and heartbeat semaphores. The reservation semaphores are used to reserve a particular peripheral 12 for a particular computer 10; the heartbeat semaphores prevent reservation semaphores from being set indefinitely in the event communication with a particular computer 10 is lost.

The heartbeat semaphores are arranged in an array, wherein each time a command is received, a row of heartbeat semaphores is set. Further, when the command completes, a column of heartbeat semaphores is returned to the requesting computer 10 and then cleared. Thus, if the heartbeat semaphore in a particular position of the column is set, then the associated computer 10 has accessed the semaphore box 16 sometime between the current access of the requesting computer 10 and its prior access. Conversely, if the particular heartbeat semaphore is cleared, then the associated computer 10 has not accessed the semaphore box 16 during the period. The requesting computer 10 should conclude that the associated computer 10 has lost communication with the semaphore box 16 if the particular heartbeat semaphore for the associated computer 10 remains cleared for some number of consecutive accesses or some predetermined interval.

The foregoing description of the preferred embodiment of the present invention has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations are possible in light of the above teaching. It is intended that the scope of the invention be limited not by this detailed description, but rather by the claims appended hereto.