Method and apparatus for operating a server system including determining the power supplied by one of a plurality of power supplies by measuring voltage on a load share signal line

[0001] Server systems may be mounted in racks or enclosures, each rack holding a plurality of individual servers or blades. A server may be relatively powerful computer, which may comprise multiple microprocessors, that may be adapted for coupling together multiple personal computers and provide remote processing functionality. Servers may be used for mission-critical services such as on-line banking, on-line shopping and the like, and each rack of servers may have multiple power supplies, these power supplies may create a fully redundant power supply system. A fully redundant power supply system may have two or more power supplies, any one of which may be capable of supplying power for the entire rack of servers. Thus, if one power supply fails, the remaining power supply or power supplies may have the capability of supplying the necessary power while the failed power supply is replaced.

[0002] To balance load, each power supply in a redundant configuration may have a load share signal that may couple to a load sharing line coupled between power supplies. Each power supply may be designed and configured to drive the line to a voltage proportional to its output current (power). Each power supply may monitor the load sharing line and attempt to raise or lower its output current to match the voltage on the load share line. In this way, the load may be shared between the two or more power supplies.

[0003] However, there may be a need to verify that sufficient power is available, or redundancy of supplied power may be maintained, prior to allowing an additional device, such as a server, installed in the rack to power-on.

[0012] Certain terms are used throughout the following description and claims to refer to particular system components. As one skilled in the art will appreciate, computer companies may refer to a component by different names. This document does not intend to distinguish between components that differ in name but not function.

[0013] In the following discussion and in the claims, the terms “including” and “comprising” are used in an open-ended fashion, and thus should be interpreted to mean “including, but not limited to . . . ”. Also, the term “couple” or “couples” is intended to mean either an indirect or direct electrical connection. Thus, if a first device couples to a second device, that connection may be through a direct electrical connection, or through an indirect electrical connection via other devices and connections.

[0014] The term “rack of servers” may be equivalent to “rack of enclosures with servers.”

[0015] FIG. 1 illustrates a schematic view of a single rack or enclosure 10 in accordance with embodiments of the present invention. The enclosure 10 may comprise a plurality of servers 12. The servers may draw power from power supplies 14 and 16. The power supplies 14, 16 may have sufficient power output capability that the enclosure 10 may operate in a fully redundant mode. If either power supply 14, 16 fails, the remaining operational power supply may be capable of supplying power to the enclosure 10. The power supplies 14, 16 may couple their power to the servers 12 by way of a mid-plane board 18. The enclosure 10 may further comprise an enclosure manager 20 which may couple to the power supplies 14, 16 as well as the servers 12 by way of the mid-plane board 18. The enclosure manager 20 may perform various functions, such as controlling fans in the enclosure (not shown) and facilitating external communications. Also, the enclosure manager 20 may be responsible for determining an amount of power delivered by the power supplies 14, 16, and thereafter determining whether an additional device, such as a server, installed into the enclosure 10 may be powered-on without adversely overloading the power supplies or affecting the fully redundant power supply operations.

[0016] FIG. 2 illustrates an interconnection of the power supplies 14, 16, enclosure manager 20, and mid-plane board 18 with regard to power distribution. The mid-plane board may comprise multiple power rails, such as: a 12 volt rail 22, a 5 volt rail 24, a 3 volt rail 26, and a 5 volt auxiliary rail 28. As the drawing of FIG. 2 illustrates, each connector, for example 30A, 30B, to which a server or other device (not shown in FIG. 2) may be coupled, may itself be coupled to each of the power rails 22, 24, 26 and 28. In operation, servers coupled to the connectors 30 may draw power as necessary from the appropriate rail.

[0017] Power supplies 14, 16 likewise may couple to the power rails and supply power to those rails. More particularly, power supply 14 may have a 3 volt power output signal, a 5 volt power output signal, a 12 volt power output signal, and a 5 volt auxiliary output power signal. Correspondingly, power supply 16 may have the same 3 volt, 5 volt, 12 volt and 5 volt auxiliary power output signals. With regard to the 3 volt, 5 volt and 12 volt power output signals, each power supply 14, 16 may couple to each power supply rail 22, 24, 26 and 28 to facilitate the ability of each power supply to provide power during normal operations, and which may comprise the entire required power given a power supply failure. The power supplies 14, 16 may also be coupled by way of the load sharing signals.

[0018] Still referring to FIG. 2, the mid-plane board 18 may comprise three traces 32, 34, 36 that may couple the 3 volt load sharing signal, 5 volt load sharing signal and 12 volt load sharing signal respectively of each of the power supplies 14, 16. By simultaneously monitoring and driving the load sharing signals, the power supplies 14, 16 may balance the amount of load provided by each power supply. Each power supply may be designed to drive a particular voltage on to each load sharing line proportional to the amount of output current being provided by the power supply on that output power rail. Since the voltage on each power rail may be constant, an indication of output current may be directly related to the power supplied to the rail. As between two or more power supplies, however, the power supply providing the most output current may drive the load sharing line to its higher voltage. In other words, as between two power supplies not providing the same amount of power, the load sharing line may have a voltage representing the larger of the supplied powers. Each power supply may monitor the load sharing signal, and attempt to adjust its output power to evenly distribute power delivery between the power supplies.

[0019] Embodiments of the present invention may utilize the load sharing signals in a steady-state condition to determine the power delivered. Using the power delivered, the embodiments of the present invention may selectively allow or disallow an additional device installed in the enclosure 10 to power-on. If power requirements of the additional device force the enclosure 10 to operate in other than a fully redundant condition with respect to available power, the enclosure manager 20 may not allow the additional device to power-on. That is, if the power delivered plus the power required for the additional device exceed a rated power capacity for either of the power supplies (in a two power supply embodiment), the additional device is not allowed to power-on. In this event, the enclosure manager may alarm, or otherwise give notice, of the reason for the device's failure to power-on. For the enclosure manager 20 to make this determination, it may be necessary for the enclosure manager 20 to determine the total power provided. This determination may be made by monitoring and analysis of the load share lines 32, 34 and 36, and the power provided across the 5 volt auxiliary line.

[0020] Still referring to FIG. 2, the enclosure manager 20 may determine the total power drawn by the enclosure 10. With regard to the 3 volt, 5 volt and 12 volt power output signals of the power supplies 14, 16, the total power may be determined based on the voltage levels present on the load share lines 32, 34 and 36 respectively. Further, each server installed in the enclosure 10 may draw power from the 5 volt auxiliary rail 28, and thus the enclosure manager 20 may also monitor the total power draw on this rail as part of the determination if the additional server may be allowed to power on.

[0021] FIG. 3 illustrates a block diagram of the internal components of the enclosure manager 20 of embodiments of the present invention. The enclosure manager 20 may comprise a central processing unit (CPU) 38. While any microprocessor or microcontroller may be used in the capacity of the CPU 38 of the enclosure manager 20, CPU 38 may be an IBM Power PC405GP. The CPU 38 may couple to a non-volatile memory 40. While many types of nonvolatile memory may be utilized without departing from the scope and spirit of the disclosure, the non-volatile memory 40 may comprise both flash read-only memory (FLASHROM) and non-volatile random access memory (NVRAM).

[0022] The CPU 38 of the enclosure manager 20 may also couple to a main memory array 42. The main memory array 42 may be synchronous dynamic random access memory (SDRAM), with the SDRAM 42 possibly being the working memory for the CPU 38. By contrast, the non-volatile memory 40 or other memory may store boot-strap programs for the CPU 38, as well as the software that may be necessary to implement the functions of the enclosure manager 20. As mentioned briefly above, the enclosure manager 20 may also facilitate external communications by way of a communication port 44 and universal asynchronous receiver transmitter (UART) 46 coupled to the CPU 38. The CPU 38 may also comprise communication buses such as a peripheral components interconnect (PCI) bus 48, and an I²C bus 50. The I²C bus 50, though shown with only a single line in FIG. 3 as well as FIG. 7, may be a dual line, multi-drop serial bus developed by Phillips Semiconductor that may comprise a clock line and one data line. Devices connected to the I²C bus may act as either primary or secondary devices, and each device may be software addressable by a unique address. Primary devices may operate as transmitters, receivers, or a combination transmitter/receiver to initiate eight-bit data transfers between the devices on the bus. The I²C bus may utilize collision detection and arbitration to prevent data corruption if two or more primaries simultaneously transfer data. Details regarding the I²C bus may be found in the “The I²C Bus Specification,” Version 2.1 (January 2000), authored by Phillips Semiconductor.

[0023] To calculate a total instantaneous power being provided by the power supplies 14, 16, utilizing the load share signals 32, 34 and 36, the enclosure manager 20 may read the voltage levels on each of the load share signal lines. Reading the load share signal lines of the embodiments may involve the use of an analog to digital converter 52 that may be coupled on its input side to a multiplexer 54, and that may be coupled on its output side to the I²C bus 50. A 12 bit analog to digital conversion may provide sufficient accuracy, thus, the analog to digital converter 52 may be a Texas Instruments Part No. ADS7823. As implied by the discussion of the exemplary drawing of FIG. 2, however, there may be four signals which need to be converted by the analog to digital converter 52, and thus multiplexer 54 may be responsible for selectively coupling each of these signals to the analog to digital converter. Multiplexer 54 may be a Fairchild 74VHC4052. Using the multiplexer 54 and analog to digital converter 52, enclosure manager 20 may sample the load share signals, as well as the 5 volt auxiliary power (discussed more fully below) to determine a total power delivered by the power supplies.

[0024] The load share signals coupled between the power supplies may be primarily designed for balancing supplied power. FIG. 4 illustrates a graph of voltage on a load share line (on the ordinate) against output current of a power supply (on the abscissa). The relationship between the load share voltage and the output current may be a straight line, such as dashed line 56. However, rarely do load share voltage values exhibit the straight line relationship. FIG. 4 further illustrates three exemplary curves 58 that may more accurately describe the relationship between a voltage produced on a load share line and output current for a particular voltage rail of a power supply. The family of curves 58A–C may exemplify that for three different power supplies of the same type, for example, three different relationships may exist. In order to accurately determine total power delivery by monitoring the load share voltage signals, the enclosure manager 20 may have the ability to compensate for the non-straight-line relationships. The relationship may comprise an offset 60, and a break point 62. Between the offset at the zero percent load share voltage and the break point 62, the relationship may generally be a straight line having no or a slight upward slope. Between the break point 62 and the 100% load share voltage, the relationship may be generally parabolic.

[0025] In at least some of the embodiments of the invention, the relationship for each load share signal voltage to output current (for each output power rail) may be modeled so that the enclosure manager may determine the relationship between the load share voltage signal and the current output. FIG. 5 illustrates one possible modeling strategy. The relationship between the load share voltage and the output current for each power rail of each power supply may be modeled using four points along the line, and therefore three segments. The modeling system exemplified in FIG. 5 may be equivalently represented in the table of FIG. 6. In the exemplary graph of FIG. 5 and the exemplary table of FIG. 6, the offset that the load share voltage maintains when the power output current reaches zero percent may be represented by voltage V₁ 64. The third point in the table of FIG. 6 may be the 50% output current (and therefore output power) mark, having a load share voltage V₃ 68. With respect to the 100% output current point, the exemplary table of FIG. 6 may likewise contain a load share voltage value V₄. In at least some of the embodiments of the present invention, the zero percent, 50% and 100% table (or graph) entries may be fixed. The location on the output current axis of the second table entry, however, may be variable from table to table. This entry may be defined by the break point. The break point may be the point in the load share voltage to output current relationship where the load share voltage changes appreciably with changes in output current. For example point 62 in the family of curves 58 of FIG. 4 and a corresponding load share voltage V₂ 66 may exemplify a break point. Thus, as any particular load sharing voltage to output current relationship may exhibit a different breakpoint, this may be accounted for in the variable table entry.

[0026] Each power supply inserted into the enclosure 10 may have three main power output signals, and correspondingly three load share signals. Each of these load share signals may exhibit different characteristics, such as those shown for lines 58 of FIG. 4. Thus, in at least some embodiments, the enclosure manager 20 may need access to a data table, such as that exemplified in FIG. 6, that may model the relationship between the output current and the load share voltage for each of the power output signals for the particular power supply. The tables may be stored in serial electrically erasable programmable read only memory (EEPROM), for example serial EEPROM 70 of power supply 14 and serial EEPROM 72 of power supply 16 (FIG. 7). The enclosure manager 20 may read the various tables for the power supplies 14, 16 from their EEPROM 70, 72 respectively just after the power supplies 14, 16 and enclosure manager 20 are powered on; however, the enclosure manager 20 may also read this information at any time. The enclosure manager 20 may read three such tables from the EEPROM 70 of the power supply 14 over the I²C bus 74 (one for each power output rail except 5 volt auxiliary). Likewise, the enclosure manager 20 may read three tables from the EEPROM 72 of the power supply 16 across the I²C bus. Thereafter, the tables may be available to the enclosure manager for calculating total power provided by the two power supply devices, for example when an additional server is installed in the enclosure.

[0027] The servers 12, or other installed devices, of the embodiments of the present invention may not necessarily power-on automatically upon being inserted into the enclosure 10. Once the enclosure manager 20 becomes aware that an additional server has been installed, possibly by establishing communication across the I²C bus 74, the enclosure manager may read information about the installed or additional server or device. Servers of the embodiments of this invention may comprise an I²C general purpose I/O (GPIO) expander, such as GPIO expander 76. Devices such as expander 76 may allow transfer of bits of information both to and from the enclosure manager 20. After installation of an additional server, such as server 12 of FIG. 7, the enclosure manager 20 may read information regarding the type of server installed. Based on the type of server installed, the enclosure manager may determine the amount of power the server requires during normal operation. Prior to power-on of the additional server, the enclosure manager 20 may read the various load share signal lines, and may calculate the amount of power currently being provided to determine whether the additional server will adversely affect the fully redundant power supply operation.

[0028] Consider for purposes of explanation enclosure manager 20 reading and calculating the amount of power delivered from the power supplies 14, 16 on the 12 volt power rail 22. The enclosure manager 20 may read a voltage on the 12 volt load share line 36 by appropriately adjusting the multiplexer 54 to couple the load share voltage to the analog to digital converter 52. The selected control of the multiplexer 54 may be accomplished by an I²C GPIO expander 53 having its serial side coupled to the I²C bus 50, and in at least some embodiments, two of its digital outputs couple to the select lines of the multiplexer 54. Thus, the CPU 38 may communicate with the device 53 and select any of the input signals. Once the appropriate multiplexer 54 input is selected, the analog to digital converter may convert the analog signal to a digital value, which may be read by the CPU 38 over the I²C bus 50. Although possible, it is unlikely that the load share voltage value read by the CPU falls precisely on a value contained in the appropriate table. In this case, the CPU 38 may interpolate between values in an appropriate table.

[0029] Referring to FIG. 5, consider an exemplary load share voltage read by the CPU 38 of V_x. The voltage V_x falls between the 50% output current entry and the 100% output current entry. The enclosure manager 20 may therefore utilize the data values of the 50% and 100% output current entries to determine the parameters which may be used to interpolate the output current (and therefore the output power) being generated for the particular power supply. After performing the lookup (and possible interpolation) for the first power supply, the enclosure manager may perform the same task for the second power supply. Each of these operations yields an output current for each power supply, and the enclosure manager may combine the results to determine the total output power for that particular power rail. The enclosure manager 20 may perform this same task for the additional power rails by reading the appropriate load share voltage signals.

[0030] Each server in the enclosure 10 may draw power from the 5 volt auxiliary power rail 28, and this too may be considered in determining total output power. Unlike the 12 volt, 5 volt and 3 volt power rails (22, 24 and 26 respectively), power supplies 14, 16 may not have a load share signal for the 5 volt auxiliary power. Thus, it may not be possible to determine the power by reading the load share signal.

[0031] Referring again to the exemplary FIG. 2, there is shown a sense resistor 78 coupled between the 5 volt auxiliary rail 58 and each of the power supplies 14, 16. The sense resistor 78 may be a very small, high-precision resistor used to measure the total current supplied to the 5 volt auxiliary rail 28. The enclosure manager 20 may read both the positive and negative sides of the resistor 78 and generate a voltage proportional to the current flow. The mid-plane board 18 may comprise an op-amp 80 having one input coupled on the negative side of the resistor 78, and a second input coupled to the positive side of resistor 78, which may also be the 5 volt auxiliary rail 28. The op-amp 80 may convert the small differential voltage create by the sense resistor 78 into a voltage that couples to the multiplexer 54 and correspondingly analog to digital converter 52 (FIG. 3). Thus, the enclosure manager 20 may, in calculating the total power delivered in the enclosure 10, determine the amount of power provided to the five volt auxiliary power rail 28.

[0032] By reading the load share voltage for each of the 12 volt, 5 volt and 3 volt power rails, and determining the amount of power delivered to these rails by each of the power supplies, the enclosure manager may determine power delivered. Further, by sensing the voltage across the sense resistor 78 as proportional to the total current delivered to the 5 volt auxiliary rail 28, the enclosure manager may calculate the power delivered from this power rail as well. Combining the various results, the enclosure manager 20 may determine the total power delivered from each of the power supplies in the system. By adding the power that an additional server would utilize (if allowed to power-on), the enclosure manager may determine whether the server would draw too much power to allow the enclosure 10 to operate in a fully redundant mode. If the additional server extends the limits of the power supplies beyond fully redundant operation, the enclosure manager may not allow the server to power-on.

[0033] It is noted that in some of the various embodiments described, the load share voltage values may exceed the maximum values which may be applied to the multiplexer 54. In circumstances such as these, it may be possible to lower the voltages by use of voltage divider networks. The voltage divider networks may reside on the enclosure manager circuit board 20, or may likewise reside on the mid-plane board 18 or some other location. To the extent that any of these voltage divider networks induce changes in the sensed voltage from an ideal relationship, such as that illustrated by dashed line 56 in FIG. 4, the enclosure manager 20 may have additional tables, that may operate on the same principles as the table of FIG. 6 and the related exemplary graph of FIG. 5, to aid in removing these deleterious effects.

[0034] The above discussion is meant to be illustrative of the principles and various embodiments of the present invention. Numerous variations and modifications will become apparent to those skilled in the art once the above disclosure is fully appreciated. For example, while the bulk of the specification may have been directed to systems with only two power supplies, the principles discussed may likewise be implemented in systems having more than two power supplies. In particular, the system and methods may be operable in an N+1 redundant power system, capable of continuing operation given the failure of any one power supply (where N is the total number of power supplies). In this case, an additional device may be allowed to power-on if the total proposed power is less than or equal to (N−1)/N of the rated power output. The system and methods may likewise be operational in an N+N redundant power system, having multiple fully redundant power grids (as opposed to just power supplies). Further still, the system and methods may be operable in an N+M redundant power system, capable of continuing operation given the failure of M power supplies. In such a circumstance, determining whether an additional device should be powered may be based on a determination of whether the total proposed power is less than or equal to (N−M)/N of the rated power output of the plurality of power supplies individually, where 1<=M<N. Further, while the specification may focus on making a power determination upon insertion of an additional server, the determination regarding available power for an operating condition may be made with respect to any device, such as a server, storage device, packet switching device, and the like. Finally, determining power output using load share signals, and ascertaining whether an additional device should be allowed to power-on as described in the specification may likewise be utilized in a system having only one power supply (or only one operational power supply). It is intended that the following claims be interpreted to embrace all such variations and modifications.