Method and apparatus for coordinating uninterruptible power supply modules to provide scalable, redundant power

[0002] This invention relates generally to the control of uninterruptible power supplies, and more particularly to the control and coordination of multiple power modules in a modular uninterruptible power supply.

[0003] As more and more segments of the business environment enter the information age, more and more computers and computing power are required. As business move from the old to the new economy their reliance on the processing, transference, and storage of digital information is becoming a more and more critical aspect of their overall business strategy. While in the past, computer crashes were seen as a mere nuisance, the loss of computing power and business data may well devastate a business's ability to survive in today's new economy. As such, the need for reliable, uninterruptible electric power to maintain the operational status of the computing equipment and the integrity of the digital data continues to rise.

[0004] To meet these requirements, uninterruptible power supplies (UPS) have been developed. These UPSs utilize a bank of electric storage batteries and solid state conversion equipment in association with the utility line voltage to provide continuous electric power to a business's computer system in the event of a loss or deviation of power quality from the utility. The number of batteries contained within an UPS is dependent upon the business's length of time that it needs to operate in the event of a utility power system failure. Likewise, the number of power modules included in a modular UPS is dependent on the overall total system load required to be supplied thereby. In the past, uninterruptible power supplies were only available in discreet sizes. However, with the recognition of the various requirements from the multitude of businesses that require UPSs, the development of modular uninterruptible power supplies has provided a flexibility heretofore unknown. Now, a business's UPS power output may grow as its business needs by simply purchasing additional power modules and adding them to their existing modular UPS.

[0005] Unfortunately, the advent of the modular UPS has also introduced new and different challenges to the UPS designer. Specifically, the modular UPS must function as a single system regardless of the number of power modules that are added to or taken away from the UPS chassis. Output power quality, system display, and transitions between operating modes must all be coordinated between the multiple power modules that make up the business's UPS system.

[0006] One such modular UPS that provides coordinated operation is described in U.S. Pat. No. 5,982,652 to Simonelli, et al. for a METHOD AND APPARATUS FOR PROVIDING UNINTERRUPTIBLE POWER USING A POWER CONTROLLER AND A REDUNDANT POWER CONTROLLER. As described in this patent, the coordination and control of the multiple power modules contained in the modular UPS is accomplished through the provision of a main intelligence module (MIM) and a redundant intelligence module (RIM). The MIM and RIM have connections to each other and to the other modules to allow monitoring of operational conditions of the UPS system, including characteristics of the input and output power, and to provide phase synchronization, frequency regulation, and voltage regulation of the output power. In this system, the MIM functions as the primary controller within the UPS system and the RIM is a redundant controller that can assume control of the UPS system upon failure of the MIM or removal of the MIM from the UPS system. All of the power modules are substantially identical and perform the functions of an uninterruptible power supply under the control of the MIM or the RIM.

[0007] In the system of the Simonelli, et al. '652 reference, the main intelligence module (MIM) is the primary computer/controller in the UPS system. It acts as a central point in the system for collecting and communicating information about aspects of the power modules, battery modules, and the redundant intelligence module (RIM). The MIM contains a primary and a backup microprocessor for controlling and coordinating this operation. The redundant intelligence module (RIM) on the other hand is merely a backup version of the MIM, and provides redundancy in the event of a MIM failure, or while a MIM is being replaced. In this backup role, the RIM includes the same major systems as the MIM except for the main processor system, although some of the systems in the RIM are described as having less functionality than the corresponding systems in the MIM.

[0008] Unfortunately, such a control strategy greatly increase the overall cost of the modular UPS system by requiring that two separate additional modules be purchased and maintained in order to allow functionality of the UPS at all. In this way, in addition to the value providing power modules, battery modules, and battery charging modules, a user is forced to also purchase a separate main intelligence modules to control the other components in the system. Further, the user is forced to purchase a redundant intelligence module (RIM) whose only main purpose is to provide backup operation should the MIM fail. In other words, this module is totally superfluous to the actual operation and functionality of the uninterruptible power supply unless a failure of the MIM occurs.

[0009] In an economy that demands maximum efficiency from business resources, such a system that requires separate primary and redundant control modules simply cannot be justified. The unfeasibility of such a system may be clearly demonstrated in a situation where a business enterprise only needs a single power module to supply its uninterruptible power requirements. In such a situation, the Simonelli, et al., '652 system would require the business enterprise to purchase two control units, the MIM and the RIM, to control the single power module in its UPS. In such a situation it becomes clear that the provision of two control units to control a single power module is a luxury not many businesses would desire to acquire.

[0010] Therefore, there exists a need in the art for a modular uninterruptible power supply that has coordinated control of its constituent components and an acceptable level of redundancy without requiring the purchase of additional multiple control unit.

[0011] An embodiment of the invention provides a method and apparatus for coordinating the control of uninterruptible power supply modules to provide scalable, redundant power. Each of the independent UPS power modules are designed to contain identical control logic, both hardware and software, so that only a single power module need be purchased to scale the UPS system based on the business enterprises power requirements. Each of these identical UPS power modules are connected to common communications, power, and logic buses within the UPS system. Under such a system, the multiple power modules act as a single UPS with all of the controls and actions taken by the individual power modules being coordinated through a single power module acting as the system's virtual master. Preferably, the master role is virtual because any of the power modules in the system contain all of the necessary control logic and software to assume the master role.

[0012] For the system of the invention to be fault tolerant it must be capable of withstanding a failure or removal of the power module that has assumed the master role. Preferably, this is accomplished by requiring another module to be prepared to provide backup to the master. In this role, designated vice-master, the power module provides the controlling signals to itself and the other power modules if and when the designated master fails. Under this system, the vice-master then promotes itself to master and assigns a new vice-master from one of the operational power modules in the system. Under the system of the invention, no additional main intelligence modules or redundant intelligence modules are required to perform this functionality. Coordination and redundancy are provided within the control of the individual power modules themselves.

[0013] A power up arbitration scheme is used to assign one of the power modules to the master's role. Preferably, the master then assigns a vice-master and assigns all other modules to act as peers. Peer modules simply follow the control commands of the master or vice-master depending on the operational status of these two units. Modules in the peer role are also drawn upon when a vice-master needs to be appointed to the master role upon detection of a problem with that unit. This hierarchy of roles provides the means to coordinate identical UPS modules into a larger UPS system without the need for additional special main and backup controllers. In the event that only a single power module is connected in the UPS system, it is quite capable of controlling itself in the role of the master, without a requirement for a vice-master or any peer modules to be connected. Under the system of the invention, there is no need for two separate control modules to control the one power module connected in the system.

[0014] Power module interconnection and coordination of control is accomplished using a common set of signals, collectively known as the redundancy management bus (RMB). The RMB of the invention includes a high-speed communication bus, a set of bi-directional logic lines, and one analog reference for each power phase of the UPS system. Each power module may drive any of the signals. It is the role of the power module that determines which signals are driven and which signals are merely monitored. The high-speed communication bus may take any one of a number of forms, including CAN, Ethernet, Serial 485, etc. In a preferred embodiment, a high-speed communication bus takes the form of a CAN bus due to CAN's inherit fault tolerance and noise immunity.

[0015] In accordance with the invention, the master is responsible for driving most of the signals on the RMB. The other power modules take direction from the master by acting on these signals. When the master power module fails or is removed from the UPS system, a master ok signal typically supplied by the master power module is no longer sent. This provides an indication that the vice-master power module should take over driving the analog output voltage target signal used by the power modules to create their output voltage waveforms. Since the role of the vice-master has previously been assigned, the remaining peers do not see a discontinuity in the output voltage target signal and therefore their inverter output waveforms are clean during this transition.

[0016] In a preferred embodiment of the invention, the microprocessor inside each of the power modules is connected to the RMB through an interface computer programmable logic device (CPLD). Preferably two digital signals from the processor are used to indicate the power module's role to the CPLD. The role logic is used to enable bus drivers for each role's group of output signals. The power module's particular role in a given UPS system determines the bus input/output direction for each communicated signal.

[0017] In a preferred embodiment of the invention, the microprocessor inside each of the power modules is connected to the RMB through an interface computer programmable logic device (CPLD). Preferably two digital signals from the processor are used to indicate the power module's role to the CPLD. The role logic is used to enable bus drivers for each role's group of output signals. The power module's particular role in a given UPS system determines the bus input/output direction for each communicated signal.

[0018] The accompanying drawings incorporated in and forming a part of the specification illustrate several aspects of the present invention, and together with the description serve to explain the principles of the invention. In the drawings:

[0019] FIG. 1 is a system level single line diagram of a modular uninterruptible power supply system constructed in accordance with the teachings of the invention;

[0020] FIG. 2 is a detailed single line block diagram of the control structure of a power module constructed in accordance with the teachings of the invention;

[0021] FIG. 3 is a control flow diagram illustrating the operating role determination process of the invention;

[0022] FIG. 4 is a control flow diagram illustrating master control of a configuration state change of the individual power modules in the UPS of FIG. 1;

[0023] FIG. 5 is a control flow diagram illustrating a control methodology to allow power modules in the UPS system of FIG. 1 to vote out a master controller;

[0024] FIG. 6 is a control flow diagram illustrating the corresponding functional control logic within the virtual master in response to the vote out control flow of FIG. 5; and

[0025] FIG. 7 is a control flow diagram illustrating the assumption of the virtual master role by a virtual vice master in accordance with the teachings of the invention.

[0026] While the invention will be described in connection with certain preferred embodiments, there is no intent to limit it to those embodiments. On the contrary, the intent is to cover all alternatives, modifications and equivalents as included within the spirit and scope of the invention as defined by the appended claims.

[0027] A preferred embodiment of the modular uninterruptible power supply (UPS) 10 constructed in accordance with the teachings of the invention is illustrated in single line block diagrammatic form in FIG. 1. As with typical modular UPS's, the modular UPS 10 of the invention includes multiple slots 22a-f into which various modular components may be placed. As illustrated in FIG. 1, modules 22a-chave installed therein power modules 12a-cconstructed in accordance with the teachings of the invention. Slots 22d-fare illustrated in FIG. 1 as being empty, thus providing the ability to expand or vary the power generation capability of the modular UPS 10.

[0028] The AC input power from the utility is connected to the UPS 10 through lines 16, and the output of the UPS 10 is supplied to the utilization equipment through output lines 18. Preferably, each of the power modules 12a-care identically constructed, and contain a power inverter 20a-cand a controller 24a-c. As will be recognized by those skilled in the art, the particular topology utilized for the construction of the inverter 20 may vary substantially based on system and performance level requirements. Preferably, the inverter 20 utilizes a double conversion H-bridge with its own internal bypass. The control for this inverter 20 is provided by the controller 24 through the control lines 32a-c. This control accomplishes both the proper operation of the power switches of the inverter to construct the output AC wave form, as well as the selection and designation of the proper operating mode for the inverter. These operating modes may include a bypass mode of operation for high efficiency or fault tolerance, and two modes of inverter operation. The first allowing the AC voltage from the utility to supply the inverter, while the second utilizing installed battery banks (not shown) to power the inverter.

[0029] As discussed above, the need for coordinated control between the individual power modules is paramount so that the UPS 10 may operate as an integrated unit. This unified and integrated control is accomplished by the separate control modules 24a-cin each of the individual, identical power modules 12a-c. The controllers 24a-ceach receive operating parameters of its own power module, such as input power parameters via line 26cand output power parameters via line 28c. Additionally, the control modules communicate with one another via a high-speed communications bus 34. This communications bus may be constructed in accordance with any of a number of high-speed communication standards, including CAN, Ethernet, serial 485, etc. Preferably, this bus is implemented as a CAN bus in the UPS system 10 due to CAN's inherent fault tolerance and noise immunity.

[0030] In addition to the high-speed communication, and for reasons which will be discussed more fully below, the controllers 24a-calso communicate via a set of bi-directional logic lines 36. Further, communication between each of the controllers 24a-cto allow proper inverter coordination is accomplished via an analog reference 38 for each power phase of the UPS system 10. Collectively, this common set of signals is known as the redundancy management bus (RMB). Advantageously, each of the power modules 12a-care capable of driving any of the signals contained on the RMB based upon an established role assumed by the power module 12a-cas will be discussed below. This role of the power module determines which signals are driven and which signals are monitored.

[0031] FIG. 2 illustrates an individual power module 12 in greater detail. As may be seen from this detailed illustration, the controller 24 includes control logic 40 which is responsible for coordinating the integrated control of this particular power module 12 with the others included in the UPS system. The control logic 40 communicates via the bi-directional communication bus 34 to send and receive status information and nontime critical control signals. The control logic 40 may also receive a position signal 42 that designates in which slot of the UPS chassis the power module 12 is installed. This signal may also come from a user interface display controller that drives a user interface for the UPS, if one is included. The control logic 40 also communicates through a computer programmable logic device (CPLD) 44 over the set of bi-directional logic lines 36. The CPLD is utilized to properly configure the control signals via lines 36 in input or output mode for communication to the control logic 40 via lines 46 based on the role that the control logic 40 has assumed in the overall system. This role is communicated to the CPLD 44 via the role selection lines 48.

[0032] As will be discussed in greater detail below, the control logic 40 may determine that its power module 12 should be in the role of master, vice master, or peer. While various control signals may be utilized, a preferred embodiment of the invention contains control signals as indicated in Table 1. This table also indicates the direction of signal transfer depending on the role assumed by the control logic 40. The role selection signals 48 for this embodiment are indicated in Table 2. Table 3 defines a preferred pin out arrangement for the CPLD 44 to allow proper coordination of the input and output information conveyance based upon the role assumed by the particular power module 12.

[0033] As may further be seen in FIG. 2, the controller 24 also includes an inverter control section 50 which operates to control the inverter based upon inputs 52 from the control logic 40, input and output power monitoring signals 26, 28, as well as a voltage reference signal 54 from the analog reference line 38. This analog voltage reference line 38 may be driven by this particular power module 12 depending on the role assumed by the control logic 40. That is, if the control logic 40 has assumed the role of master, the CPLD 44 will enable 56 an analog voltage reference module 58 to generate and transmit the voltage reference signal on the analog voltage reference line 38. In the case where this power module 12 has assumed the role of master, its own analog voltage reference circuitry 58 will be driving the inverter control 50. For modules that have assumed roles other than the master, their analog voltage reference circuitry 58 will not be enabled via line 56 from the CPLD 44, and therefore the voltage reference signal 54 will be that determined by another power module.

[0034] The determination of the proper role to be assumed by the control 40 of each individual power module 12 is determined initially at power up, as illustrated in flow diagrammatic form in FIG. 3. Once the control logic powers up 60 it checks the master okay signal to determine if any other units have asserted the master role 62. This is allowed at power up because the CPLD 44 (see FIG. 2) initially configures the control IO lines 46 in a receive mode until a role is established via the role selection lines 48. If another unit has already asserted the role of master, as may be the case if a power module is inserted to the UPS during operation, the control logic 40 will request a role 64 from the master. This role request signal is preferably transmitted via the high-speed bi-directional communications bus 34. Once the master has determined a role for this particular power module, it will receive this information 66 and thereafter properly configure its communications for the assigned role 68 before terminating this process 70. As discussed above, the configuration of the communications based on the role assignment allows the bi-directional signal lines to be properly interpreted as inputs or outputs in accordance with the direction information contained above in Table 1.

[0035] If, however, upon power up 60 it appears that no other power module has asserted the master role 62, this power module will begin to perform its master arbitration 72. This master arbitration is designed to bias a race between identical power modules so that only one winner attempts to assert the master role. Known methods for this arbitration include the generation of a random number within the power module. This random number is then used as a multiplier for a fixed time delay which must time out prior to the attempted assertion of the master role. In a preferred embodiment, this master arbitration utilizes position information communicated to the control logic 40 either from physically sensing the particular slot in which it is installed, or via a location signal transmitted from a common user interface and display controller if such is provided in the UPS 10.

[0036] Once the power module has completed its master arbitration 72, it checks to determine if any other control unit has already asserted the master role 74. If another unit has asserted the master role, this power module transmits a role request 64 to the master so that it may be assigned its proper role in the UPS configuration. Once it receives its role 66 it will then configure its communications for that role 68 before ending this process 70. If, however, no other power module has yet asserted the master role, the control logic 40 will assert that it is the master 76. Once this assertion has been made, the power module will then configure its communications for its role 68 before ending this process 70.

[0037] While in the role of master, that power module is responsible for controlling the state configuration of the entire UPS. When a state change is required, it is the responsibility of the master to coordinate this state change so that all power modules will perform the state change transitions at nearly the identical time. This coordination through the master unit is illustrated in the simplified flow diagram of FIG. 4. Once started 78 the master then determines that a configuration state change is needed 80. This determination may be based on a number of different factors, including user input, operational status of the power generation and distribution, fault detection, etc. Once the new configuration is determined by the master, it then transmits the new configuration information to the vice master and the peer power modules in the UPS system 82. This information is preferably transmitted via the high-speed communication bus 34 (see FIG. 1) to all of the other power modules. This information is used by the other power modules to prepare for the state change to minimize the system disturbance resulting therefrom.

[0038] Once the new configured information has been transmitted to all of the other power modules in the UPS system, the master then toggles 84 one of the control lines known as the synch line to initiate the state change. The sync line preferably initiates a high-level or non-maskable interrupt in controller 40 of each of the power modules in the UPS system, including itself. The high level interrupt is then processed and the units perform the state change 86 before ending the sequence 88. This control structure is utilized in a preferred embodiment to allow for nearly simultaneous transition between operating modes on each of the individual power modules in the system. Controlling these state changes requires tight timing on the execution skew between these distributed power modules, and the synch line and the high-speed communication bus are used in combination to accurately synchronize actions and events between the collection of the uninterruptible power supply modules.

[0039] While the high-speed communication bus provides an excellent means of triggering actions, microsecond synchronization of actions on multiple different processors is difficult because of the asynchronous nature of the communications. Communication latencies and interrupt delays limit the accuracy to only a few milliseconds. While such delays may be acceptable in systems utilizing ferroresonant transformer coupled outputs, such delays may be catastrophic in a transformerless system such as is used in a preferred embodiment of the invention. By utilizing the synch line and tying it to a high level interrupt for each of the controllers, the communication variability is removed. By processing the state change as a high level interrupt, the actions across the multiple controllers for each of the multiple power modules are all started at approximately the same time within the limits of the processor's high level interrupt handling. In a preferred embodiment of the invention, this is typically a few tenths of microseconds, and is in any event, much tighter than the communications bus would allow if it were used independently.

[0040] In addition to providing redundancy through the role assumption scheme discussed above, the system of the invention also includes a method for removing the current master in the event that other units sense a problem that the master cannot sense for itself. As indicated in the flow diagram of FIG. 5, once this process is started 90 and the unit senses any type of problem with the master 92 that qualifies as a removable problem, the unit transmit a vote out signal 94 to the master before ending 96. Qualifying problems are defined by the system requirements, and may include communication failures with the master, load sharing and balance problems due to a problem in the master's transmission of the V_errorsignal, etc.

[0041] Within the master, as illustrated in the flow diagram of FIG. 6, once this process is started 98, the power module checks to determine that it is the master 100. If the power module is not in the role of master, the process ends 102. However, if the power module is in the role of master, it first checks to determine if it has received a vote out signal from its vice master 104. If it has received the vice master vote out signal 104 it then checks to determine if it has received a vote out signal from at least one peer 106 in the UPS system. If the master has not received a vote out signal from two units, one being the vice master and one being at least one of the peers, the master does not relinquish its role as master. If, however, the master has received a vote out signal from both the vice master and one of the peers, it then drops its master role assertion 108 before ending this process 102.

[0042] Once the master has dropped its assertion of the master role, it is then the duty of the vice master to assume those responsibilities as illustrated in the flow diagram of FIG. 7. Upon initialization of this process 110 the power module checks to determine if its role has been assigned as the vice master 112. If not, this process ends 114 and the power module continues as a peer in the UPS system. However, for the power module that has assumed the role of vice master, it continuously checks to make sure that the master is still okay 116. As soon as it is determined that the master is no longer okay, the vice master then begins to transmit the V_errorsignal 118 so that there is no, or only a minimal disturbance in this signal to the other power inverters. The vice master then asserts the master role 120 and reconfigures its communications for the master role 122.

[0043] Once this has been accomplished, the new master then chooses one of the peer modules as the vice master and transmits its new assignment to that power module 124 before ending this process 114. The assignment of the vice master may be accomplished through any number of control strategies from the available peers in the UPS system. Preferably, the vice master simply picks one of the peers based on a location priority and transmits the new role assignment thereto. Alternatively, the new master could transmit a master arbitration signal to each of the peers in the UPS system and assign the vice master based upon the winner of that arbitration.

[0044] The foregoing description of various preferred embodiments of the invention has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Obvious modifications or variations are possible in light of the above teachings. The embodiments discussed were chosen and described to provide the best illustration of the principles of the invention and its practical application to thereby enable one of ordinary skill in the art to utilize the invention in various embodiments and with various modifications as are suited to the particular use contemplated. All such modifications and variations are within the scope of the invention as determined by the appended claims when interpreted in accordance with the breadth to which they are fairly, legally, and equitably entitled.