Cooling system control and servicing based on time-based variation of an operational variable

[0001] This application is a continuation of U.S. Ser. No. 13/442,897, filed Apr. 10, 2012, and entitled “Cooling System Control and Servicing Based on Time-Based Variation of an Operational Variable”, and which is hereby incorporated herein by reference in its entirety.

[0002] As is known, operating electronic devices produce heat. This heat should be removed from the devices in order to maintain device junction temperatures within desirable limits. Failure to remove heat can result in increased device temperatures, potentially leading to thermal runaway conditions. Several trends in the electronics industry have combined to make thermal management a significant consideration, including heat removal for electronic devices, particularly in technologies where thermal management has traditionally been less of a concern, such as CMOS. In particular, the need for faster and more densely packed circuits has had a direct impact on the importance of thermal management. First, power dissipation, and therefore heat production, increases as device operating frequencies increase. Second, increased operating frequencies may be possible at lower device junction temperatures. Further, as more and more devices are packed onto a single chip, power density (Watts/cm²) increases, resulting in the need to remove more power from a given size chip or module.

[0003] An alternate approach to traditional air-cooling of electronic devices or components is to use a cooling liquid. In a cooling system comprising a cooling liquid, the cooling liquid may be in direct or indirect contact with the electronic component to be cooled. In particular, the performance and reliability of high-power CMOS circuits may be improved using liquid-cooling, such as refrigeration or water, rather than air-cooling. In such a system, performance of the liquid cooling system can be a significant factor in the performance of the electronic component (e.g., processor, multichip module, or server) being cooled.

[0004] One method of handling a significant degradation in observed temperature(s) for a component being cooled is to cut the clock speed employed, for example, in half. Such a method lowers the power being generated by the logic device sufficiently so that damage concerns due to high temperatures are eliminated. The approach is essentially simple thermal protection to prevent damage to the logic device or component. More proactive methods of cooling system monitoring and replacement would be commercially advantageous.

[0005] The shortcomings of the prior art are overcome and additional advantages are provided through the provision of a method of operating a cooling system cooling at least one component. The method includes, for instance, monitoring over a period of time variation of an operational variable of the cooling system or of the at least one component cooled by the cooling system, and based, at least in part, on the variation of the operational variable over the period of time, determining whether to adjust control of the cooling system to limit the variation of the operational variable.

[0006] Control systems and computer program products relating to one or more aspects of the present invention are also described and claimed herein. Further, services relating to one or more aspects of the present invention are also described and may be claimed herein.

[0007] Additional features and advantages are realized through the techniques of one or more aspects of the present invention. Other embodiments and aspects of the invention are described in detail herein and are considered a part of the claimed invention.

[0024] As used herein, the term “electronics rack” includes any frame, rack, housing, blade server system, etc., having at least one heat generating electronic component of a computer system or electronic system, and may be, for example, a stand-alone computer processor having high, mid or low end processing capability. In one embodiment, an electronics rack may compromise multiple nodes or books, each having one or more heat-generating electronic components requiring cooling. Each “heat-generating electronic component” may comprise an electronic device, an electronic module, an integrated circuit chip, a multi-chip module, (comprising, for example, one or more processor chips), etc.

[0025] The word “refrigerant” is used herein to refer to any coolant which can be employed in a vapor compression/expansion system. One example of refrigerant within a cooling system in accordance with an aspect of the present invention is R-134A coolant (i.e., 1,1,1,2 tetrafluoroethane), however, the concepts disclosed herein are readily applied to other types of refrigerants, other dielectric fluids (e.g., fluorocarbon fluid), or other types of coolants while still maintaining the advantages and unique features of the present invention.

[0026] Generally stated, disclosed herein is an automated control facility for a cooling system cooling at least one electronic component. The control facility includes monitoring over a period of time variation of an operational variable of the cooling system or of the at least one electronic component, and based, at least in part, on variation of the operational variable over the period of time, automatically determining whether to adjust control of the cooling system to limit variation of the operational variable. In the implementations presented below, depending on the frequency and magnitude of the variation of the operational variable, and whether control of the cooling system has been previously adjusted, the method may further include automatically determining a probability of fail or an expected residual life for one or more elements of the cooling system, and responsive to the predictive probability of fail exceeding a first acceptable threshold or the expected residual life being below a second acceptable threshold, automatically scheduling for a cooling system repair or replacement.

[0027] Initially, FIGS. 1A-3B are described below as detailed examples of a cooling system and electronic component within which the control facility disclosed herein may be employed. Note in this regard that the control concepts presented below with reference to FIGS. 4A-9 may be implemented in combination with a cycle steering application, if desired for a particular application.

[0028] FIG. 1A depicts a cooling system 100 as an exemplary embodiment of the present invention. Cooling system 100 includes a condenser 104 and two evaporators 106 and 108. Evaporators 106 and 108 cool heat generating electronic components 110 and 112, respectively. In this embodiment, components 110 and 112 are each one or more multi-chip modules (MCMs), but it is understood that other components (e.g., single processors, memory) may be similarly cooled.

[0029] Both evaporators 106 and 108 are supplied refrigerant from a common condenser 104. An expansion valve 114 receives high pressure liquid refrigerant from condenser 104 and generates low pressure liquid refrigerant to evaporator 106. An expansion valve 116 receives high pressure liquid refrigerant from condenser 104 and generates low pressure liquid refrigerant to evaporator 108. Expansion valves 114 and 116 are electrically controllable. A controller 120 provides control signals to expansion valve 114 and expansion valve 116 to control refrigerant flow and pressure drop across each expansion valve. In an exemplary embodiment, expansion valves 114 and 116 each includes a stepper motor that responds to control signals from the controller 120. The stepper motor opens or closes an orifice in the expansion valve to regulate refrigerant flow and pressure drop. Controller 120 executes a computer program to control the expansion valves 114 and 116, in accordance with one or more aspects of the present invention.

[0030] The low pressure liquid refrigerant exits the expansion valves 114 and 116 and is supplied to evaporators 106 and 108, respectively. By way of example only, the refrigerant in each evaporator 106 and 108 is converted to a low pressure vapor refrigerant, in part, though further fixed expansion structure 107, 109, respectively, and subsequently provided to a common compressor 122. High pressure vapor from compressor 122 is supplied to condenser 104. Fan 126 establishes air flow across condenser 104 to facilitate cooling the high pressure vapor refrigerant to high pressure liquid refrigerant.

[0031] In one embodiment, a plurality of temperature sensors may be distributed throughout the cooling system 100. The sensors may be thermistors or other known temperature sensors. Sensor T1 measures air temperature entering condenser 104. Sensor T2 measures the temperature exiting condenser 104. Sensors T3 and T3′ provide redundant measurement of refrigerant temperature exiting condenser 104. Sensor T4 measures refrigerant temperature entering condenser 104. Sensor T6 measures refrigerant temperature entering evaporator 106 and sensor T7 measures refrigerant temperature existing evaporator 106. Sensor T8 measures refrigerant temperature entering evaporator 108. Sensor T_hat1 measures temperature at electronic component 110 and sensor T_hat2 measures temperature at electronic component 112.

[0032] Each temperature sensor employed generates a temperature signal which is supplied to controller 120 and shown as T_in. In one embodiment, the control or controller 120 may automatically adjust the expansion valves 114 and/or 116 in response to one or more of the temperature signals to, for example, maintain the electronic component logic modules 110 and 112 at a predefined temperature. In one aspect, controller 120 controls expansion valves 114 and/or 116 to obtain desired superheat valves while maintaining each electronic component at a desired temperature. Each component 110 and 112 may be maintained at a different temperature or the same temperature, even if the components have different heat loads.

[0033] Evaporators 106 and 108 may be connected to the refrigerant supply and refrigerant return lines through quick disconnect connectors 130. The controllable expansion valves 114 and 116 allow an evaporator to be removed for maintenance or upgrade while the other evaporator, condenser and compressor continue to operate. For example, expansion valve 114 can be closed and the refrigerant from evaporator 106 removed by the suction of the compressor 122. Evaporator 106 can then be removed for service, upgrade, etc.

[0034] Although two evaporators are shown connected to one modular refrigeration unit (MRU) (condenser, compressor, expansion valves and controller), it is understood that more than two evaporators may be coupled to each MRU, each via a respective cooling loop.

[0035] FIG. 1B is a simplified depiction of one embodiment of a cooling system 100′. In this cooling system 100′, a single cooling loop 150 is illustrated comprising an evaporator 151, a compressor 152, a condenser 153 with an associated air-moving device 154, an electronic expansion valve 155, and a control 160, such as stepper position controller or proportion-integral-derivative (PID) controller. As illustrated, evaporator 151 is coupled to an electronic component 170, such as a multichip module employing, for example, a thermal interface material 171. A temperature sensor T_hat measures temperature at electronic component 170 and provides the temperature reading to controller 160 for use in automatically adjusting electronic expansion valve 155 to, for example, control temperature of refrigerant passing through evaporator 151. In accordance with aspects of the present invention, variation of the temperature T_hat may be monitored over a period of time and employed in determining whether to automatically adjust control of the cooling system to, for example, limit variation of the temperature sensed at sensor T_hat.

[0036] By way of further detailed example, one embodiment of one method of monitoring and controlling the temperature of a hybrid cooling system 100 is described below with reference to FIGS. 2 and 3A. FIG. 2 illustrates a flowchart that shows how a Modular Refrigerant Unit (MRU) code 200, which contains a method to monitor and regulate the MCM (or more generally, electronic component) temperature under primary MRU cooling, by interface with a Power Control Code (PCC) 210, which contains a method to determine and communicate the thermal state or range that equates to a specific temperature and voltage condition of each MCM, and a Cycle Steering Application (CSA) code 220, which contains a method of matching the various logic clocks to a thermal degrade state that exists. In one embodiment, the MRU code, PCC code and CSA code, all interact to form a single temperature-power-logic control system generally indicated as 230.

[0037] FIG. 3A shows one embodiment of a system schematic wherein MRU code 200, PCC code 210 and CSA code 220 are physically located in a server that has four Processor (PU) books or nodes 242, 244, 246, 248, respectively, each having at least one electronic component or MCM cooled in primary mode by one of two MRUs 250, 252 and in backup mode by two blowers 254. The backup blowers 254 provide air cooling of all PU books 242, 244, 246, 248, for example, in case of an MRU failure or a light logic load state. Each MCM is operably connected to a main system board generally indicated at 256. The MRU code 200 is in each MRU 250, 252. The PCC code 210 is split between Base Power Cage Controllers or Base Power Assemblies 260, 262 and digital converter assemblies (DCA) cage controllers (DCA 01, 02, 11, 12, 21, 22, 31, 32). The Base Power Assemblies 260, 262 provide high voltage DC power to the entire server 240 and the DCAs convert the high DC power to low DC voltages used by each circuit. The CSA code 220 may be located in the first Processor book 244 (labeled PU Book 0) of multi-node server 240.

[0038] Each MCM (not shown) in each PU book 242-248 may, in one embodiment, include a hat 274 in operable communication with a cooling unit 10 and connected to a thermal sensor assembly 276. Each thermal sensor assembly 276 may include, for example, three thermistors configured to sense a temperature of a corresponding MCM.

[0039] The thermal sensors may be compared for miscompare properties and for meaningful values to make sure the temperatures measured are accurate. In one embodiment, one sensor may be directly sensed by the Modular Refrigeration Unit (MRU) indicated generally at 278 and the other two by the power supply feeding the MCM power indicated generally at 280 to insure redundancy and accuracy of the reading. The MRU may read an MCM hat thermistor sensor directly through its drive card to enable continual monitoring and thermal regulation in case of a cage controller (cc) failover. MCM hat thermistors that are read by each DCA power supply as well as by the MRU may be compared to each other by the MRU and Power Control Code to identify any faulty sensors and eliminate the faulty sensors from consideration generally indicated at 286 in FIG. 2. This insures redundancy of control and cooling status function. The power supply thermistor also serves as thermal protection of the MCMs, dropping power if the temperatures are near damage limits.

[0040] The control of the primary cooling system is done by using a Proportional Integral Derivative (PID) control loop of an electronic expansion valve to each evaporator as described with reference to FIGS. 1A & 1B, and generally indicated at 290 in FIG. 2. The PID control loop regulates the coolant flow to each MCM being cooled. Coolant flow is increased by opening the electronic expansion valve in the respective cooling loop if (for example) the MCM is too warm or is higher than targeted and coolant flow is reduced by closing the valve position if (for example) the MCM is too cold or cooler than targeted.

[0041] When the PID control has opened an electronic expansion valve to the fully open position, providing maximum coolant to a given MCM, the compressor speed may then execute its own PID control loop to deliver additional cooling capacity to the MCM. In other words, a second PID control loop may control the compressor speed if the valve regulating the flow of coolant to a respective evaporator has reached its maximum cooling position.

[0042] Similarly, the blower speed of blower 126 cooling the refrigerant condenser 104 may be controlled based on the cooling capacity needs from the MRU. More specifically, blower speed controls may provide more air for cooling the MRU condenser 104 when the thermistors T1 and T2 on the condenser 103 and ambient air indicate that inadequate condensing is taking place. Also, the speed of condenser blower 126 may be increased in a warm ambient.

[0043] In one embodiment, MCM power data 284, read by the Power Control Code 210 and provided to the MRU code 200 every few (e.g., 2.5) seconds, determines if a given MCM no longer has its clocks functioning. If the MCM power stays low (indicating a non-functional Processor book) for sufficient time, then the refrigerant coolant supply is stopped by completely closing the expansion valve in the cooling loop to that MCM only and turning on the backup blowers 254 at a reduced speed. In this manner, other MCMs in the same server can stay refrigerant cooled while the MCM that has clock-stopped or otherwise ceased to function logically will be air cooled. Refrigerant cooling of an MCM without adequate logical power can lead to condensation forming on its external surfaces. For example, when regulating light heat loads to a fixed temperature, the expansion device must significantly close the refrigerant flow rate, which lowers the pressure and hence the refrigerant temperature inside the evaporator cooling that MCM. when the clocks are off, the expansion valve closes so far that the evaporator pressure may be sub-atmospheric, which creates very cold local temperatures. With these cold local temperatures with low heat flux, outside regions of the MCM can get cold enough to form condensate after extended operation in this condition.

[0044] The MRU code 200 also provides a function that enables virtually all of the refrigerant to be removed from the evaporator of a corresponding cooling unit before the refrigerant lines are opened for servicing the MCM or cooling hardware. This can be provided by closing the electronic expansion valves for some period before turning off the compressor(s), resulting in a partial vacuum that removes the refrigerant from the evaporator and connecting hoses, which advantageously provides better ecology and consistent refrigerant charge before and after the MRU is reconnected.

[0045] Still referring to FIGS. 2 and 3A, one detailed description of the Power Control Code (PCC) 210 which principally includes a method for monitoring the actual thermal or degrade state and for making suitable power and cooling adjustments, as well as reporting this state to the CSA code 220, follows below. In one embodiment, the thermal states of each MCM are monitored and the state of each MCM is communicated to a function that determines the proper clock cycle time, called the Cycle Steering Application (CSA) code 220. This function tells the CSA code 220 both which cycle time range of the circuits are now operating in and whether the cause of a failure of the primary cooling means has been repaired or not.

[0046] In particular, PCC 210 continually monitors and posts “cooling state” data to the CSA code 220 indicated generally as 292. The thermal state is defined by discrete temperature ranges that are associated with a given clock speed as the proper speed to operate. In other words, the full operating temperature range from coldest to ambient to shut-down for thermal protection is subdivided into smaller discrete operating ranges. The coldest steady state temperature range is called the normal state, and is the temperature range kept under normal primary cooling means (e.g., MRUs 250, 252 and cooling units 10). When the primary cooling means no longer functions properly, the cooling state, sensed via the MCM sensors 276, may be reported as a specific “degrade state”. By way of example, there may be between 2 and 4 degrade states between normal operation and thermal shut-down, but more or less are also contemplated, and hence, these concepts are not limited to between 2 and 4. Within a given degrade state, there exists one “optimum” set of clock speeds.

[0047] The PCC 210 reads the actual current 294 and voltage 284 being supplied to each MCM as well as its temperature 286. Based on the leakage characteristics of the CMOS technology, the capacity left in the power supply providing the current to the MCM, and operating temperatures, the PCC 210 may either increase or decrease or leave along the applied voltage level to each set of circuits indicated generally 296.

[0048] When the voltage is increased, the increased voltage enables a higher range of operating temperatures before a given degrade state is indicated by the CSA code 220 to slow the clocks. Hence, the higher voltage can delay the need to operate in a slower clock range. This is because CMOS switches are faster at higher voltages, somewhat offsetting the slowing effects of warmer circuits.

[0049] Normally, it is desirable to increase voltage applied to the circuit to offset some of the slowing effect on circuit switching of warmer circuits. Typically, a 6% increase in voltage will cause circuits to switch about 4% faster, offsetting a 25° C. temperature rise. However, with recent circuit technology, power increases strongly with higher temperature and increased voltage. In some cases it may require the voltage to be dropped when the junction temperature rises significantly, even though this lowering of voltage will increase the amount of slowing of the clock frequency that is needed. There are three possible responses to loss of normal cooling: doing nothing, increasing voltage, and lowering voltage. A voltage alteration may be done to all components in a system or just to specific electronics components that are exceeding normal cooling limits.

[0050] The Cycle Steering Application (CSA) code 220 provides a fail-safe method of adjusting the clock speeds in an optimum manner when the cooling state changes. This method of clock speed adjustment includes determining if a cooling failure has been repaired prior to increasing the clock speeds to prevent oscillating clock speeds. It should be noted that the clock speed follows the temperature and voltage conditions a all times. Further, the time from a change of circuit temperature to a corresponding change in clock speed is slow enough that the temperatures of the circuits change minimally, less than about 1° C., during this process.

[0051] The CSA node 220 includes an interrupt handler that reads directly from the PCC 210 the cooling state of each MCM as well as receiving interrupts on these states.

[0052] For systems with multiple processor books or nodes, the CSA code 220 determined which MCM has the slowest cooling state. This is the state that governs the safe clock speed of the system indicated generally at 310 in FIG. 2. The multiple clock boundaries on multiple oscillators with predefined ratios are always maintained.

[0053] The CSA code 220 determines if any cooling defective hardware registers are set whenever a cooling state is increased calling for a faster clock speed. If the hardware defect register is set, it means the cause of the cooling degradation has not yet been fixed and the change in cooling state is likely due to transient change in ambient or other transient conditions. Hence, the server clock speeds are not re-adjusted faster until the defective cooling hardware is replaced and the register cleared. This is true even after the machine is re-initial microcode loaded or rebooted. If there is uncertainty in the cooling state due to communication problems, the slowest, safest cooling state is employed by the CSA code 220.

[0054] When the CSA code 220 determines it is appropriate to make a change in several clock speeds, it alters the phase look loops (PLL) on the clock synthesizers in a sequence of very small steps until its new targeted clock speed is reached generally indicated as 312. The phase lock loops are stepwise changed, always retaining the optimum operating ratio between the various clocks that may be affected. The steps are sufficiently small to pose no risk to proper operating due to change in clock ratios during this adjustment process.

[0055] Every step is performed in a two step commit algorithm, e.g., the current step and the next step PLL values are saved in a persistent storage concept made up by using SEEPROMS residing on the current and backup cage controller 262, 262. After the change is written to the PLL and read back for verification, the saved current value is updated. This is done to provide protection in case a speed change is interrupted by a cage controller switchover.

[0056] The width of the small steps taken on the phase lock loops is less than the normal jitter of the phase lock loop normal output. This allows the step variation not to be detected by the target clock receiving circuitry. In this manner, all of the affected clocks are stepped in small increments until the targeted clock speed is achieved.

[0057] The PLLs are on two oscillator cards 263, one in charge, one in backup mode. At all times the optimum ration between clocks is maintained as the phase lock loops are moved in minimal increments or decrements.

[0058] Prior to power good time, the CSA code 220 issues a “Pre-Cooling” command to insure that the MCM temperatures are in proper normal state prior to turning on the clocks. This also prevents a sudden spike in temperature from a surge of power when the CMOS logic begins to switch. Without pre-cool, the temperature spike could cause a quick degrade state to occur because the refrigerant system takes some time to get is cooling cycle established. When pre-cooled state is reached the PCC 210 notifies the CSA code 220 of the same and IML is initiated.

[0059] The PLLs are initially loaded with a pattern, which is hard wired on the cards and loaded in parallel at power good time. Normally, PLLs are loaded serially, but this is exposed to shift errors which would lead to wring clock speed settings.

[0060] The exact process of initializing clocks includes first verifying the right oscillator card 263. Then, the pattern matching the actual system speed is loaded into the line drivers and read back to insure that there are no errors or hardware failures. Next, the loaded and verified pattern is read into the phase lock loops, with this pattern again read back to be verified. Now the system clock is started using the phase lock loop output as input. At the completion of IML, the system is degraded to it slowest clock state and upgraded back to its normal state with the required number of small incremental steps to the phase lock loops. This insures that all necessary patterns can be loaded into the phase lock loops without system error. This process takes a fraction of a second to complete on every server that is IMLed.

[0061] The pattern to be loaded for speed adjustment purposes such as when going from one cooling state to another is generated by a set of digital I/O lines controlled by the FGAs DIO engines, which is a part of the cage controller (cc) hardware. The FGAs DIO engines are digital I/O lines controlled by cage controller code that interface to the PLLs that control the system oscillators 263. They are CSA code driven which is running on the PU Book 0 cage controller (cc). Before changing the PLL pattern due to a change in cooling state, the existing pattern is monitored to make sure the adjusting processes were not interrupted, by saving the line settings of the current pattern.

[0062] The CSA code 220 issues a warning server reference code (SRC) to the operator whenever the CSA code leaves normal clock speed. When the service is completed, the PCC 210 removes the error states and interrupts the CSA code 220. The CSA code 220 removes SRC once notified.

[0063] The CSA code 220 monitors the actual speeds used for an IML to assure these speeds are never increased in actual operation even though the cooling state later permits the increased speed. The reasons for this is that the initialization of “Elastic Interfaces” (EI) done during IML allows only the speed reduction and its clearing, not faster speeds than those present during IML initialization and self-tests.

[0064] Hence, the CSA code 220 notifies the operator that re-ILM should be avoided while a cooling failure service register is flagged so that when the cooling hardware problem is repaired, the server can return to its fast normal speed without needing a subsequent re-IML. Also contemplated is a repair and verify procedure that verifies that the clocks have returned to full speed while a customer engineer is present.

[0065] The above-described cycle steering approach is essentially a thermal protection approach to preventing damage to the logic device or component by reducing clock speed, and thus logic performance. Alternative methods of cooling system monitoring and predictive repair/replacement are presented herein and described below with reference to FIGS. 3B-9.

[0066] FIG. 3B is a simplified depiction of one embodiment of a data center 300 comprising one or more electronics racks 310 and a cooling system 320 comprising a first modular refrigeration unit (MRU1) and a second modular refrigeration unit (MRU2), each comprising a first cooling loop and a second cooling loop cooling respective nodes of a respective electronics rack 310. In this embodiment, each node comprises multiple processor chips, such as five processor chips. The cooling system 320 further includes a remote overview monitor/controller, which may be disposed within the data center or remote from the data center. In one implementation, the remote overview monitor/controller is remote from the data center and collects cooling information from a plurality of data centers, each of which may comprise multiple electronics racks, one or more of which may include cooling system elements such as depicted in FIG. 3B. In one specific embodiment, the remote overview monitor/controller is implemented as part of an offered monitoring service providing predictive failure analysis and/or expected residual life analysis on the cooling systems of one or more data centers.

[0067] In the graph of FIG. 4A, one of the nodes (node 2) of the electronics rack depicted in FIG. 3B, has temperature of, for example, its five processor chips (NODE2EP0, NODE2EP1, NODE2EP2, NODE2EP3, NODE2EP4) monitored and plotted over the course of a day. As described herein, chip temperature is one example of an operational variable that may be monitored over a period of time, such as a 24-hour cycle. Other operational variables might include temperature of coolant passing through the respective coolant loop of the cooling system, power consumed by the processor chips being cooled, work being performed by the processor chips, etc. Note also that as used herein, the term “operational variable” relates to or characterizes performance of the cooling system itself or the one or more components being cooled by the cooling system. Severe performance modulations in an operational variable over a period of time have been discovered to correlate to cooling system failures. Thus, predictive analysis is also described herein, which allows for proactive scheduling of servicing or replacement of a cooling system based, at least in part, on monitored variations of the chosen operational variable over a defined period of time. Processor chip temperature is thus discussed herein as one example only of an operational variable which may be employed with the operational and predictive control concepts disclosed herein.

[0068] As illustrated in FIG. 4A, one or more nodes of the monitored electronics rack may have strong or significant modulations over the period of time during which chip temperature is monitored. In FIG. 4B, temperature modulations of a different node (node3) are plotted, wherein the modulations in processor chip temperatures are significantly less than those plotted in FIG. 4A. As described herein, the reduced modulations of FIG. 4B are preferred for operation of the cooling system. Strong modulations in an operational variable, such as temperature of the processor chips, can be detrimental to the longevity of the cooling system. For example, strong modulations accelerate mechanical wear-out, lead to performance degradation of the processor chips (such as lower-on currents, and higher-off currents at high temperatures), lifetime degradation of processor chips, for example, higher Negative Bias Temperature Instability (NBTI) at high temperatures, and are correlated to a reduced lifetime of the cooling system, e.g., refrigeration system, such as the above-described modular refrigeration unit (MRU).

[0069] The solution proposed herein is to assume causality between strong operational variable modulations over a monitored period of time and cooling system lifetime. Thus, in addition to monitoring and dynamically adjusting control of a cooling system to limit variation of an operational variable, disclosed herein is an automated predictive failure analysis for one or more elements of the cooling system and/or expected residual life analysis for one or more cooling system elements, which (in one implementation) may be periodically performed, depending upon the monitored oscillations of the operational variable.

[0070] FIG. 5 depicts one embodiment of a process for monitoring variation of an operational variable of a cooling system or of a component being cooled by the cooling system over a period of time, and for storing selected oscillation data and transmitting the data to a remote monitor/control, in accordance with one or more aspects of the present invention.

[0071] The process begins by initiating a data collection window (i.e., period of time) collecting operational variable data, such as a 24-hour period, and collecting operational variable data 500. After collecting the first operational variable data, processing waits a time interval t₁ 505, before again collecting operational variable data 510. In the embodiment depicted, processing then determines whether the absolute value of the change in the operational variable data (Δ) between the two most recent collection points is greater than a first acceptable threshold (threshold1) 515. If “no”, then processing returns to wait time interval t₁ before again collecting operational variable data, and determining whether the absolute value of the change (Δ) in operational variable data between the last two collection points is greater than threshold1 515. If the change in the absolute value of the operational variable data is greater than threshold1, then processing stores that change information (for example, locally) 520. The process repeats until reaching the end of the specified data collection window 525, at which point the data may be transmitted to the remote monitor/control 530 for further analysis, as described herein. Thereafter, processing initiates a next data collection window, and repeats the process.

[0072] By way of specific example, if the operational variable is a temperature of the component being cooled, such as temperature of one or more processor chips of a node, then the processing of FIG. 5 may include every time interval t₁ (e.g., every five minutes), determining whether one of the processor chips in the node moves in temperature up or down more than the acceptable threshold1, e.g., 2° C., relative to the previously collected operational variable data. If that is the case, then the change is stored locally, for example, within the electronics rack or data center, and at the end of a specified time, for example, once a day or once a week, the stored operational variable changes exceeding threshold1 may be transmitted to the remote monitor/control. Note that the length of the data collection window, as well as the frequency of data collection and/or the frequency of transmission, need not be constant or predetermined, but could, for example, be dynamically determined or altered via the remote monitor/control.

[0073] As noted above, in one example, the operational variable is a temperature associated with the cooling system or the one or more components being cooled by the cooling system. For example, the operational variable may be temperature of a processor chip or node being cooled by a cooling loop of the cooling system. As part of the monitoring of operational variable data changes, processing can determine a variable referred to herein as the “daily oscillation metric”. In one embodiment, the daily oscillation metric is defined herein to comprise a sum of operational variable changes over a defined threshold in each set of data collected. Note in this regard, that the processing of FIG. 5 may be performed, in one embodiment, for each cooling loop of the cooling system within an electronics rack. Each cooling loop may be cooling one or more components of the electronics rack, such as a node of the rack comprising multiple processor chips. The daily oscillation metric (DOM) advantageously combines two aspects of operational variable data monitoring. Specifically, frequent operational variable changes may be detrimental to the cooling system. In addition, high-amplitude operational variable changes may be detrimental to the cooling system. The DOM captures these changes in a single daily metric.

[0074] In one aspect, described herein below with reference to a cooling system comprising one or more refrigeration units is a process for automatically adjusting control, such as via proportional-integral-derivative (PID) control parameters of the refrigeration unit(s). PID control is a generic control loop feedback mechanism, and PID control parameters include a proportional gain, an integral gain, and a derivative gain. Table 1 depicts sample control parameter values for a refrigeration unit's PID parameters, before and after a dynamic change in control to, for example, ensure less aggressive and slower variation of the operational variable within the monitored period of time, and subsequently, less oscillation.

[0075]

	TABLE 1
	Before Change	After Change
	Proportional Gain	1.5	0.9
	Integral Gain	3.0	2.5
	Derivative Gain	0.4	0.3

[0076] In FIG. 6A, chip temperature data is plotted before and after PID parameters are changed for a node comprising five processor chips, plotted over a four-month time interval. FIG. 6B depicts the daily oscillation metrics for the corresponding data, before and after the PID parameter changes, plotted over the same time interval. Note with respect to FIG. 6B that the DOM exhibits no sensitivity to set point shifts, and little sensitivity to spikes or gaps, but is sensitive to increased reporting frequency. In FIG. 6C, the daily oscillation metrics are plotted against frequency of occurrence, wherein a small number of daily oscillation metrics 600 are illustrated to be outliers. It is these outliers which are to be identified and addressed, in accordance with the aspects of the present invention.

[0077] The solutions disclosed herein condense (in one implementation) the daily oscillation metrics into a value for each cooling loop. This value is referred to herein as the oscillation metric. The average of the daily oscillation metric over a time interval, for example, three months, may be used, and a percentage of days with the daily oscillation metric above a threshold (for example, in the case of a temperature-based operational variable, 150° C.) may be determined. Further, the weighted moving average of the daily oscillation metric may be considered.

[0078] By way of example, FIG. 7A depicts an exemplary oscillation metric illustrating a number of daily oscillation metrics above, for example, 150° C. As described below in connection with FIG. 7B, periodically, or on demand, the oscillation metric may be evaluated to determine, for example, whether at any given time it exceeds an acceptable threshold, threshold2. By way of example, threshold2 might comprise 200° C. in an example where the operational variable is processor chip temperature. If the oscillation metric exceeds threshold2, then control of the cooling system may be dynamically adjusted to limit the variation of the operational variable. This may include changing PID control parameters employed in the MRU control (in one example).

[0079] Referring to the processing of FIG. 7B, in one implementation, the remote monitor/control periodically determines an oscillation metric by condensing the daily oscillation metrics into a single value for each cooling loop 700. Each oscillation metric is then compared against an acceptable threshold, threshold2 for the oscillation metric, and processing determines whether the oscillation metric is greater than the set threshold 705. Assuming that the oscillation metric is below the acceptable threshold 705, then no action 710 is taken. Otherwise, processing determines whether control of the respective cooling loop has been previously adjusted by determining, for example, whether the original PID parameters are still in use within the respective cooling loop 715. If “yes”, then (in one implementation) the original PID parameters are dynamically adjusted 720 to limit the variation of the operational variable, and thereby reduce the oscillation metric. If the original PID parameters are no longer in use in the respective cooling loop, and the oscillation metric exceeds the acceptable threshold, then (in one implementation) processing automatically performs predictive failure analysis and/or expected residual life analysis on one or more components of the cooling loop 725. One embodiment of this analysis is depicted in FIG. 8 for a cooling loop comprising a modular refrigeration unit such as described above with reference to FIGS. 1A, 1B, 3A & 3B.

[0080] Note that in an alternate implementation, performing predictive failure analysis and/or expected residual life analysis, could be independent of the oscillation metric evaluation and/or the dynamic PID parameter change. It might be automatically performed, for example, based on operator request or automatically periodically, such as monthly. In the description provided below, the cooling loop at issue is assumed to comprise a modular refrigeration unit with a compressor, and an electronic control or expansion valve(s). Referring to FIG. 8, performing predictive failure analysis may include carrying out log-normal and/or Weibull analysis for each cooling loop, that is, on each MRU 800. Additionally, or alternatively, the expected residual life (ERL) for each MRU may be determined based on Weibull analysis for the probability of compressor fails and/or log-normal analysis for electronic expansion valve fails. If the probability of fail of a particular MRU within a set period is greater than an acceptable threshold, or the expected residual life for that MRU is less than an acceptable residual life threshold 805, then processing may automatically initiate, for example, schedule MRU repair and/or replacement 820, otherwise, no action is taken 810, and processing returns 815 to, for example, the processing flow of FIG. 7B to continue evaluation of the oscillation metric.

[0081] By way of example, conditional failure distribution under Weibull and log-normal with covariate may be determined using Equations (1) & (2) below.

[0082] Assuming T to be life of the refrigeration unit, and the underlying distribution to be log-normal with covariates, then predicting probability of fail after an extra δT may be determined by:

[0083] $\begin{array}{cc}P\left(X\le T+\delta T❘X>T\right)=1-\frac{\Phi \left(\frac{-\ln \left(T+\delta T\right)+\left(\mu +\sum _{i=1}^n{\alpha }_i{\gamma }_i\right)}{\sigma }\right)}{\Phi \left(\frac{-\ln T+\left(\mu +\sum _{i=1}^n{\alpha }_i{\gamma }_i\right)}{\sigma }\right)},& \left(1\right)\end{array}$
where:

• • [0084] Φ is the normal cumulative distribution function;
  • [0085] μ is the location parameter;
  • [0086] σ is the scale parameter; and
  • [0087] γ_i and α_i, i=1 . . . n are covariates and their covariates coefficients, respectively.

[0088] Assuming T to be life of the refrigeration unit and the underlying distribution to be Weibull with covariates, then predicting probability of fail may be determined by employing:

[0089] $\begin{array}{cc}P\left(X\le T+\delta T❘X>T\right)=1-\exp \left(-\left\{{\left(\frac{T+\delta T}{\beta *}\right)}^c-{\left(\frac{T}{\beta *}\right)}^c\right\}\right),& \left(2\right)\end{array}$
where:

• • [0090] c is the shape parameter;
  • [0091] β is the scale parameter;
  • [0092] β* is =βexp(Σ_i=1ⁿα_iγ_i); and
  • [0093] γ_i and α_i, i=1 . . . n are covariates and their covariates coefficients, respectively.

[0094] Given a probability threshold P_t, then one can solve Equation (1) or Equation (2) for δT.

[0095] The log-normal and Weibull expected residual life with covariates may be determined, in one embodiment, using Equations (3) & (4), set out below.

[0096] Suppose that an MRU is of age T. Then in the case of log-normal distribution with covariates, the expected residual life (ERL) may be given by:

[0097] $\begin{array}{cc}E\left(X❘T\right)=\exp \left(\mu +\sum _{i=1}^n{a}_i{y}_i+\frac{{\sigma }^2}{2}\right)\frac{\Phi \left(\frac{-\ln T+\left(\mu +\sum _{i=1}^n{a}_i{y}_i\right)+{\sigma }^2}{\sigma }\right)}{\Phi \left(\frac{-\ln T+\left(\mu +\sum _{i=1}^n{a}_i{y}_i\right)}{\sigma }\right)}-T,& \left(3\right)\end{array}$
where:

• • [0098] Φ is the normal cumulative distribution function;
  • [0099] μ is the location parameter;
  • [0100] σ is the scale parameter; and
  • [0101] γ_i and α_i, i=1 . . . n are covariates and their covariates coefficients, respectively.

[0102] Similarly, if the underlying failure distribution is Weibull with covariate, and putting β*=βexp(Σ_i=1ⁿα_iγ_i), then:

[0103] $\begin{array}{cc}E\left(X❘T\right)={\beta }^{*}\Gamma \left[\frac{1}{c}+1\right]\exp \left({\left(\frac{T}{{\beta }^{*}}\right)}^c\right)\left(1-F_{\mathrm{Gamma}\left(\frac{1}{c}+1,1\right)}\left[{\left(\frac{T}{{\beta }^{*}}\right)}^c\right]\right)-T,& \left(4\right)\end{array}$
where:

• • [0104] c is the shape parameter;
  • [0105] β is the scale parameter;
  • [0106] β*=βexp(Σ_i=1ⁿα_iγ₁);
  • [0107] Γ is the Gamma function;
  • [0108] γ_i and α_i, i=1 . . . n are covariates and their covariates coefficients, respectively; and
  • [0109] F is the cdf of Gamma distribution with

[0110] $\propto =\frac{1}{c+1}$
and β=1.

[0111] In one example, the acceptable residual life might comprise 2 months, and if the expected residual life is less than the acceptable residual life, then the remote monitor/control may automatically initiate repair or replacement of the respective MRU.

[0112] Note that modification of operating parameters during the life of an element of the cooling system can change the reliability of the element. If lifetime of parts are distributed according to a log-normal, then this change in regime could modify the hazard rate, and therefore distribution parameters should be expected to change accordingly. Proposed herein is a definition of equivalent life and a modification of a likelihood function to calculate this new set of parameters.

[0113] In a given cohort of part, let T′ be the life of a part prior to any changes. At T′, a change is introduced in operating conditions and the parts live for an extra duration T. Assuming that there is a bundle of parts with different ages, some of which have seen a change in operating conditions and such that a fraction of these parts have failed, some before and some after the changes in operating conditions, the resultant failure distribution can be determined.

[0114] If one assumes that the underlying failure distribution is log-normal, then with a proper censoring scheme, one can fit a log-normal distribution to T*=T′+T, but this straightforward approach does not take into account the possibility of a different hazard rate before and after the change. In fact, the proper functional way to write T* would be:
T*=ƒ(T*)+T  (5)
where ƒ is a function that still have to be explicited.

[0115] Assume that the failure distribution prior to any change is characterized by the pair (ρ₁, σ₁). After the change, these parameters have shifted to (μ₂, σ₂). Therefore, the equivalent time to fit to a log-normal distribution is rescaled to:

[0116] $\begin{array}{cc}{T}^{*}=\exp \left({\mu }_2-{\mu }_1\frac{{\sigma }_2}{{\sigma }_1}\right)T^{\prime \frac{{\sigma }_2}{{\sigma }_1}}+T.& \left(6\right)\end{array}$

[0117] In fact, when T=0, it is easy to verify that:

[0118] $\begin{array}{cc}\frac{\ln \left(\exp \left({\mu }_2-{\mu }_1\frac{{\sigma }_2}{{\sigma }_1}\right)T^{\prime \frac{{\sigma }_2}{{\sigma }_1}}\right)-{\mu }_2}{{\sigma }_2}=\frac{\ln \left(T^{\prime }\right)-{\mu }_1}{{\sigma }_1},& \left(7\right)\end{array}$
so that the cumulative hazard of T′ under (μ₁, σ₁) is equal to the cumulative hazard of T* under (μ₂, σ₂). To obtain parameter pairs (μ₁, σ₁) and (μ₂, σ₂), we have to modify somewhat the log-likelihood function:

[0119] $\begin{array}{cc}\mathrm{LL}=-\sum _{i=1}^n\log \left(\frac{\Phi \left(\exp \left({\mu }_2-{\mu }_1\frac{{\sigma }_2}{{\sigma }_1}\right)T^{\prime \frac{{\sigma }_2}{{\sigma }_1}}+T\right),{\mu }_2,{\sigma }_2}{1-\Phi \left(-\ln \left(\exp \left({\mu }_2-{\mu }_1\frac{{\sigma }_2}{{\sigma }_1}\right)T^{\prime \frac{{\sigma }_2}{{\sigma }_1}}+T\right),{\mu }_2,{\sigma }_2\right)}\right)-\sum _{i=1}^n\log \left(1-\Phi \left(-\ln \left(\exp \left({\mu }_2-{\mu }_1\frac{{\sigma }_2}{{\sigma }_1}\right)T^{\prime \frac{{\sigma }_2}{{\sigma }_1}}+T\right),{\mu }_2,{\sigma }_2\right)\right),& \left(8\right)\end{array}$
where φ and Φ are the log-normal pdf and cdf, respectively.

[0120] The first summation in Equation (8) is the negative of the sum of the log hazard for failed parts, and the second summation is the negative of the sum of log cumulative hazard for all parts.

[0121] Therefore, one is attempting to find parameters (μ₁, σ₁) and (μ₂, σ₂) that maximize Equation (8). This can be achieved with a non-linear optimization routine (available in many statistical software programs), and once parameters are obtained, one can proceed to calculate equivalent life. Furthermore, if covariates are available, then they can be tested for significance with respect to equivalent life.

[0122] Note that disclosed herein is the concept of using monitoring over a period of time of one or more operational variables of a cooling system or of a component being cooled by the cooling system, to determine whether to adjust control of the cooling system to limit the variation of the operational variables. In one example, the operational variable is a component temperature, and the control of the cooling system may involve automatic control of one or more PID control parameters. This automatic adjustment may be without interruption to the cooling system or the electronic system being cooled. In cases where insufficient improvement is noted, then a predictive method, based (for example) on Weibull and/or log-normal analysis, may be employed to identify cooling loops, and thus, respective refrigeration units, which may fail soon or have a low expected residual life, so that maintenance or replacement of the respective cooling loop or refrigeration unit can be initiated proactively. Note that the mathematical equations described herein take into account the possibility of a change in PID control parameters during the life of the MRU, and still allow for proper Weibull and/or log-normal analysis to be performed.

[0123] As will be appreciated by one skilled in the art, one or more aspects of the present invention may be embodied as a system, method or computer program product. Accordingly, one or more aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “circuit,” “module” or “system”. Furthermore, one or more aspects of the present invention may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon.

[0124] Any combination of one or more computer readable medium(s) may be utilized. The computer readable medium may be a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0125] Referring now to FIG. 9, in one example, a computer program product 900 includes, for instance, one or more non-transitory computer readable storage media 902 to store computer readable program code means or logic 904 thereon to provide and facilitate one or more aspects of the present invention.

[0126] Program code embodied on a computer readable medium may be transmitted using an appropriate medium, including but not limited to, wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0127] Computer program code for carrying out operations for one or more aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language, such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the “C” programming language, assembler or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0128] One or more aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

[0129] These computer program instructions may also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.

[0130] The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

[0131] The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of one or more aspects of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

[0132] In addition to the above, one or more aspects of the present invention may be provided, offered, deployed, managed, serviced, etc. by a service provider who offers management of customer environments. For instance, the service provider can create, maintain, support, etc. computer code and/or a computer infrastructure that performs one or more aspects of the present invention for one or more customers. In return, the service provider may receive payment from the customer under a subscription and/or fee agreement, as examples. Additionally or alternatively, the service provider may receive payment from the sale of advertising content to one or more third parties.

[0133] In one aspect of the present invention, an application may be deployed for performing one or more aspects of the present invention. As one example, the deploying of an application comprises providing computer infrastructure operable to perform one or more aspects of the present invention.

[0134] As a further aspect of the present invention, a computing infrastructure may be deployed comprising integrating computer readable code into a computing system, in which the code in combination with the computing system is capable of performing one or more aspects of the present invention.

[0135] As yet a further aspect of the present invention, a process for integrating computing infrastructure comprising integrating computer readable code into a computer system may be provided. The computer system comprises a computer readable medium, in which the computer medium comprises one or more aspects of the present invention. The code in combination with the computer system is capable of performing one or more aspects of the present invention.

[0136] Although various embodiments are described above, these are only examples. Further, other types of computing environments can benefit from one or more aspects of the present invention.

[0137] As a further example, a data processing system suitable for storing and/or executing program code is usable that includes at least one processor coupled directly or indirectly to memory elements through a system bus. The memory elements include, for instance, local memory employed during actual execution of the program code, bulk storage, and cache memory which provide temporary storage of at least some program code in order to reduce the number of times code must be retrieved from bulk storage during execution.

[0138] Input/Output or I/O devices (including, but not limited to, keyboards, displays, pointing devices, DASD, tape, CDs, DVDs, thumb drives and other memory media, etc.) can be coupled to the system either directly or through intervening I/O controllers. Network adapters may also be coupled to the system to enable the data processing system to become coupled to other data processing systems or remote printers or storage devices through intervening private or public networks. Modems, cable modems, and Ethernet cards are just a few of the available types of network adapters.

[0139] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprise” (and any form of comprise, such as “comprises” and “comprising”), “have” (and any form of have, such as “has” and “having”), “include” (and any form of include, such as “includes” and “including”), and “contain” (and any form contain, such as “contains” and “containing”) are open-ended linking verbs. As a result, a method or device that “comprises”, “has”, “includes” or “contains” one or more steps or elements possesses those one or more steps or elements, but is not limited to possessing only those one or more steps or elements. Likewise, a step of a method or an element of a device that “comprises”, “has”, “includes” or “contains” one or more features possesses those one or more features, but is not limited to possessing only those one or more features. Furthermore, a device or structure that is configured in a certain way is configured in at least that way, but may also be configured in ways that are not listed.

[0140] The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the claims below, if any, are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the invention. The embodiment was chosen and described in order to best explain the principles of one or more aspects of the invention and the practical application, and to enable others of ordinary skill in the art to understand one or more aspects of the invention for various embodiments with various modifications as are suited to the particular use contemplated.