Estimator program for estimating the availability of an application program that runs in a cluster of at least two computers

[0002] This invention relates to data processing systems of the type which include a cluster of at least two computers that execute application programs in a “failover” mode of operation; and more particularly, this invention relates to methods for estimating the “availability” of the application programs in the above type of data processing systems.

[0003] To explain the failover mode of operation as that term is used herein, consider the case where the cluster includes only two computers. Initially, the cluster operates in a first state wherein both of the computers are available to run the application programs. But in that first state, only one of the computers (computer #1) is servicing requests to use the application programs. The cluster remains in the first state until a stoppage occurs in computer #1.

[0004] Then, a transition is made to a second state wherein the other computer (computer #2) assumes responsibility for handling all requests to use the application programs but does not yet run those programs. This second state lasts only temporarily, and it is herein called the failover state. Then a transition is made to a third state.

[0005] In the third state, computer #2 services requests to use the application programs; and at the same time, repair work is performed on computer #1 to try to fix the cause of the stoppage. If computer #1 is made operable before computer #2 stops, then a transition is made back to the first state. Otherwise, if computer #1 is not made operable before computer #2 stops, then a transition is made to a fourth state wherein no requests to use the application programs are serviced.

[0006] The cluster remains in the fourth state until one of the computers is made operable. When that occurs, a transition is made back to the third state. There, the one computer which is operable services all requests to use the application programs; and at the same time, repair work is performed on the stopped computer.

[0007] By the availability of an application program is herein meant the probability that at any particular time instant, at least one of the computers will actually be servicing the requests to use the application programs. In the above described cluster of two computers, the application programs are not available for use in both the second state and the fourth state.

[0008] In the prior art, methods which are somewhat related to the availability of an application program in a cluster of computers is described in a book which is entitled “Reliable Computer Systems” (second edition) by Daniel P. Siewiorek and Robert S. Swarz, copyrighted 1992 by Digital Equipment Corporation and published in Digital Press (hereinafter Siewiorek). There, in FIG. 5-19c on page 314, a three state Markoff model is shown to describe how a the cluster of two computers operates. Also, an equation 32 on page 316 expresses the operability of a computer in the cluster whose state diagram corresponds to FIG. 5.19c.

[0009] However, one problem with Siewiorek is that the above state diagram and equation do not account for any time which it takes to switch the responsibility for handling requests to use the application programs from one computer to another. In particular in FIG. 5.19c, there is no failover state. Thus Siewiorek only addresses when a computer in a cluster is operable, and does not address when an application program is available for use.

[0010] Another problem with Siewiorek is that it only accounts for a single source of stoppage that occurs at a single rate “λ” which has a single repair rate “μ”. This however, is unrealistic because in an actual cluster of several computers, each application program can become unavailable due to a hardware stoppage or a software stoppage or a system administrator stoppage which occur at different frequencies with different repair times.

[0011] Still another problem with Siewiorek is that the expression for the operability of a cluster of two computers, as given by equation 32, is quite complex. In addition, other expressions for the operability of a cluster of more than two computers, as provided by Siewiorek, are even more complex. This is evident from page 839 of the Siewiorek wherein the third formula from the top of the page applies to a cluster of N computers and is extremely complex.

[0012] Accordingly, a primary object of the present invention is to provide a method for estimating the availability of application programs in a cluster of computers by which the above problems are overcome.

[0013] In accordance with the present invention, a novel estimator program is provided which performs method steps for estimating the availability of an application program that runs on any computer in a cluster of at least two computers. In one particular embodiment, the estimator program begins by receiving input parameters which include multiple downtime periods for each computer in the cluster that occur at respective frequencies due to various downtime sources. Also, the estimator program receives an application failover time period for switching the running of the application program from any one of the computers to another. Next, the estimator program uses the input parameters to generate a single computer virtual downtime period and a single computer virtual time between stops and a single computer virtual stoppage rate. Then, the estimator program estimates a first annual stoppage time for the application program, due solely to concurrent stoppage of all of the computers, as a function of the ratio of the single computer virtual downtime period over the single computer virtual time between stops. Next, the estimator program estimates a second annual stoppage time for the application program, due solely to switching the running of the application program from one computer to another as a function of the single virtual stoppage rate and the application failover time period. Then, the estimator program determines the availability of the application program on the cluster from the sum of the first and second annual stoppage times.

[0014] FIG. 1 shows a data processing system, that executes application programs in a cluster of two computers, to which the present invention can be applied to estimate the availability of the application programs.

[0015] FIG. 2 is a timing diagram which shows when hardware stoppages, software stoppages, and system administrator stoppages occur in any one of the two computers within the cluster in the FIG. 1 system.

[0016] FIG. 3 shows an example of how the hardware stoppages, software stoppages, and system administrator stoppages of FIG. 2 affect the availability of the application programs in the FIG. 1 system.

[0017] FIG. 4 is a state diagram which shows all of the operable and inoperable states that occur within the cluster in the FIG. 1 system.

[0018] FIGS. 5A and 5B show a process which estimates the availability of the application programs within the cluster in the FIG. 1 System, in accordance with the present invention.

[0019] FIG. 6 shows an estimator program, that is embodied in a program storage device, which estimates the availability of the application programs by performing the process steps of FIGS. 5A and 5B.

[0020] Referring now to FIG. 1, it shows a data processing system to which the present invention can be applied to estimate the availability of an application program. This FIG. 1 data processing system includes several components, each of which is identified and described below in Table I.

[0021] To initiate the use of one of the application programs in the FIG. 1 system, an operator 80aat a workstation 80 makes an input via his keyboard or mouse. For example, the operator 80acan use his mouse to select a particular item that is displayed on his monitor, and that selection can implicitly ask for certain information that is provided by a particular application program.

[0022] In response to such an input by the operator 80a, the workstation 80 sends a request via the communication network 70 to the cluster 60. From the point of view of the operator 80a, the cluster 60 is a single indivisible node. Thus, computer 10 can respond to the request by running the particular application program and sending a result back to the workstation 80; or, computer 20 can respond to the request in the same fashion.

[0023] Initially, in the cluster 60, all requests on the network 70 to use the application programs are handled by the computer 10. This mode of operation continues until computer 10 becomes unavailable. Then, all requests on the network 70 to use an application program are handled by computer 20.

[0024] Once computer 20 starts to handle all requests on the network 70 for use of the application programs, that mode of operation continues until computer 20 becomes unavailable. Then, if computer 10 is available, all requests on the network 70 to use the application programs are handled by computer 10.

[0025] At various times, computer 10 and computer 20 will both be unavailable; and when that occurs, all requests on the network 70 to use an application program will fail. Thereafter, if computer 10 becomes available before computer 20, then computer 10 will handle all requests on the network 70 to use the application programs. Conversely, if computer 20 becomes available before computer 10, then computer 20 will handle all requests on the network 70 to use the application programs.

[0026] Each of the computers 10 and 20 can become unavailable at any time for three different reasons; and this is illustrated in FIG. 2. There, a curve 101 shows the stoppage of either one of the computers 10 or 20 due to an error in the software 10bor 20b. Also in FIG. 2, a curve 102 shows the stoppage of either one of the computers 10 or 20 due to a fault in the hardware 10aor 20a. Further in FIG. 2, a curve 103 shows the stoppage of either one of the computers 10 or 20 due to human errors by the system administrator 50a.

[0027] Inspection of the curves 101, 102 and 103 show that the software caused stoppages, the hardware caused stoppages, and the system administrator caused stoppages each occur in a random manner with different time durations and different frequencies. For example, in curve 101, two software caused stoppages occur with time durations Tb1 and Tb2; and they are spaced by time intervals Ta1, Ta2 and Ta3. In curve 102, one hardware caused stoppage occurs with a time duration Td1; and it is spaced by time intervals Tc1 and Tc2. In curve 103, three system administrator caused stoppages occur with time durations Tf1, Tf2 and Tf3; and they are spaced by time intervals Te1, Te2, Te3 and Te4.

[0028] Hardware stoppages are caused by the failure of a physical structure within the computers 10 and 20. These physical structures include various items such as power supplies, circuits within the instruction processor, circuits within the random access semiconductor memory, etc. As a realistic average, one hardware stoppage occurs every 1,000-20,000 hours with a time duration for repair of 1-60 hours.

[0029] Software stoppages are caused for a faults within the operating system and/or the application programs 10band 20b. High quality certified software has a small number of faults, whereas low quality uncertified software has a large number of faults. As a realistic average, one software stoppage occurs every 1-1,000 hours with a time duration of 1-60 minutes.

[0030] System administrator stoppages are caused by both intentional and unintentional acts of the system administrator 50a. For example, an intentional stoppage occurs when the system administrator stops one of the computers 10 and 20 to perform maintenance on it or to upgrade the software. An unintentional stoppage occurs when the system administrators makes any error which causes one of the computers 10 or 20 to malfunction. For example, the system administrator 50acan inadvertently delete a file from computer 10 which is needed by the application programs. As a realistic average, one system administrator stoppage occurs every 1-500 hours with a time duration of 1-30 minutes.

[0031] Turning now to FIG. 3, it shows an example of how the hardware stoppages, software stoppages, and system administrator stoppages affect the availability of an application program in the cluster 60. Included in FIG. 3 are seven curves 111-117. Curves 111, 112, and 113 show the operation of computer 10; curves 114, 115 and 116 show the operation of computer 20; and curve 117 shows the downtime for the cluster 60 as a whole. During the cluster downtime of curve 117, the application programs are not available for use by the workstations 80.

[0032] Initially in FIG. 3, at time T0, both of the computers 10 and 20 are operable as is shown by curves 112 and 115. Also at time T0, all requests on the network 70 to use the application programs are handled by computer 10, and this is shown by curves 111 and 114.

[0033] Thereafter, at time T1, a hardware stoppage occurs in computer 10. Due to that hardware stoppage, computer 20 enters a failover mode of operation wherein the responsibility of handling requests on the network 70 to use the application programs is transferred to computer 20.

[0034] In FIG. 3, the reconfigure mode of operation lasts for one minute; and this is shown by curve 116. During that time interval, none of the application programs on the cluster 60 are available for use by the workstations 80, and this is shown by curve 117.

[0035] Thereafter, at time T2, computer 20 begins to service the requests from the workstations 80 to use the application programs. This is shown by curve 114. That mode of operation continues until the operation of computer 20 ends due to a hardware stoppage, a software stoppage, or a system administrator stoppage. Meanwhile, corrective action is taken by the system administrator to fix the hardware problem in computer 10. In FIG. 3, this corrective action takes place during a six hour time period which starts at time T1 and ends at time T3.

[0036] Next, at time T4 in FIG. 3, a software stoppage occurs in computer 20. Thus, computer 20 becomes inoperable as shown by curve 115; and computer 10 enters the failover mode of operation as shown by curve 113. In that failover mode, the responsibility for handling all requests on the network 70 to use the application programs is transferred to computer 10. This reconfiguration mode of operation last for one minute.

[0037] Then as shown by curve 111 at time T5, computer 10 begins handling all of the requests which occur on the network 70 to use the application program. That mode of operation continues until computer 10 becomes inoperable due to a hardware stoppage, a software stoppage, or a system administrator stoppage. Meanwhile, the system administrator takes corrective action on computer 20 to fix the software stoppage that started at time T4. In FIG. 6, this corrective action is completed in a time interval of one hour, which ends at time T6.

[0038] Thereafter, computer 10 becomes inoperable due to a system administrator stoppage; and this is shown at time T7 in curve 112. In response, computer 20 enters the failover mode of operation wherein it becomes responsible for handling all requests on the network 70 to use the application programs. This failover mode of operation lasts for one minute; and then, at time T8, the servicing the requests by computer 20 begins. Meanwhile, the system administrator 50aacts to correct the system administration stoppage problem. This corrective action is completed in a half-hour time period which ends at time T9.

[0039] Turning next of FIG. 4, the availability of an application program in the cluster 60 is indicated by a state diagram. Initially, the cluster 60 operates in a state S1 wherein both of the computers 10 and 20 are available to run the application programs. But in state S1, only one of the computers 10 or 20 is servicing requests from the workstations 80 to use to application program, while the other computer is idle.

[0040] Cluster 60 remains in state S1 until a hardware stop or a software stop or a system administrator stop occurs in the computer which is servicing the workstation requests. When a hardware stop occurs, a transition is made from state S1 to state S2A; when a software stop occurs, a transition is made from state S1 to state S2B; and when a system administrator stop occurs, a transition is made from state S1 to state S2C.

[0041] In each of the states, S2A, S2B, and S2C, one of the computers 10 or 20, enters the failover mode of operation. In that mode, the computer which previously was not servicing any requests from the workstations, becomes responsible for handling all requests to use the application programs. This failover mode of operation lasts for only a short period of time, such as one minute.

[0042] Next, from the failover states S2A, S2B, and S2C, respective transitions are made to the states S3A, S3B, and S3C. In state S3A, a hardware error in one of the computers 10 and 20 is being fixed while the other computer is servicing requests from the workstations 80. Similarly in state S3B, a software error in one of the computers 10 or 20 is being fixed, while the other computer is servicing the workstation requests. Likewise, in state S3C, an administrator stoppage in one of the computers 10 or 20 is being worked on, while the other computer is servicing the workstation requests.

[0043] From each of the states S3A, S3B, and S3C, a transition is eventually made back to state S1 or to another state S4. This transition is made back to state S1 if the one computer which stopped is made operable before the second computer stops. Otherwise, if both of the computers 10 and 20 stop concurrently, then a transition is made to state S4.

[0044] Cluster 60 remains in state S4 until one of the computers 10 or 20 is made operable. When that occurs, a transition is made to state S3A if a hardware problem remains to be fixed; a transition is made to state S3B if a software problem remains to be fixed; and a transition is made to state S3C if a system administrator stoppage remains to be fixed.

[0045] In each of the states S1, S3A, S3B, and S3C, the application programs in the cluster 60 are available for use by the workstations 80. Conversely, in each of the states S2A, S2B, S2C, S4, the applications programs in the cluster 60 are not available for use by the workstations 80.

[0046] Now in accordance with the present invention, steps are provided for estimating the amount of time that is spent in the failover states S2A, S2B, and S3C; and, steps are provided for estimating the amount of time that is spent in state S4 where both of the computers 10 and 20 are inoperable. Then, using those estimations, the availability of an application program on the cluster 60 is determined. These steps will now be described in conjunction with FIGS. 5A and 5B.

[0047] In step 1 of FIG. 5A, an estimate is made of the average number of hardware stops per year (HWSTOPS/YR) that occur in each one of the computers 10 and 20. As an example in step 1, the average number of hardware stops per year per computer is estimated to be 0.8. This step 1 estimate is herein called “A” for ease of reference in the subsequent steps.

[0048] In step 2 of FIG. 5A, an estimate is made of the average time duration of each hardware stop (ATDHW). As an example in step 2, the average time duration of each hardware stop is estimated to be six hours. This step 2 estimate is herein called “B” for ease of reference in the subsequent steps.

[0049] In step 3 of FIG. 5A, an estimate is made of the average number of software stops per year (SWSTOPS/YR) that occur in each one of the computers 10 and 20. As an example in step 3, the average number of software stops per year per computer is estimated to be 12. This step 3 estimate is herein called “C” for ease of reference in the subsequent steps.

[0050] In step 4 of FIG. 5A, an estimate is made of the average time duration of each software stop(ATDSW). As an example in step 4, the time duration of each software stop is selected to be one hour. This step 4 estimate is herein called “D” for ease of reference in the subsequent steps.

[0051] In step 5 of FIG. 5A, an estimate is made of the average number of system administrator stops per year (SASTOPS/YR) that occur in each one of the computers 10 and 20. As an example in step 5, the average number of system administrator stops per year per computer is estimated to be 20. This step 5 estimate is herein called “E” for ease of reference in the subsequent steps.

[0052] In step 6 of FIG. 5A, an estimate is made of the average time duration of each system administrator stop (ATDSA). As an example in step 6, the average time duration of each system administrators stop is estimated to be one-half hour. This step 6 estimate is herein called “F” for ease of reference in the subsequent steps.

[0053] In step 7 of FIG. 5A, an estimate is made of the average time duration of each failover time period (FT). As an example in step 7, the average time duration of each failover period is estimated to be one minute. This step 7 estimate is herein called “G” for ease of reference in the subsequent steps.

[0054] Next, in step 8 of FIG. 5A, the estimates which were made in steps 1-7 are used to determine a virtual stoppage rate for each of the computers 10 and 20. In particular in step 8, this “single computer virtual stoppage rate” (SVCSR) is set equal to the sum of the quantities “A” of step 1, “C” of step 3, and “E” of step 5. A numerical example of this calculation is shown in step 9 wherein the single computer virtual stoppage rate is determined to be 32.8 stops per year.

[0055] Next, in step 10, the estimates which were made in steps 1-7 are used to determine a virtual downtime period for each of the computers 10 and 20 which occurs at the above single computer virtual stoppage rate. In particular in step 10, this “single computer virtual downtime” (SCVD) is set equal to the sum of three products divided by the single computer virtual stoppage rate (SCVSR). Those three products are the quantities A times B, C times D, and E times F as estimated in step 1 through step 6. A numerical example of step 10 is shown in step 11 wherein the single computer virtual downtime is determined to be 0.817 hours.

[0056] Next, in step 12 of FIG. 5A, a time interval which occurs between the virtual stoppages in each of the computers 10 and 20 is determined. In particular in step 12, this “single computer time between virtual stops” (SCTBVS) is set equal to the quantity 8760 divided by the single computer virtual stoppage rate (SCVSR). Here, 8760 is the number of hours per year. A numerical example of step 12 is shown in step 13 wherein the single computer time between virtual stops is determined to be 267.0 hours.

[0057] Next, in step 14 of FIG. 5B, a ratio “R” is determined. This ratio “R” is an estimate of the probability of a single one of the computers 10 or 20 stopping due to a hardware stoppage or a software stoppage or a system administrative stoppage. To obtain this ratio “R”, the single computer virtual downtime of steps 10 and 11 is divided by the single computer time between virtual stops of steps 12 and 13. A numerical example of step 14 is shown in step 15 wherein the single computer virtual downtime of step 11 and the single computer time between virtual stops of step 13 are used.

[0058] Next, in step 16 of FIG. 5B, the ratio “R” of steps 14 and 15 is used to estimate the probability that both of the computers 10 and 20 stop concurrently. In particular in step 16, this “probability of concurrent stoppage” (PCS) is set equal to the quantity 2(R²). A numerical example of step 16 is provided in step 17 herein the ratio “R” from step 15 is used.

[0059] Next, in step 18 of FIG. 5B, an estimate is made of the total time per year during which the computers 10 and 20 are concurrently stopped. In particular in step 18, this “concurrent stoppage per year” (CS/YR) is set equal to the probability of concurrent stoppage from step 16 times the quantity 8760. A numerical example of step 18 is provided in step 19 wherein the probability of concurrent stoppage from step 17 is used.

[0060] Next, in step 20 of FIG. 5B, an estimate is made of the total stoppage time per year during which one of the computers 10 or 20 is in the failover mode of operation. In particular in step 20, this “failover stoppage time per year” (FS/YR) is set equal to the single computer virtual stoppage rate of step 8 times the failover time that is selected in step S7. A numerical example of step 20 is shown in step 21 wherein the single computer virtual stoppage rate of step 9 and the failover time of step 7 are used.

[0061] Next, in step 22 of FIG. 5B, an estimate is made of the availability of the application programs within the cluster 60 to anyone of the workstations 80. In particular in step 22 of FIG. 5B, the “application availability” (AA) is set equal to 1 minus the sum of two stoppages divided by 8760. Those two stoppages which are summed are the concurrent stoppage per year of steps 18 and 19, and the failover stoppage per year of steps 20 and 21. A numerical example of step 22 is performed in step 23 wherein the concurrent stoppage per year of step 19 and the failover stoppage per year of step 21 are used.

[0062] A preferred method of estimating the availability of an application program which runs on any computer in a cluster of two computers has now been described in detail. In addition, however, various changes and modification can be made to those details without departing from the nature and spirit of the invention.

[0063] For example, the method of estimating the availability of an application program as shown in FIGS. 5A and 5B preferably is embodied in a computer program which is stored on a media that is readable by a personal computer. This is illustrated in FIG. 6 wherein item 120 represents a magnetic disk in which the computer program 130 is stored. Program 130 is read from the magnetic media 120 by a personal computer 140 where it is executed by an instruction processor 140ain an interactive fashion with an operator 150.

[0064] Initially in the program 130, a visual display is generated on a monitor 140b, within the personal computer 140, which requests that the input parameters A-G be manually input by the operator 150. This is indicated by reference numeral 131. Program 130 waits until the input parameters A-G are manually input via the keyboard 140cin the personal computer, or until the program is terminated; and this is indicated by reference numeral 132.

[0065] After the input parameters A-G are received, the program 130 generates the single computer virtual stoppage rate (SCVSR) and the single computer virtual downtime (SCVD) by performing step 8 through step 11 of FIG. 5A as indicated by reference numeral 133. Then program 130 estimates the concurrent stoppage per year (CS/YR) by performing step 12 through step 19 of FIGS. 5A and 5B; and then program 130 estimates the failover stoppage per year (FS/YR) by performing step 20-step 21 of FIG. 5B. This is indicated by reference numerals 134 and 135.

[0066] Next, program 130 determines the application availability (AA) by performing step 22-step 23 of FIG. 5B as is indicated by reference numeral 136. Then, the application availability which is determined in step 136 is displayed by the program 130 on the monitor 140b; and this is indicated by reference numeral 137.

[0067] All of the steps 131-137 can be repeated multiple times. Thus the operator 150 can input different values for the input parameters A-G and thereby see how a change in the input parameters affects the availability of the application program. This enables the operator 150 to select, by trial-and-error, a set of input parameters A-G which will result in a FIG. 1 system that provides a particular application availability which is desired.

[0068] As another modification, information which is an important factor to the availability of an application program can be obtained by performing only a subset of the steps in FIGS. 5A and 5B. For example, the probability of concurrent stoppage (PCS) can be obtained by performing only step 1 thru step 6 and step 8 thru step 17. This PCs is particularly useful because it indicates the risk that none of the computers in the FIG. 1 system will be operable.

[0069] As another modification, the process of FIGS. 5A and 5B can be changed such that it applies to a data processing system which is the same as shown in FIG. 1, except that the cluster 60 includes three computers. Initially, all requests on the network 70 to use the application programs are handled by a first one of the three computers. This mode of operation continues until a stoppage occurs in the first computer. Then a failover state is entered temporarily wherein the application programs are unavailable while the responsibility for handling requests to use the application programs is passed to a second one of the three computers. Thereafter, all requests to use the application programs are actually serviced by the second computer; and that mode of operation continues until a stoppage occurs in the second computer. Then a failover state is entered temporarily wherein the application programs are unavailable while the responsibility for handling requests to use the application programs is passed to the third computer. Thereafter, all requests to use an application program are actually serviced by the third computer. Subsequently when a stoppage occurs in the third computer, a failover state is temporarily entered only if either the first or second computer is operable; and from that failover state, the sequence continues as described above. Otherwise, if all three computers stop concurrently, then all requests to use the application programs will fail until one of the computers becomes operable.

[0070] To account for this modification in the process of FIGS. 5A and 5B, only step 16 needs to be changed. Specifically, step 16 is replaced with step 16′ as shown in FIG. 5B. There, the probability of concurrent stoppage of all three of the computers is estimated as 6(R)³, where R is the ratio as determined by step 14.

[0071] In like manner, the process of FIGS. 5A and 5B can be generalized to apply to a cluster of N computers, where N is 2, 3, 4, etc. To account for this modification, only step 16 needs to be replaced with step 16″ as shown in FIG. 5B. There, the probability of concurrent stoppage of all N of the computers is estimated as (N!)(R)^N, where R is the ratio as determined by step 14 and (N!) is N factorial.

[0072] As another modification, a less accurate estimation of the application availability (AA) can be obtained by setting some of the parameters “A”-“F” which are input in step 1-step 7, equal to zero. For example, the average number of system administrator stops per year in step 5 and the average time duration of each system administrator stop in step 6, can be set to zero. Likewise, a less accurate estimation of the probability of concurrent stoppage in step 16 can be obtained in the same fashion.

[0073] Accordingly, it is to be understood that the present invention encompasses all such modifications and is defined by the appended claims.