Scheme to Perform Event Rollup

1. Technical Field

The present invention relates in general to cluster multiprocessing systems and in particular to event handling within cluster multiprocessing systems. Still more particularly, the present invention relates to rollup of queued events within a high availability cluster multi-processing system.

2. Description of the Related Art

High availability (HA) is gaining widespread commercial acceptance as an alternative to fault tolerance for mission-critical computing platforms. Fault tolerant data processing systems rely on specialized hardware to detect hardware faults and switch to a redundant hardware component, regardless of whether the component is a processor, memory board, hard disk drive, adapter, power supply, etc. While providing seamless cutover and uninterrupted performance, fault tolerant systems are expensive due to the redundant hardware requirement. Additionally, fault tolerant systems do not address software errors, a more common source of data processing system failure.

High availability utilizes standard hardware, but provides software allowing resources to be shared system wide. When a node, component, or application fails, an alternative path to the desired resource is quickly established. The brief interruption required to re-establish availability of the resource is acceptable in many situations. The hardware costs are significantly less than fault tolerant systems, and backup facilities may be utilized during normal operation.

Highly available systems are often implemented as clustered multiprocessor (CMP) systems. A cluster includes a plurality of nodes or processors connected to shared resources, such as shared external hard disks. Typically, each node runs a server or "back end" application permitting access to the shared resources. A node may "own" a set of resources--disks, volume groups, file systems, networks, networks addresses and/or applications--as long as that node is available. When that node goes down, access to the resources is provided through a different node.

Within clustered multiprocessing systems, it is advantageous to provide an event rollup function. In highly available clusters, various events may occur, including node failure, adapter failure, application failure, etc. Processing these events typically requires coordinated multiphase actions across the cluster, with barriers in between phases. However, certain events may be "rolled up" or subsumed within other events. That is, a first event may require only actions which are a subset of the actions required by a second event. Thus, occurrence of the second event while a response to the first event is pending obviates the need for specifically responding to the first event, since responding to the second event achieves the desired result.

Rolling up events may substantially reduce the overhead of event processing, particularly in large clusters. At least one prior art cluster software package--HACMP for AIX.RTM., available from International Business Machines Corporation of Armonk, N.Y.--provides some event rollup capabilities. However, only a limited set of events are rolled up: adapter failures are rolled up into node failures. Moreover, the rollup function is hardcoded into the cluster software and may not be specified or changed by a user. The event rollup information may not be altered by a user in any manner, much less dynamically (changing the event rollup information without having to stop and then restart cluster services).

It would be desirable, therefore, to provide cluster software with facilities permitting dynamic specification or alteration of event rollup information by a user.

It is therefore one object of the present invention to provide an improved cluster multiprocessing system.

It is another object of the present invention to provide improved event handling within cluster multiprocessing systems.

It is yet another object of the present invention to provide dynamically modifiable rollup of queued events within a high availability cluster multiprocessing system according to user-defined information.

The foregoing objects are achieved as is now described. A user definable set of event rollup relationships are maintained as a configuration element. Transitive closure of event rollup relationships is checked at the time of specification to detect cycles and prevent runtime errors. When an event to be processed is detected, the event is compared to defined rollup relationships and queued events to determined if the detected event may be rolled up into at least one queued event or vice versa. If the detected event may be rolled up into a queued event, the detected event is not queued. When a queued event may be rolled up into the detected event, the queued event is deleted if processing has not already begun.

The above as well as additional objects, features, and advantages of the present invention will become apparent in the following detailed written description.

The novel features believed characteristic of the invention are set forth in the appended claims. The invention itself however, as well as a preferred mode of use, further objects and advantages thereof, will best be understood by reference to the following detailed description of an illustrative embodiment when read in conjunction with the accompanying drawings, wherein:

FIG. 1 depicts a diagram of a cluster multiprocessing system in which a preferred embodiment of the present invention may be implemented;

FIG. 2 is a high level flow chart for a process of handling an event queue in accordance with a preferred embodiment of the present invention;

FIG. 3 depicts a high level flowchart for a process of dynamically changing user defined event rollup relationships in accordance with a preferred embodiment of the present invention; and

FIG. 4 is a high level flowchart for a process of resolving event rollup choices in accordance with a preferred embodiment of the present invention.

With reference now to the figures, and in particular with reference to FIG. 1, a diagram of a cluster multiprocessing system in which a preferred embodiment of the present invention may be implemented is depicted. System 102 includes a plurality of server nodes 104-110, each typically identified by a unique name. Each node 104-110 may be a symmetric multiprocessor (SMP) data processing system such as a RISC System/6000.RTM. system available from International Business Machines Corporation of Armonk, N.Y. or a data processing system functioning as a Windows NT.TM. server.

Each node 104-110 within system 102 includes an operating system, such as the Advanced Interactive Executive (AIX.RTM.) operating system available from International Business Machines Corporation of Armonk, N.Y. or the Windows NT.TM. operating system available from Microsoft Corporation of Redmond, Wash. Nodes 104-110 within system 102 also include high availability cluster software capable of running on top of or in conjunction with the operating system. This high availability cluster software includes the features described below.

Nodes 104-110 are connected to public local area networks 112-114, which may be an Ethernet, Token-Ring, fiber distributed data interface (FDDI), or other network. Public networks 112-114 provide clients 116-120 with access to servers 104-110. Clients 116-120 are data processing systems which may access, each running a "front end" or client application which queries server applications running on nodes 104-110.

Typically, each node 104-110 runs server or "back end" applications which access data on shared external disks 122-126 via disk buses 128-130. Nodes 104-110 may also be connected by an additional network 132 or networks. For example, a private network may provide point-to-point connection between nodes 104-110 within system 102, with no access provided to clients 116-120. The private network, if available, may be utilized for lock traffic, and may be an Ethernet, Token-Ring, FDDI, or serial optical channel connector (SOCC) network. A serial network may also provide point-to-point communication between nodes 104-110, used for control messages and heartbeat traffic in the event that an alternative subsystem fails.

As depicted in the exemplary embodiment, system 102 may include some level of redundancy to eliminate single points of failure. For example, each node 104-110 may be connected to each public network 112-114 by two network adapters (not shown): a service adapter providing the primary active connection between a node and network and a standby adapter which substitutes for the service adapter in the event that the service adapter fails. Thus, when a resource within system 102 becomes unavailable, alternative resources may be quickly substituted for the failed resource.

Those of ordinary skill in the art will appreciate that the hardware depicted in the exemplary embodiment of FIG. 1 may vary. For example, a system may include more or fewer nodes, additional clients, and/or other connections not shown. Additionally, the present invention may be implemented within any software which collects and processes events. Systems providing high availability are merely employed for the purposes of illustrating and explaining the invention.

Referring to FIG. 2, a high level flowchart for a process of handling an event queue in accordance with a preferred embodiment of the present invention is illustrated. The event queue in the exemplary embodiment is a replicated event queue maintained by a coordination component of the high availability cluster software. The coordination component is a distributed entity having a daemon running on each node within the cluster. The coordination component subscribes to other components of the high availability cluster software such as a component for handling node and adapter failures, a component for handling forced failovers by a system administrator, and/or a component for handling application server failures. The coordination component is notified by other components of events which have occurred through an API, collects the events, and determines which of the events should be enqueued for processing.

A wide variety of events may occur within a cluster multiprocessing system, most of which may be classified as a failure event, a recovery or reintegration event, or a dynamic reconfiguration event. Some specific examples of events which may occur include a node joining the cluster, a node leaving the cluster, a network failure, a network being connected to the cluster, replacement of a failed network adapter with a standby adapter, or a forced failure by a systems administrator.

When an event occurs, the event may be queued for processing. During event processing, a recovery program designated to handle the event is executed. The recovery program consists of distributed actions with intervening barriers between each action pair within the sequence of actions. The distributed actions are recovery statements which are executed in parallel on different nodes within the cluster. For example, when a network adapter fails and a standby adapter must be substituted, a recovery program statement swapping the standby adapter's network address with the failed service adapter's network address is executed. Optional barriers between statements allow the one statement to be completed on all nodes within the cluster before a subsequent statement is begun on any node within the cluster. For example, if an application server is to be changed to a different node, every node in the cluster must stop utilizing the application server on the current node before the application server is restarted on the new node.

The cluster management software in accordance with a preferred embodiment of the present invention permits users to define event rollup relationships. The event rollup relationships are defined utilizing the event names, such as "node.sub.-- down" and "adapter.sub.-- down", and are maintained in a list associated with the high availability software coordination component. For the purposes of generality, however, the notation "e1->e2" will be employed to indicate definition of an event rollup relationship where event e1 may be rolled up into event e2.

A user may wish to specify event rollup relationships for a variety of reasons. For example, where a network adapter failure is detected within a node having the highest priority for a given resource, and the node containing the network adapter has also failed, it may be desirable to simply recover from the failed node, which may obviate any need for switching to a standby adapter. Alternatively, if the standby adapter is located within the failed node, the user may not wish to attempt to restart an application using the standby adapter, which may be unreliable.

For highly available cluster systems, user defined event rollup relationships may be stored in the cluster configuration component. A user may specify a rollup relationship between any pair of events as part of the cluster configuration. The event rollup relationship definitions may take the form of:

or

Events may be defined to roll up into events which are themselves rolled up into other events. For example, an "application.sub.-- down" event may be rolled up into an "adapter.sub.-- down" event, and the "adapter.sub.-- down" event may itself be rolled up into a "node.sub.-- down" event. The event rollup relationships defined may be dynamically changed at runtime without rebooting the cluster.

Turning to FIG. 2, the process for handing an event queue subject to customized event rollup relationships begins at step 202, which depicts detection of an event to be processed. The process then passes to step 204, which illustrates checking the current rollup relationships defined for the cluster and the cluster event queue. The process next passes to step 206, which depicts a determination of whether the detected event may be rolled up into at least one queued event. If so, the process simply passes to step 216, which illustrates the process becoming idle until a subsequent event is detected. No further action is required. The detected event need not be queued since it may be rolled up into an event already queued. Since transitive closure of event rollup relationships was determined at configuration and cycles were eliminated as described below, no determination of whether a queued event may be rolled up into the detected event should be necessary.

Referring again to step 206, if the detected event may not be rolled up into a queued event, the process proceeds instead to step 208, which depicts queuing the detected event. The process next passes to step 210, which illustrates a determination of whether any queued event may be rolled up into the detected event. Multiple queued events may be identified in which these queued events may be rolled up into the detected event. In such a circumstance, all such queued events which may be rolled up into the detected event are treated identically.

If at least one queued event may be rolled up into the detected event, the process proceeds to step 212, which depicts a determination of whether the queued event which may be rolled up into the detected event is being processed. If not, the process passes to step 214, which illustrates deleting the queued event which may be rolled up into the detected event from the queue.

If no queued event may be rolled up into the detected event, the process proceeds from step 210 to step 216, described above. Similarly, where a queued event which may be rolled up into the detected event is already being processed, the process proceeds from step 212 to step 216. Hybrid situations may occur where one queued event which may be rolled up into the detected event is being processed and must be left alone, while one or more other queued events which may be rolled up into the detected event are not yet being processed and may be deleted.

With reference now to FIG. 3, a high level flowchart for a process of dynamically changing user defined event rollup relationships in accordance with a preferred embodiment of the present invention is depicted. The process relates to a user interface allowing deletion of existing rollup relationships and definition of new rollup relationships. The process begins at step 302, which depicts initiation of a change in the defined event rollup relationships. The process then passes to step 304, which illustrates copying the current event rollup relationships, allowing the current rollup definitions to be utilized while changes are made or attempted.

The process next passes to step 306, which depicts receiving a user-defined event rollup relationship. The process next passes to step 308, which illustrates a determination of whether the newly received event rollup relationship conflicts with existing event rollup relationships. Since any event may be defined as being rolled up into a number of other events, the conflict determination focuses on obtaining transitive closure to detect cycles. Transitive closure of rollup relationship definitions and detection of cycles are both meaningful for successfully maintaining a user-modifiable list of defined event rollup relationships. For example, if the rollup relationships e1->e2 and e2->e3 already exist, the rollup relationship e3->e1 cannot be permitted. The transitive closure algorithm and its use in the present invention is discussed in further detail below.

If a conflict exists, the process passes from step 308 to step 310, which depicts notifying the user that the newly-defined event rollup relationship cannot coexist with existing rollup relationships. The user must delete or otherwise modify existing rollup relationships in order to define the relationship for which a conflict was detected.

Referring again to step 308, if no conflict is detected, the process proceeds instead to step 312, which illustrates integrating the newly-defined event rollup relationship into the defined event rollup relationships. The process next passes to step 314, which depicts swapping the current event rollup relationships with the modified event rollup relationships, allowing the newly-defined relationship to be utilized. The process then passes to step 316, which illustrates the process becoming idle until another event rollup relationship definition is initiated by a user.

Referring to FIG. 4, a high level flowchart for a process of resolving event rollup choices in accordance with a preferred embodiment of the present invention is illustrated. The process depicted obtains transitive closure and detects cycles, and is similar to algorithms used for detecting deadlocks in distributed systems. The process begins at step 402, which depicts receiving an event rollup relationship definition. The rollup relationship definition may be characterized as including a subject event and a target event, where the notation e4->e5 indicates that subject event e4 may be rolled up into target event e5.

The process then passes to step 404, which illustrates comparing the received definition to existing rollup relationship definitions. The process next passes to step 406, which depicts a determination of whether the target event for one rollup relationship definition ("A") is also the subject of at least one other definition ("B"). The target event of the received definition is compared to the subject event of each existing definition. For example, if the definition e4->e5 were received, the existing rollup relationship definitions would be checked to determine if event e5 is the subject event of an existing definition. The target events of existing definitions are also compared to the subject event of the received definition. Thus, in the example described, existing definitions will be checked for event e4 as a target event.

If the received definition contains a target event which is also the subject of another existing definition, or contains a subject event which is the target of another definition, the process proceeds to step 408, which illustrates generating a transitive relationship definition. In the above example where the definition e4->e5 was received, if the existing definitions include the relationship definition e5->e6, the transitive definition e4 ->e6 would be generated. Similarly, if existing definitions include the relationship e6->e4, the transitive relationship definition e6->e5 would be generated. Multiple transitive relationship definitions may be generated, since both the target and subject events of a received definition may be part of existing definitions, and since either a target or subject event may form a part of multiple existing definitions. All possible transitive definitions created by the received definition are determined and added to the list of rollup relationship definitions.

The process then passes to step 410, which depicts a determination of whether a cycle is created by the received definition. For example, where the definitions e5->e6 and e6->e4 already exist, a received definition of e4->e5would create a cyclical relationship definition. The generation of transitive relationships simplifies the detection of cycles since, once all possible transitive definitions have been determined, all definitions--existing, received, and transitive--may be compared to determine if two exist in which the target of one is the subject of the other and vice versa. Cycles can create runtime errors as a result of events being rolled up into themselves. For example, if a particular application server goes down, an "application.sub.-- down" event may be enqueued. Before the "application.sub.-- down" event is processed by all nodes in the cluster, however, the application server may be brought back up and go down again, resulting in a second "application.sub.-- down" event. If the second "application.sub.-- down" event is rolled up into the first, some nodes within the cluster may not properly recover.

If no cycles are created by the received definition, the process proceeds to step 412, which illustrates saving the received definition and all generated transitive definitions. If a cycle is created, the process proceeds instead to step 414, which illustrates notifying the user of the conflict and discarding the received and generated transitive definitions. From either step 412 or 414, or from step 406 if the received definition has no transitive relationship to existing definitions, the process passes to step 416, which illustrates the process becoming idle until another event rollup relationship definition is received.

The present invention supports rollup among an arbitrary number of events within a cluster. User specified event rollup information, which may be dynamically changed, is utilized, permitting customization of event rollup relationships. Transitive closure of event rollup information is automatically generated whenever an event rollup relationship is specified or changed, allowing cycles to be detected at configuration time and avoiding runtime errors.

It is important to note that while the present invention has been described in the context of a fully functional data processing system, those skilled in the art will appreciate that the mechanism of the present invention is capable of being distributed in the form of a computer readable medium of instructions in a variety of forms, and that the present invention applies equally regardless of the particular type of signal bearing media used to actually carry out the distribution. Examples of computer readable media include: recordable type media such as floppy disks and CD-ROMs and transmission type media such as digital and analog communication links.

While the invention has been particularly shown and described with reference to a preferred embodiment, it will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the invention.