Network Management Method Using Redundant Distributed Control Processors

1. Field of the Invention

The present invention relates to systems and methods for globally managing distributed data processing systems, and in particular, relates to methods for managing a plurality of processors connected by a communications network.

2. Background of the Invention

Computer solutions are being increasingly implemented in the form of distributed computer processors connected by a communications network. An example of such a network is a series of workstations, such as the IBM Personal System/2 (IBM PS/2) computer or the IBM RISC System/6000 workstation (Personal System/2, PS/2, and RISC System/6000 are trademarks of the IBM Corporation). These workstations may be connected by a local area network (LAN) operating as a token ring, ethernet, or other form of network.

Management and control activities for the network are typically assigned to one of the processing elements on that network. Network control is required to manage global activities impacting the entire network and for maintaining status information. The network manager node or control point performs services in response to requests from client workstations. The control point typically comprises a workstation with processing capabilities and attached files or databases for maintaining control and status information. An example of a network with a service management control point is described in U.S. Pat. No. 5,005,122, issued on Apr. 2, 1991.

The designation of a single workstation node as a network control point can lead to problems if the control point has a hardware or software failure: the network operation will cease until the failure is corrected. One possible solution to this single point of failure is to provide redundant control points. Such redundancy, however, creates its own problems. First, each of the clients must be able to determine which of the candidate control points is currently providing management functions. Second, when the network is started and when workstations are brought on-line, the control point processors must be able to determine their current status as primary or alternate. Finally, the global data used by the control points to manage the network and global activities must be consistent across the primary and alternate processors so that a switch to an alternate control point does not cause corruption of the network activities.

A fault tolerant multiprocessor system is disclosed in U.S. Pat. No. 4,817,091. This fault tolerant system requires periodic transmission messages with the status of each processor on the device. The switch from primary to backup processors occurs when the processor fails to detect a predetermined signal during a fixed time window. Upon failure to detect the signal, the processor itself will initiate steps to assume control of the operation.

A technical problem therefore exists in creating a distributed computer network with redundant control points that is able to automatically manage the designation of primary and alternate control points, the identification of the current primary control point by client processors, and the maintenance of synchronized data between candidate control points.

It is therefore an object of the present invention to provide a system and method for managing a distributed network having a plurality of control point processors. The process and apparatus of the present invention provide facilities for network management and methods and processes for managing the interaction between control point processors. The present invention initializes the control point processors and causes each processor to initiate negotiations with the other control point processors to determine which should be designated as the primary. Negotiation results in the designation of a single primary and one or more alternate control points. The present invention also provides processes for synchronizing the data storage between the control points. Finally, the present invention allows client processors to query each control point in an attempt to locate the current primary control point. Failure of the designated control point to respond to a client request will initiate a process whereby the client notifies an alternate control point of the need for that processor to assume the primary control point responsibilities.

These and other objects of the invention will be described in greater detail with reference to the attached drawings.

FIG. 1 is a block diagram showing a distributed network capable of management by the present invention.

FIG. 2 is a block diagram showing the structure of a control point processing element according to the present invention.

FIG. 3 is a state diagram showing the states which backend processes assume.

FIG. 4 is a schematic diagram illustrating the processing of control point startup.

FIGS. 5(a) and 5(b) are flowcharts depicting the process steps involved and control point startup and negotiation of primary control status.

FIG. 6 is a schematic diagram illustrating the flow of messages during a client request for management action by a primary control point processor.

FIG. 7 is a flowchart depicting the process steps involved in client primary control point determination.

FIG. 8 is a schematic diagram illustrating the flow of messages during initial global data synchronization and synchronization maintenance.

FIG. 9 is a flowchart depicting the process steps of initial synchronization and synchronization maintenance.

A distributed network for use with the system and method of the present invention is shown conceptually in FIG. 1. The network comprises a plurality of processors 102, 104, 106, and 108 connected to a communications network 116. Any number of such processors could be attached depending only on the physical limitations of the network. Each of the processors has data processing capabilities and optionally may have permanent data storage such as disks 112 and 114. The preferred embodiment employs an IBM Token Ring Network management scheme, however, it will be appreciated that any network protocol can be used with the present invention.

The workstations in the network are typically designated as having specific functions. Among the functions to be allocated are control point management processing activities. The control point processor manages global activities (affecting the entire network) and maintains overall status of the network. In the preferred embodiment, redundant control point processors are designated to ensure operation in spite of the failure of one of the control points.

Redundant processors labelled Control Point 1 (CP1), 102, and Control Point 2 (CP2), 104 are shown in FIG. 1. The other network attached processors such as 106 and 108 are termed "client" processors and use the services of control points 102 or 104. As will be discussed in greater detail below, one of the control points will be designated as the primary control point while the others will be designated as alternate control points. More than two control point processors may be provided within the scope of the present invention. The control point processors are collectively referenced as control point candidates.

The structure of a control point processor 102 is shown in greater detail in FIG. 2. A control point processor is configured to process two types of programs. The first set of programs 200 are transaction-type programs that service client requests for service or perform other control point requested functions. A second set of programs 210 are termed "backend" programs. The backend programs process requests from the transaction programs to access or modify control point tables stored on permanent storage 112. One of the specialized backend tasks is monitor 212. Monitor 212 manages the overall backend processing including tracking whether this control point is a primary or alternate control point. Other backend processes perform access and updates to a subset of network data tables. The data required for global management, in the preferred embodiment, is divided into a number of tables with dedicated backend processing. These tables are stored in storage devices such as disks 112 and 114. Division into a number of tables reduces potential conflict during access and update cycles. The backend processes operate in synchronization with transaction processes to manage data exchange.

The backend processes typically transition between three main states as shown in FIG. 3. Upon initialization, the backend process is placed in a "starting" state 301. When the monitor determines that the process backend should be activated, a monitor "start request" 302 is sent causing the backend process to move into a request processing state 304. The backend process remains in this state until the monitor sends a "start remote" command which causes the process to establish a remote as well as local update capability.

A failure in the remote processor will cause the backend task to return to "request process" state 304. After the remote processor is fixed, a "remote start" command can be issued to restart remote processing. A local processing failure causes the failing component to terminate and the alternate control point to request it be made primary.

The backend processing state 301 "starting" ignores any client requests for services and returns a successful completion status to the requester. In state 304 process requests, the backend process handles the required processing and causes updates to be made to the local tables.

As discussed above, the monitor process 212 in the control point 102 determines whether this is the primary or an alternate control point. The determination of status is made initially at startup and can be modified during operation. The process for arbitrating or negotiating control point status will be described with reference to FIG. 4 and the flowchart of FIGS. 5(a) and 5(b). FIG. 4 shows the general structure of two control point processors, each with a monitor program 402 and 404 as well as transaction programs 406, 408 and backend programs 410 and 412. The control point elements would communicate over the network as discussed above. However, for simplicity, FIG. 4 shows a direct connection between the control point processors.

Control point startup initiates the following activities to determine control point status. The following discussion assumes the first control point processor (400 in FIG. 4) is being started with control point 2, 401, also existing on the network. The workstation will initially start monitor program 402. Monitor 402 issues a start to any needed local backends. Local backends may include a configuration controller, a session administrator and a terminal administrator. Once the backends have been started the monitor starts a specialized negotiation backend whose responsibility is determining control point status.

The negotiation program implements the following process for determining which of the control point candidates is primary. The process first creates a named resource X on control point 400. Resource X exists only while it is being accessed by the negotiation backend and returns a signal to any other process attempting to create this resource. If the negotiation process receives a message indicating resource X already exists, the process is terminated with no further action being taken.

Once the resource has been created, negotiation process 410 issues a negotiate request to the other control point 401. If processor 400 is not able to access processor 401, the negotiation backend 410 sends a request to the monitor 402 to become the primary control point.

The second control point processor 401 follows the following process when it receives a negotiate request from another control point (see FIG. 5(b)). Again, this function would be performed by any other candidate control point and processor. Control point processor 401 starts a negotiate program backend 414. This negotiate program attempts to create named resource X on second control point processor 401. If resource X already exists, the negotiate program compares the network address or network name of resource X on processor 401 with that from the first processor 400. In the preferred embodiment, if the network address of second processor 401 is lower than the network address from resource X of first processor 400, then processing continues. Otherwise, a failure message is returned to negotiation backend 410 and the negotiation program exits. Negotiation backend 410 sends a message to monitor 402 to await a "set alternate" request from control processor.

If resource X does not exist on second processor 401, then negotiation program 414 attempts to send monitor 404 of control point 401 a "set alternate" request. If monitor 401 fails to reply, or replies "success", then a success status is returned by negotiation program 414 to the first control point processor 401, negotiation program 411. If monitor 404 replies "failure" then negotiation program 414 returns "failure" to control point 400. Monitor failure 404 would occur if the second control point 401 is already operating as the primary control point. Resource X is held until the first control point 400 confirms the reply from negotiation program 401.

Control processor 400 receives the status from control processor 401. If "success" is received, then the monitor is requested to be set to primary. Otherwise, the monitor is set to alternate. The first processor confirms receipt of reply by the other processor 402 and exits processing.

The result of the negotiation activities is the designation of a single processor as the primary control point. The next problem addressed by the present invention is determination by a client processor of which of the control point candidates is, in fact, the primary control point. As shown in FIG. 6, a client workstation 602 may issue a network request 604 requiring some action by the control processor. Client processor 602 must determine which is the primary control point for servicing of that request. The flowchart in FIG. 7 describes the process of the present invention for determining the primary control point.

Each client processor 602 is provided with a static table listing all candidate control point processors. When the client desires to request service from the primary control point, the list of candidate control points is searched and each is queried for status. The status determination process is initiated by client 602 sending a request 604 to a first control point 606. The control point attempts to start a transaction program 608. If this fails, a failure status is returned to client 602. If successful, the transaction program will attempt to register with control point monitor 610. Registration request 609 sent between transaction program 608 and monitor 610 is an attempt to determine the status of control point 606. The monitor may return a status of "primary" or "not primary". If "primary" is returned, a successful completion code is returned. If no response or a "not primary" response is received, a "not primary" status is returned to client 602. If the transaction program 608 receives a "not primary" return code from the monitor, it will send a message 612 over the network to alternate control processor 614 in an attempt to determine the status of that control processor. If a successful response is returned to control point 606, (i.e. 614 is available as the primary control point) transaction program 608 returns a "not primary" message to the requesting client 602. Otherwise, if the second control point 614 fails to respond or responds with an error, then transaction program 608 issues a request to monitor 610 requesting it to change its status to "primary" control point.

Once a client process 602 connects to a transaction program 608 in a primary control point, the transaction program performs the requested transaction. If the final result 620 returned to client 602 is "not primary", then client 602 will send a request to the next control point in the table with a similar attempt to determine whether or not it is the primary control point. If all control point candidate addresses are processed without success, then no control point is available for client processing.

Global data used to manage the network and activities is contained in tables residing in data files or databases in the control point processors. In the preferred embodiment, the data is managed by a database management system as a set of database tables. Remote data services facilities of the OS/2 operating system are used to implement remote data distribution.

The global tables for network management must be synchronized between candidate control point processors to allow fallback in the case of control point failure. Synchronization is required when the network is initially started and when a control point processor is restarted after a failure. Synchronization activities will be discussed with reference to FIG. 8. The flowchart of FIG. 9 presents the process steps for data synchronization.

When a control point 804 receives a "set alternate" status as the result of control point negotiation, it must synchronize its tables with those of the primary control point 800. It must then request that the primary keep the tables in the alternate control point 804 updated so that it may assume control point responsibilities upon failure of the primary.

Primary control point 800 is assumed to be operating with certain data stored in tables 802. Alternate control point 804 begins processing by sending a "remote starting" message to the primary control point 800. The primary control point will start remote backends 806. It will also send to each local backend 808 a message to "start remote". This causes queues to be established in the control point processor 800 for forwarding data from the local to remote backends.

An initial synchronization program 810 is started in control point 800. This synchronization program accesses data 802 directly without using the services of local backend 808. Synchronization program 810 causes the data from 802 to be copied to the alternate control point 812 database.

As discussed above, subsets of tables are serviced by an individual backend process. The synchronization process requires that the table in the primary control point database 802 be locked so that it can be copied without being subject to corrupting modifications during the copying process. Once locked, the table is copied from the primary control point database 802 to the alternate control point database 812. The synchronization process operates on the alternate control point. Upon successful completion, a "start backend" message is sent to the primary control point causing the remote backends to be started. Upon reply from the primary control point, the tables are unlocked.

The above process steps cause tables 812 to be synchronized with tables 802. To maintain synchronization, primary control point 800 must continually pass changes to its data to the alternate control point 804. Remote backend 806 and the primary control point 800 manage this process by sending updates through the remote backend 814 in alternate control point 804 causing the update of database 812.

The above description has pointed out certain features of the present invention. It will be understood that many alterations or modifications can be made to describe the system and processes without departing from the true spirit of the invention. The invention is intended to be limited only by the following claims.