Replicated resource management system for managing resources in a distributed application and maintaining a relativistic view of state

[0002] The present invention relates to a system for controlling and coordinating distributed applications and for maintaining a relativistic view of state which can be used, for example, to insure a consistent view of resources in a distributed computing function.

[0003] Distributed processing has many different forms and embodies many different techniques depending on the nature of the data and the objectives of a given application. Typical objectives include: transaction performance, locality of data, minimization of network traffic, high availability, minimization of storage requirements, extensibility, cost to produce, etc. Many applications have certain objectives of overriding significance which dictate the distributed processing technique employed.

[0004] There is one type of application for which the overriding concern is high availability (resiliency). This type includes switching systems used in a communications network, as well as control systems in the areas of avionics, industrial control, and stock trading. In these systems, it is assumed that any component may fail and the system must continue to run in the event of such failure. These systems are designed to be “fault-tolerant” usually by sacrificing many other objectives (e.g. cost, performance, flexibility and storage) in order to achieve high availability.

[0005] Traditionally, fault-tolerant systems have been built as tightly coupled systems with specialized hardware and software components all directed toward achieving high availability. It would thus be desirable to provide a more generally applicable technique for achieving high availability without reliance on specialized components.

[0006] Another important factor for a distributed application is the need to know the relative state of a plurality of peer processes before certain actions occur. Such coordination of effort requires that each process know not only what it thinks about the state of the other processes, but also what the other processes think about its state. This is called a “relativistic” view of process state.

[0007] For example, assume a system has been developed that uses four cooperating processes to perform a task. Also assume that all four processes must be active before the coordinated task can begin; an active process is defined as a process that has successfully contacted its peers. On system start-up, each process must contact all other processes. After contacting its peers, each process must wait until all other processes have contacted their peers. Thus, only after all processes have contacted their peers, and all the peers know of this contact status, can the task begin.

[0008] This is actually a very common situation when attempting to coordinate a task among several processes. It would be desirable to provide a mechanism for gathering a relativistic view of state among various processes.

[0009] In one aspect, the present invention is a method for constructing high-availability distributed applications. This method, referred to herein as “Replicated Resource Management” (RRM), provides a core set of services which enable the replication of data (e.g., state) information in a plurality of processes and optionally the sharing of workload by those processes. This core set of service enables synchronization of each new RRM process with existing RRM processes, recovery in the event of a process level or other failure, and an application level interface (to the RRM) that enables easy and effective use of RRM services in a distributed application.

[0010] RRM utilizes a distributed memory space, which an RRM process may examine at will. An RRM process may join and leave a distributed computing function, also at will. The shared memory space persists across process boundaries so long as at least one RRM process remains (or a persistent store is employed).

[0011] An analogy may be drawn between RRM and a long-running meeting or plurality of meetings in different rooms. People may come and go from a given meeting, and the meeting may continue indefinitely. As people enter a meeting, they are given information that allows them to participate in the meeting. As they leave, the meeting continues, using information they may have supplied. As long as at least one person remains in the room, the meeting can be said to be continuing (in session). If the last person leaves the room, all information gathered and distributed in the meeting may be recorded, so that an interested party may retrieve this information at a later date.

[0012] In the above analogy, the meeting room is analogous to a shared memory space or resource pool. The people participating in the meeting are analogous to RRM processes, and the information exchanged is analogous to RRM resources. This is illustrated in FIG. 1, where each RRM process has an associated resource manager (RM) which identifies several resource pools of interest, i.e., several meetings which are of interest to the person. In this way, a resource manager (process) participates in just its areas (pools) of interest, maintains a registry of resources in these associated pools, and includes mechanisms for communicating with other resource managers (processes) in order to maintain a consistent view of the state of the resources in the associated pools.

[0013] In another aspect, the invention provides a mechanism which allows a distributed application to develop a “relativistic view” of the state of its processes. More generally, the mechanism may be used to provide a relativistic view of any process or object. In the following discussion, we refer to objects instead of processes. The invention provides three procedures for defining a relativistic view of state, which may be used separately or together.

[0014] First, a state vector is provided. This is a one-dimensional associative array of logical object name to logical object state. The name is an “index” into the vector, and the state is stored in a “slot” associated with the index. A state vector is generated by an object and describes what that object thinks is the state of all objects in the vector. It is the object's view of itself and “relative” view of all other objects. A state vector is illustrated in FIG. 4.

[0015] Second, a state matrix (see FIG. 5) is a two-dimensional associative array composed of state vectors. Rows and columns of the matrix are indexed by logical object name; the intersection of a row and column defines a “slot”. A row describes what an object thinks is the state of all objects in the row (this is the same as the state vector described above), while a column describes what all objects think is the state of a single object. General rules are provided for determining whether each of the state vectors and/or state matrices is determinant. For example, a determinant contact status may be achieved when every active resource manager agrees on the present state of all other active resource managers.

[0016] Third, application specific logic (ASL—i.e., a set of rules specific to an application) may be applied to the general rules associated with state vectors and state matrices for providing application specific state vectors (ASSVs) and application specific state matrices (ASSMs), respectively. Because state vectors and state matrices are objects, ASL may be added to them by derivation. As illustrated in FIG. 6, a zero, cardinality relationship (o, m) indicates that a state matrix is composed of zero or many state vectors. This relationship is inherited by the application specific state vectors and application specific state matrices. Again, by way of example, the ASSVs and ASSMs may be used to determine the contact status of processes in a distributed application.

[0017] These and other features of the present invention will be more fully understood by the following detailed description and drawings.

[0018] FIG. 1 is a schematic system level overview of the replicated resource management (RRM) system of this invention;

[0019] FIG. 2 is a schematic illustration of communication between resource managers RM1-RM4;

[0020] FIG. 3 is a schematic architectural view of hosts A and B, each having their own processes and resource managers, but sharing a distributed memory space containing a number of resource pools;

[0021] FIG. 4 is an example of a state vector;

[0022] FIG. 5 is an example of a state matrix comprising a plurality of state vectors;

[0023] FIG. 6 is a schematic illustration showing derivation of application specific state vectors (ASSV's) and application specific state matrices (ASSM's);

[0024] FIG. 7 is an example of a state matrix for a distributed application, showing the contact status of its component processes;

[0025] FIG. 8 is a flow chart of a method for updating a state matrix; and

[0026] FIG. 9 is a schematic illustration of a general purpose computer for implementing the invention.

[0027] RRM Overview

[0028] A glossary of terms is included at the end of the detailed description. A particular replicated resource management (RRM) system described herein is implemented in the C++ object-oriented programming language.

[0029] FIG. 1 is a system level overview of the major RRM components. A plurality of resource managers (10, 11, 12, 13, 14) are provided, each having an associated set of resource pools (16, 17, 18). A distributed memory space (24) exists across the network for the duration of an RRM processing session, and is segmented by the resource pools (16, 17, 18). Each resource manager (RM): participates in the distribution and synchronization of the resource objects (20) in its associated pools; includes a central registry of resources in its associated pools; and contains mechanisms for communicating with other resource managers. As described hereinafter, a resource manager specifies its pools of interest to all other active resource managers, after which time it will receive all of the resource objects in the pools of interest.

[0030] A resource pool (16, 17, 18) is a distributed object whose purpose is to collect and organize a coherent set of distributed resources. Resource pools are embodied through a set of one or more resource managers on one or more host systems (see FIG. 3).

[0031] A resource object (20) has attributes (data) that are significant to the overall system. Resource objects are contained in resource pools and like the resource pools, are distributed objects. Resource objects exist in all the same resource managers as their containing resource pool. Resources are sub-classed by users of a system in order to provide application level object information, i.e., users derive their objects from a resource class. In this way, the user's information is replicated across all applicable resource managers.

[0032] Applications run on top of the RRM system (i.e., they use the RRM's services). Several different applications may view the same resource pools at any given time. This allows information to be shared between applications performing different tasks on, or with, the resource objects maintained by the RRM system.

[0033] FIG. 2 illustrates four resource managers (30-33), exchanging information in order to maintain a coherent set of distributed resources. This exchange of information will be further described below in regard to the maintenance of a relativistic view of state between objects.

[0034] FIG. 3 illustrates a distribution of RRM components in a communications network. For example, host A (40) has a plurality of processes designated as Process A1 (42) and Process A2 (43). Each process contains a resource manager (44 and 45 respectively). The processes A1 and A2 each have their own pools of interest, designated by arrows directed to shared resource pools (50-51). The pools, designated as resource pools X and Y, each contain one or more resources (52-55), designated as X1 through XN, and Y1 through YN, respectively. As shown in FIG. 3, Process A1 is interested in Resource Pool X; Process A2 is interested in both of Resource Pools X and Y. A second host B (60) contains its own set of processes, designated as Process B1 (62) and Process B2 (63), with their own resource managers (64, 65). Process B1 is interested in Resource Pool X, while Process B2 is interested in both of Resource Pools X and Y.

[0035] The following is a more detailed description of RRM operation and services, with subheadings provided for ease of understanding.

[0036] RRM Services

[0037] RRM provides many services—some of which are transparent to applications, while others are used by applications.

[0038] Internal services are transparent to the applications, i.e., the applications see the effects of these services but do not need or want to access them directly. Some of the more important transparent services may include:

[0039] process and network status (state) determination

[0040] message and object transport

[0041] object-level message delivery

[0042] automatic object and state replication

[0043] RRM synchronization and constancy maintenance

[0044] basic fail/over capability

[0045] External (application level) services are used by applications directly through the RRM's application interface (API). The major services may include:

[0046] distributed object-level messaging and remote procedure calls (RPCs)

[0047] object-level contact established/lost notification

[0048] distributed object-level attribute changes and attribute change notification

[0049] hooks for application level object ownership arbitration and ownership change

[0050] Each application that wishes to use the RRM will:

[0051] include the RRM subsystem;

[0052] initialize the RRM subsystem with:

[0053] its peers (i.e., other processes this RRM should communicate with); and

[0054] the resource pools this application is interested in.

[0055] The processes that compose an RRM system may be started in a completely asynchronous manner, that is, they may be started in any order and at any time. Therefore, it is important that each resource manager handles or avoids all potentially destructive/dangerous start-up interactions.

[0056] When an RRM process starts, it attempts to contact all other RRM processes it has been configured to know about. If no other RRM processes exist in the network, the RRM process starts itself in stand-alone mode. This allows an application to run nearly as quickly as it would run without the RRM capability.

[0057] If an RRM process determines that it is running in a network with other RRM processes, it establishes a dialog (connection) with the other processes in the network. During this discovery process, each RRM process learns about the relative state of all other RRM processes in the network. This is done through a resource manager state vector exchange protocol, i.e., the use of state matrices filled by the vectors received at each resource manager (see subsequent discussion). Once a stable (deterministic) network state has been achieved (all active RRM processes are reporting the same state information about each other), synchronization of resource pools is initiated.

[0058] Similar to synchronization of peer processes, initialization also involves synchronization of the resource pools in which a given application has an interest. This is described in greater detail below under “Consistency and Synchronization.”

[0059] Once the RRM subsystem is initialized with the above information, its services may be used by the application. At this point, the application may do the following:

[0060] examine/use the resource objects in the resource pools

[0061] instantiate new resource objects in the resource pools

[0062] delete resource objects from the resource pools

[0063] change the attributes of resource objects

[0064] receive attribute change notifications for resource objects

[0065] use object-level messaging and object-level remote procedure calls

[0066] receive contact lost/established notifications

[0067] RRM application level services are provided almost entirely through the resource object base class. Thus, an application that wants to use the RRM will need to derive its distributable/managed objects (resources) from the RRM's resource object.

[0068] When using the RRM, an application developer needs to decide what objects/information should be distributed to peer applications. This decision can involve many complex trade-offs between overall system performance, resiliency, and application level requirements.

[0069] RRM performance can be difficult to quantify as it is greatly influenced by application level design and constraints. However, performance within the RRM is affected by two major factors:

[0070] distribution and synchronization of state

[0071] parallelism of processing in RRM processes

[0072] RRM processes maintain a distributed, shared memory space. As such, each resource object's state information is accessible within each participating RRM process. This “sharing of state” comes with a cost, both in terms of maintaining state synchronization information and in actually passing state information between RRM processes.

[0073] This cost can be offset through the parallelism inherent in RRM processes. Since several RRM processes can use the shared state information simultaneously, and can communicate through object level messaging and remote procedure calls, the cost of state maintenance is offset through the distribution of processing load. Carefully designed applications can use the facilities of the RRM to enhance performance, while simultaneously gaining resiliency.

[0074] Resiliency, which can be one of the major benefits of RRM, is gained through the sharing of state (resource objects) and the concept of resource object ownership. Each RRM process “owns” certain resource objects. Object ownership denotes what process is responsible for performing all processing for the “owned” object. In short, the owner of an object does all work associated with that object. When a process fails, all objects owned by the failed process must have their ownership changed to processes that are still participating in the distributed computing function. This is done automatically by the RRM (see “Failure and Recovery” below).

[0075] Applications that wish to be resilient must decide what state information needs to be distributed so that they may recover in the event of a process failure. Consider the following example: a network switch control system consisting of one active switch controller and one hot-standby controller. In the event of failure of the first switch controller, the hot-standby takes control of the switches managed by the first controller. The goal of this system is to provide virtually uninterrupted network service in the event of switch controller failure.

[0076] Varying degrees of resiliency and network service can be supplied by this system. If a very high availability system is desired, the following state information may be distributed:

[0077] the network's physical topology (switches, edge devices, and links); and

[0078] all connections present in the network.

[0079] Given this information, the hot-standby controller could non-disruptively take control of the switched network in the event of controller failure. However, several other levels of service may be offered depending on the type and amount of state information being distributed. At least two other schemes could be employed:

[0080] propagate no resource objects, but use the RRM's process level failure detection capabilities; or

[0081] propagate topology information by creating resource objects that represent the network topology.

[0082] The first option would provide resiliency through process level failure detection, but would cause major disruptions in network availability. In this case, the switch controller that takes over the network will have no information about the current state of the network. Thus, before it could take control of the network it would have to discover or confirm the current physical topology of the network, and place all switches in a known state (or discover all connections in the network). All of these operations would take time during which network service would be disrupted.

[0083] The second option provides resiliency through process level failure detection and maintains the current physical topology of the network. Thus, the discovery/confirmation stage listed above is not required. However, all switches must still be placed in a known state before network service can be restored.

[0084] Performance and resiliency are just two application level requirements that RRM can positively affect. The RRM may also be used to:

[0085] simplify the development of client-server (or full peer-to-peer) applications;

[0086] provide a distributed, shared object environment.

[0087] For example, groupware applications such as a distributed white board could be implemented using the RRM. In this case, the distributed, shared object is the white board. In this application, several clients may see and manipulate the white board simultaneously across the network. The RRM provides the infrastructure enabling the sharing of objects in this application.

[0088] Interprocess Communications (IPCs)

[0089] The RRM processes must communicate with each other to perform the RRM function. RRM interprocess communications (IPC) may take several forms based on the underlying system services, but all RRM-IPCs are message based (i.e., use message passing). Some system services that may be used for RRM-IPCs include:

[0090] shared memory

[0091] pipes

[0092] sockets Each of these communication mechanisms exhibit slightly different behaviors and uses different semantics. Thus, the RRM must have a message-based isolation layer that sits above the underlying system services. This isolation layer provides a consistent IPC interface to the higher layer RRM functions.

[0093] By way of example, TCP sockets (integrated by the TCP Server module) can be used for guaranteed data delivery. UDP sockets can be used for non-critical data delivery and messaging. See D. Comer, “Internetworking with TCP/IP,” Vol. 1, 2nd ed (1991), for a discussion of the TCP and UDP protocols. A TCP Server is a communications entity that uses TCP sockets (stream based, connection oriented, guaranteed data delivery) to emulate UDP sockets (packet based, connectionless, nonguaranteed data delivery). The resulting service is a packet based, connectionless, guaranteed data delivery service that provides several higher layer functions as well.

[0094] The TCP Server delivers data as packets through a connectionless interface. It also provides notification (through callbacks) of server status (contact established/lost), and data received to registered clients. Both of these functions are required by the RRM, that is, the RRM must know what processes are participating/able to participate, and it must know when data is received.

[0095] Thus, a TCP Server can provide three major services to the RRM:

[0096] guaranteed data/message delivery (IPCs);

[0097] notification when data is received; and

[0098] notification of server status changes (contact established/lost).

[0099] Alternatively, these functions may be provided by another protocol.

[0100] Detection of Process Level Failures

[0101] The RRM should detect process level failures in a timely and reliable manner. Each IPC mechanism used by the RRM may also be used to detect process level failures. To increase reliability and timeliness of process failure detection, several IPC mechanisms may be used in parallel. For example, both the TCP Server protocol and a UDP Keep Alive protocol (based on hello/heartbeat protocols) can be used for process level failure detections.

[0102] Distributed Shared Memory Model

[0103] Objects managed by the RRM can be thought of as being stored in a distributed memory space which exists across the network. Each RRM process “sees” this memory space and can access any objects in the distributed memory. As RRM processes come and go, the distributed memory remains (until there are no longer any RRM processes). Thus, the distributed memory is persistent across both process (time) and location (space) boundaries.

[0104] Two major aspects of the distributed memory space are:

[0105] its organization (memory models); and

[0106] the identification of objects within the memory space.

[0107] For example, with a “flat” memory model a resource manager would see all objects in existence in the memory space. This has the benefit of complete visibility into all the system's objects. However, it has disadvantages in that all RMs must maintain the state of all objects (utilizing excessive memory), and the manageability of objects is reduced (there is no knowledge or identification of object functionality and no way to organize objects according to their applications).

[0108] Thus, the present invention uses memory “pools” which allow the distributed memory space to be segmented. This permits resource managers (RMs) to only look at certain “pools of interest”, thereby reducing the memory requirements of RMs that are only interested in a subset of the memory space. The use of memory pools also allows certain application specific memory spaces (pools) to be defined. For example, in a virtual network switching system a pool of active call objects may be defined. Any application wishing to view all the active calls in the network may then join this pool only, not the entire distributed memory space. Finally, by creating a global, default pool for all applications, a flat memory model can be supported by using the default pool (this is the degenerate case).

[0109] Every object in the distributed memory space must be uniquely identified. When an object is created in a normal application, this can be accomplished through the use of a memory pointer. However, in a distributed shared memory this is no longer possible (because several applications may use the same value for the memory pointer within their local memory space). Thus, objects in the distributed memory space are identified by unique (across time and space) object identifiers (OIDs). In the present embodiment, every object is given a network unique OID when it is created. Once an object is created, its OID never changes. Uniqueness of OIDs must be maintained across process boundaries for the duration of the existence of the distributed memory space. For example, OIDs can take the following form:

[0110] PoolOID.ServiceAddress.InstanceID

[0111] where:

[0112] PoolOID is the name of the memory pool which contains the resource object

[0113] ServiceAddress uniquely identifies the process within the network that created the object, e.g., for IP sockets, ServiceAddress may take the form Host Name: Port Number

[0114] InstanceID is a unique ID identifying an instance of an object under the prefix given by PoolOID.ServiceAddress

[0115] An object in the distributed memory space is accessed by de-referencing its OID. This is equivalent to de-referencing a normal memory pointer. Thus, objects in the distributed memory can be treated as if they are part of local, system memory.

[0116] Object-level Messaging (OLM)

[0117] Because RRM processes can come and go at will, RRM must support location-independent, object-level messaging. That is, given an object's OID, an application using the RRM services must be able to send a message to that object without knowing:

[0118] how many replicas of the object exist; and

[0119] where (in what processes) the object exists. In contrast, if each application was required to maintain location information for replicated objects (along with pools, etc.), application programming in the RRM environment would become extremely complex.

[0120] Thus, when a message is sent to an object, the application program should not need to know anything more than the object's OID. The RRM Service delivers the message to the object appropriately. All object-level messages should preferably be sent and received in a non-blocking, asynchronous manner.

[0121] Location-independent, object-level remote procedure calls (ORPCs) allow an object's methods to be called (no matter where the object resides) and the results of each method call to be returned to the calling process. The calling process does not need to know where (by what process) the ORPC will be serviced.

[0122] ORPCs are completely asynchronous. Specifically, a thread in a calling process may wait for the response to an ORPC, but the entire calling process does not block waiting for an ORPC. Also, many ORPCs may be outstanding at any point in time (many threads may be blocked waiting for ORPCs). If a process failure is detected, all outstanding ORPCs that have been sent to that process are cleared and then retried (sent to the new owner).

[0123] Currently there are competing standards for the classical functional Remote Procedure Call (RPC) mechanism. Two camps exist, Open Network Computing (ONC) (supported by AT&T, Sun, Novel, and Netwise) and the enhanced Network Computing System (NCS) (from HP/Apollo with backers IBM, DEC, Apollo, and Microsoft).

[0124] Two major standards for functional RPC's emerged from the above camps, ONC ROC and NCS RPC. The NCS RPC mechanism was selected by the Open Software Foundation (OSF) as the basis for RPCs in its Distributed Computing Environment (DCE). Because of this, the NCS RPC mechanism is commonly known as DCE RPC while the ONC RPC mechanism is commonly known as Sun RPC. See “Power Programming with RPC,” by John Bloomer, O'Reilly & Associates, Inc., Copyright 1992.

[0125] The RPC mechanism described in this invention extends the classical functional RPC mechanism by making it both object-oriented and fault-tolerant.

[0126] Object Ownership

[0127] Work flow within the RRM system is controlled through object “ownership”. Each RRM process may “own” zero or many resource objects. Each resource object has one, and only one, “owner” at any given time. The process that “owns” an object is responsible for doing all processing, or work, associated with that object.

[0128] Work flow within the RRM system may be localized, distributed, or balanced based on object ownership. Algorithms for each of these types of behaviors can be defined externally by application level processing without changing the core RRM functionality. The RRM provides basic object ownership behavior as well as hooks for object ownership change.

[0129] When an object is created, its default owner is the process in which it was created. Over time, a resource's owner may change—its OID is never changed, rather it is the “value” of the ownership property which is changed when an object's ownership is assigned to another process.

[0130] When an RRM process fails or terminates, all the resource objects it owned must have their ownership changed to one of the remaining RRM processes. This may be accomplished by providing a default ownership arbitration algorithm in the base resource object class. The default algorithm may be overridden by derived, application level classes as desired.

[0131] Upon RRM process failure, every resource object in all the pools of interest must be notified of the RRM failure. Each resource object must then decide if it was owned by the failed RRM. If it was, it must call its ownership arbitration method to determine the identity of its new owner. Since the same algorithm is run by every resource object in all processes, and all RRM processes maintain a consistent view of all participating RRMs, the same new owner is determined in all locations (in parallel).

[0132] Resource object ownership may be changed explicitly by the owning resource manager process. This may be done to redistribute processing load or to allow a resource manager to leave a pool of interest without leaving its objects unattended.

[0133] State Replication Management

[0134] The RRM must manage and coordinate the replication of state between RRM processes. All state is replicated at the object level, e.g., the level of granularity for state replication is an object's attributes.

[0135] State replication is coordinated at both the memory space (resource pool) and the process (resource manager) levels.

[0136] There are two major components of state replication:

[0137] replication of the object itself (replication of the construction of the object); and

[0138] replication of the object's attributes.

[0139] Replication of an object's construction occurs when an object is first created within a resource pool, and when a resource pool is synchronized. Replication of an object's attributes occurs incrementally as the attributes' state changes. Both of these replication functions require an object transport function.

[0140] The RRM provides a means by which software objects are transported between processes. The object transport function is able to take any “supported” software object and convert it to a transportable (distributable) representation that may be sent to and received from any RRM processes. Supported software objects are objects that can be distributed (streamed).

[0141] Thus, the object transport function obtains an object's distributable representation from that object and creates an object from its distributable representation; this is accomplished by an object packetization service, also known as streaming and unstreaming, or packing and unpacking. A second function of the object transport function is to send and receive an object's distributable representation; this is supported by using the RM's IPC layer.

[0142] An object's distributable representation must include the object's type, attributes and their values. If an object's attribute is a reference to another object, and referential integrity must be maintained, the reference must be to a distributable object, and the reference must be distributed as a reference object. Reference attributes are resolved at the receiving side of the transport function so that referential integrity between objects is maintained.

[0143] As an object's attribute changes, these changes are propagated to all of the RRM processes containing that object. Attribute changes are controlled at the object level through a method, i.e., “writeattr” for each attribute change. Attribute changes are sent to other processes using a “commit” method. Each of these operations are handled at the object level (they are methods of the resource object). Attribute change notification occurs through the “attribute-changed” virtual method.

[0144] Synchronization

[0145] Several levels of synchronization (consistency checks) occur within the RRM. The highest is done at the process level to ensure that all RRM processes are being coordinated and that they have a consistent view of both the distributed memory and the state of the network (the processes they are communicating with).

[0146] Process level consistency checks may include:

[0147] determining the state of all RRM processes in the network (initial, contact established, contact loss) from the perspective of each RRM process;

[0148] ensuring that every active RRM process has the same view of the state of all other RRM processes; and

[0149] ensuring that each RRM process has a consistent, synchronized view of the distributed memory in which it is interested.

[0150] Process level synchronization is achieved when:

[0151] a consistent view of the state of all processes in the network is achieved; and

[0152] all pools of interest for this RRM process have been synchronized (see memory level synchronization below).

[0153] When an RRM process enters a distributed computing function it must send a join pool request to all active RRM processes for each resource pool it is interested in. Each one of the active processes must respond with either a positive acknowledgment (meaning it is also interested in the pool specified), or a negative acknowledgment (it is not interested). Each of the positively responding processes must then send all of the objects it is responsible for (“owns”) in the pool to the process joining the pool (when a pure in memory replication scheme is used). If the process joining the pool already has objects in that pool that it is responsible for, it must send those objects to each of the processes that responded positively. In this way, each pool is synchronized in both directions for every process that is interested in the pool.

[0154] A separate level of synchronization and consistency checks occurs at the memory level. This ensures that: (1) a consistent view of memory pool existence and RRM process attachment is maintained (i.e., what processes have joined what pools); (2) each pool's objects are distributed and maintained consistently; and (3) the relative state of pool synchronization is distributed and checked.

[0155] Memory level consistency checks include:

[0156] agreement as to which RRM processes are interested in each pool; and

[0157] relativistic synchronization of each pool.

[0158] Memory level synchronization includes:

[0159] distribution of all objects created within a pool to all RRM processes interested in (joined to) the pool (synchronization of object existence); and

[0160] bidirectional pool synchronization and agreement upon the pool's synchronization state between all RRM processes joined to each pool.

[0161] Memory level synchronization is achieved when:

[0162] every resource pool an RRM process is interested in is synchronized with respect to the given RRM process; a resource pool is synchronized with respect to a given RRM process when all other RRM processes have sent their owned objects to the given RRM process and the given RRM process has sent its owned objects to all other interested RRM processes.

[0163] The final level of synchronization occurs at the object level. This synchronization takes two forms, object creation synchronization and object state synchronization. Object existence is synchronized by the resource pools when an object is created in a resource pool. Upon object creation, all interested RRM processes are notified of the object's creation when the object arrives at each interested RRM process.

[0164] An object's state is synchronized as its attributes are modified (written to). In this case, all instances of an object are updated as the object's state changes. See the previous discussion of attribute change management, i.e., the “write_attr( )” and “commit( )” methods.

[0165] Failure and Recovery

[0166] All RRM process failures are detected by the remaining RRM processes as they occur. Recovery involves rearbitration of object ownership by the remaining RRM processes. Failed processes may rejoin the distributed computing function after failure.

[0167] Determination of which process is responsible for each object is done through object ownership arbitration. Each process must block all other threads that may be executed as it determines the new owner for each affected object. Once all affected objects have a new owner, normal processing continues. Thus, ownership change and transfer of control is automatic within a process.

[0168] Arbitration of object ownership allows the remaining RRM processes to continue the distributed processing function in the event of catastrophic process level failures. Once an RRM process fails, it may try to reenter the distributed computing function by rejoining its pools of interest. This is handled exactly the same way as if the RRM process had entered the network for the first time (see process level synchronization previously described).

[0169] Relativistic State

[0170] In the particular embodiment described herein, the present invention provides a mechanism that allows a distributed application to develop a relativistic view of the state of its processes. This relativistic view may then be used for several purposes, including, but not limited to:

[0171] the coordination of tasks;

[0172] determining the validity of information obtained from other processes;

[0173] fault detection and isolation;

[0174] voting and agreement paradigms.

[0175] However, this invention is intended to be a general purpose distributed computing solution and thus may be used to provide a relativistic view of the state of any object.

[0176] Three mechanisms are provided for defining a relativistic view of the state of processes/objects: 1) state vector (SV); 2) state matrix (SM); and 3) application specific (AS) logic for providing AS state vectors and AS state matrices. The following discussion will refer to objects, but is understood to be applicable to processes as well.

[0177] 1. State Vector

[0178] A state vector is a one-dimensional associative array of logical object name to logical object state. Each name is an “index” into the vector, and each state is stored in a “slot” associated with an index. A state vector is generated by an object and describes what that object thinks the state of all objects in the vector is. It is the object's view of itself and “relative” view of all other objects.

[0179] FIG. 4 illustrates a state vector 70 generated by Object A. In this example, Object A thinks itself to be in State 1 and thinks Object B is in State n, Object C is in State 3, and Object D is in State 2. This vector may be extended indefinitely.

[0180] The following general rules may apply to any State Vector:

[0181] The state of a State Vector is defined to be “determinant” if and only if each slot in the vector has an equivalent logical state; that is, all state values across the vector are logically equivalent. If a vector's state is determinant its logical state as a whole is equivalent to the state of any slot. Finally, any subset of the vector (degenerating to single logical name-value pair) is in turn a State Vector.

[0182] 2. State Matrix

[0183] A State Matrix is a two-dimensional associative array composed of State Vectors. Rows and columns of the State Matrix are indexed by logical object names. Rows in this matrix describe what each object thinks is the state of all other objects in the row (this is the same as the State Vector described above). Columns in this matrix describe what all objects think is the state of a single object.

[0184] FIG. 5 illustrates a state matrix 80 generated by the combination of State Vectors for objects A through D. In this case, each object has provided a State Vector (row) to the matrix. The column vector for Object A indicates that Object A thinks itself to be in State 1, Object B thinks object A is in State 3, Object C thinks Object A is in State n, and Object D thinks Object A is in State 3. Once again, this matrix may be extended indefinitely.

[0185] The following general rules may apply to any State Matrix:

[0186] A State Matrix is “row determinant” if and only if each of its Row State Vectors are determinant. A State Matrix is “column determinant” if and only if each of its Column State Vectors are determinant. A State Matrix is “determinant” if and only if it is both row determinant and column determinant. When a State Matrix is determinant its state as a whole is equivalent to the state of any one of its slots. Finally, any subset of row and column logical names (indexes) of the matrix (degenerating to single row/column logical name) and their associated slot values in turn defines a State Matrix.

[0187] In alternative embodiments, the general rules for a state matrix may require it to be: 1) row determinant; 2) column determinant; 3) matrix determinant; or 4) any combination of 1)-3).

[0188] 3. Application Specific Logic (ASL)

[0189] The preceding sections described the general rules associated with State Vectors and State Matrices. These general rules provide the foundation for organizing and understanding relativistic state information. Application Specific Logic (ASL) is a set of rules specific to an application that provides meaning beyond that of the basic State Vectors and State Matrices.

[0190] Because State Vectors and State Matrices are objects, ASL may be added to them through specialization (derivation). FIG. 6 illustrates the derivation of Application Specific State Vectors (ASSVs) 90 and Application Specific State Matrices (ASSMs) 91 from the base State Vector (SV) 92 and State Matrix (SM) 93 objects, respectively. In this diagram the (O,m) cardinality relationship indicates that the State Matrix is composed of zero or many State Vectors. Note that this relationship is inherited by ASSVs and ASSMs.

[0191] 4. Example of ASL Usage

[0192] The following example, illustrated in FIG. 7, demonstrates the usage of ASSVs and ASSMs to determine the contact status of processes in a distributed application. In this example Contact Status Vector and Contact Status Matrix objects have been derived from the original SV and SM objects and ASL has been added.

[0193] Consider a distributed application that may run with one or many processes (1 through n) at any given time. On start up, each process initializes its internal Contact Status Matrix (CSM) 96 and attempts to contact a set of other processes. A CSM is indexed by logical process name and has the following possible state values:

[0194] INITIAL—no information about the process has been obtained yet

[0195] ESTB—contact with the process has been established

[0196] LOST—the process either can not be contacted or contact with a previously contacted process has been lost

[0197] Every time the contact status of a process changes (to either ESTB or LOST) from the perspective of a process, that process sends a Contact Status Vector to all other processes with which it has established contact. Upon receiving a Contact Status Vector from a process each process places this vector in its CSM at the row indexed by the sending process' logical name. In this way each process is able to build up a complete relativistic view of process state.

[0198] Application Specific Logic is then used to determine what this process state means. In the case of CSMs, determinism is defined by the following rules:

[0199] A CSM for a process is determinant when:

[0200] 1) the process's row vector does not contain any INITIAL states, and

[0201] 2) the Contact Status Vectors obtained by:

[0202] a) copying the column Contact Status Vectors of the CSM,

[0203] b) each column vector of the CSM (excluding column vectors for processes that are LOST from the perspective of the present process) is determinant.

[0204] Basically, the first rule ensures that the process attempts to contact all other processes before trying to determine if the CSM is determinant. The second rule excludes any processes from consideration if they are thought to be lost (the assumption is that lost processes can not provide valid data). Thus, as shown in FIG. 7, the CSM for process 1 is determinant.

[0205] 5. Updating State Information

[0206] FIG. 8 is a flowchart of a method for updating state information 100. Starting at Step 101, a particular system (e.g., process 1) updates its own contact status vector (Step 102). Next, process 1 contacts the other systems (Step 103). Further action depends upon whether the contact is successful (Step 104); if not successful, process 1 updates its state vector with “lost” for the noncontacted system, e.g. process 2 (Step 105). If contact was successful, process 1 updates its state vector with ESTB for the contacted system, e.g., process 3 (Step 106). In either case, process 1 then sends out a copy of its state vector to the other systems, e.g., processes 2-n (Step 107). Then, process 1 continues to contact other systems (Step 108); when all processes have been attempted to be contacted, the method ends (Step 109).

[0207] Applications

[0208] The RRM has many areas of potential application. For example, in a switched communications network having a distributed call management system, a resource manager can be provided for each of a plurality of distributed call management processes. See for example the switched network described in copending and commonly owned U.S. Ser. No. 08/188,238 filed Jan. 28, 1994 by K. Dobbins et al., hereby incorporated by reference in its entirety. A distributed memory space can be divided into separate pools for each of the following services: topology, directory, policy and call. A first benefit of incorporating RRM in this system is that every process is provided with a consistent view of the network. For example, every process can have access to a policy pool which may specify, for example, that node-a is permitted to communicate with node-b. As a further example, when a node in the network topology changes, it is important that every switch be notified of the change. Using RRM, state changes are seen everywhere so that every switch has the same view of the network topology. A second benefit is fault tolerance; for example, because the call pool has an object for every call, if one call management process goes down, another call management process can take over the calls in the call pool previously owned by the failed process; this can be done quickly with essentially no interruption in service. A third benefit is load sharing. The various call management processes can distribute ownership of calls in order to balance the workload. A fourth benefit is scalability; additional call processes can be added to the system easily without requiring reprogramming of the prior systems. Yet another important benefit from a programming perspective is that the programmer does not need to know where any given method or change of object is executed, only that the result is the same.

[0209] Another potential use of RRM is for distributed text-based conferencing. For example, each of three users at different locations can pull up a common screen on their computers with, for example, three windows, each window representing a separate conversation (a pool). The contents of a given conversation are shared buffer objects (resources). Each user would be able to input information into a buffer (screen), virtually at the same time. The process owning the shared buffer would act as a “gatekeeper” to the window, arbitrating inputs from the three users. Every character input into the buffer object is in essence a state change, which is arbitrated and if permitted appears substantially instantaneously on each of the three user screens.

[0210] The arbitration logic decides what other users can do to the object. For example, if a first user (first process) needs to use an object owned by a second user (second process), the first user sends a request to the second process which will perform the method and update the object. The state change then goes automatically to every process, not just the requesting process.

[0211] In contrast, in the prior art it was necessary to replicate a database, table, or list, on an individual basis. It was not possible to get the most recent data unless one contacted a central repository, which quickly became a bottleneck and/or was a potential point of failure. In contrast, in the present embodiment every state change goes to every interested process, not just the requesting process. Also, because the data is distributed, there is no bottleneck and no one point of failure.

[0212] The embodiment described herein assumes a communications network is modelled in memory, such as cache memory in a network manager. The network manager may be a Sun workstation sold by Sun Gateway 2000 with a Sun Solaris Microsoft Windows NT operating system; the network management software may be the Spectrum™ Network Management software sold by Cabletron Systems, Inc., Rochester, N.H., USA and described in U.S. Pat. No. 5,261,044 by R. Dev et al., which is hereby incorporated by reference in its entirety. Typically, the network manager resides in a host and the resources of a network provide alerts to the host which are received and interpreted by the network manager in accordance with known techniques. The resources of a network may include a plurality of individual workstations, personal computers, modems, applications resident in host computers, communication satellites, and the like. In a modelled network, a change made by an application to one resource of the network, which affects the other resources of the networks, will be immediately known and accounted for since the model of the network within the data cache manager accounts for these inter-relationships.

[0213] The Spectrum™ software and the software of the present invention may be written in an object-oriented programming language such as C++. In an object-oriented approach, data and methods (logic in the form of executable code) are combined to form objects. The objects are logically grouped by classes and superclasses in a tree (hierarchy) structure, wherein an object inherits structure from its class, i.e., objects in the same class share the same data organization. An object can represent a single resource of a process being controlled; in a network, an object can represent processors, applications which control communication and data flow within a network, switches, storage devices, printers, routers, modems, etc.

[0214] The program of the present invention may be implemented in a general purpose computer 141 such as shown in FIG. 9. As can be seen, the general purpose computer includes a computer processing unit (CPU) 142, memory 143, a processing bus 144 by which the CPU can access the memory, and access to a network 145. This invention does not require the network topology to be modeled; can be used in many applications.

[0215] Glossary of Terms

[0216] Resource Manager (RM)—A central component of the Replicated Resource Management (RRM) system—it provides a registry of Resource Objects and a mechanism for communicating with other Resource Managers.

[0217] Object Identifier (OID)—A network unique logical identifier for an object. OIDs maintain uniqueness across both time (process) and space (pool) boundaries for the life-span of an object.

[0218] Service Address—An abstract address which uniquely identifies an RRM process in a distributed computing function. In IP networks, it is formed by a (Host-Name, port number) tuple. Each Resource Manager has a service address.

[0219] Service OID—Each Resource Manager process is assigned a network unique identifier. In IP nets, it is generated by combining the globally unique Host-Name with the specific port number used by that Resource Manager process.

[0220] Resource Pool (RP)—A distributed object whose purpose is to collect and organize a coherent set of distributed resources. It can be thought of as a shared memory space. Resource Pools are embodied through a set of one or more Resource Managers on one or more host systems. Resource pools are used to segment the overall distributed memory space so as to limit the distribution of objects.

[0221] Pool OID—Each Resource Pool is assigned a network unique identifier.

[0222] Resource Object—A software object (as in object-oriented programming) whose data may be significant to the overall system. Resource objects are contained within Resource Pools and, like the Resource Pools, are distributed objects. They exist in all Resource Managers associated with their containing Resource Pool. Resource objects are sub-classed by users of the system so as to provide application level object information (i.e., users derive their objects from the resource class). In this way, the users' information is replicated across all applicable Resource Managers.

[0223] Resource OID—Each resource is assigned an OID at creation time. A Resource OID may be logically constructed as follows: PoolOID.ServiceOID.InstanceID (where InstanceID is a unique logical value given PoolOID.ServiceOID). This logical naming allows a resource to exist in a pool across process boundaries without OID conflicts.

[0224] Pools of Interest—Each Resource Manager has a set of associated resource pools in which it is interested. The Resource Manager will specify its Pools of Interest, after which time it will receive all the Resource Objects in its associated Pools of Interest.

[0225] Resource Ownership—A Resource Object is owned by a single Resource Manager at any point in time. It is this Resource Manager (or the application code running within its context) that is responsible for maintaining the state of the Resource Object and distributing any changes to the other participating Resource Managers (interested in the Resource Pool containing that Resource Object). Resource ownership may change at any time through the following circumstances: 1) the current owner reassigns ownership to another Resource Manager; or 2) the current owner leaves the network (i.e., fails). In the event of Resource Manager failure, all of its owned resources become unowned and may be claimed by any remaining Resource Manager. Resource specific arbitration logic is responsible for determining which Resource Manager will claim the orphaned resources. The Resource Manager claiming ownership then notifies the application in which it is running that the resource ownership has changed. A default ownership arbiter is provided to evenly distribute resources among remaining Resource Managers.

[0226] Resource Synchronization—Only the process with the owning Resource Manager may modify the contents of a Resource Object. A resource Object's contents may be modified using a ‘write_attr( )’ method. When a set of changes is complete, a ‘commit( )’ method causes the changes to be synchronized with all other Resource Managers interested in the pool containing the changed Resource Object.

[0227] RRM supports two different synchronization modes. The first, entitled, “Foreground Synchronization,” does not return from the commit( ) method until all Resource Managers in the resource's pool have confirmed reception and processing of the synchronization. The second, entitled, “Background Synchronization,” returns as soon as the synchronization messages have been sent to the associated Resource Managers. Foreground Synchronization may be used for resources with zero-tolerance for incomplete synchronization. Background Synchronization may be a default mode of operation—it yields much better transaction performance with only a small window for mis-synchronization. In the event of failure of Foreground Synchronization, a transaction.backout( ) method is available for removing the changes made to the Resource Object.

[0228] Resource Attributes—RRM supports two main types of attributes: value attributes and reference attributes. Value attributes include basic data types: integer, float, string, etc., as well as composites of basic data types (e.g., tuples, lists and dictionaries). Value attributes are synchronized by transferring the actual value to the associated (interested) Resource Managers. Reference attributes are references to other Resource Objects. Reference attributes are synchronized by transferring the OID of the referenced Resource Object to the interested Resource Managers.

[0229] Synchronization—Synchronization may occur on one or more of three levels: Resource Objects; Resource Pools; Resource Managers (from lowest to highest). For example, synchronization on a Resource Pool level occurs when a Resource Manager joins a pool (either explicitly or implicitly). A pool is joined “explicitly” through a join pool message sent from the Resource Manager requesting to join the pool. “Implicit” joins occur when a Pool Status Vector (relative to that pool) indicates that a new Resource Manager is interested in that pool. Pool Synchronization is determined through the use of a Pool Status Matrix (see below).

[0230] Initialization—Initialization occurs when a Resource Manager object is instantiated. The initialization method does not return until the new RM has established contact with all available RMs and each of those RMs has synchronized with the new RM. Therefore, after creation of a Resource Manager, an application can be ensured that it has a local, current copy of all the Resource Objects in each of its Pools of Interest.

[0231] Failure Detection—Failure detection is provided by the communication layer. For example, each Resource Manager (RM) may use a TCP Server object to communicate with other RMs. Whenever the process or system in which an RM is running terminates, the local TCP Server (or a UDP Keep Alive Module) notifies its RM that it has lost contact with a given Service Address. Once the RM has confirmed a Determinant Contact Status (see below), the RM notifies all of its Resource Pools that the corresponding Service Address has gone down and synchronization to that RM is suspended. Any Resource Objects within affected pools (pools that the failed RM was interested in) are notified of the process failure.

[0232] Contact Status Vector—The Contact Status Vector is an associative array of Resource Manager Service Addresses to contact status (state) mappings. Whenever the local contact status of an RM changes, the changed RM sends a Contact Status Vector to all active RMs to notify them of the state change. Upon reception of a Contact Status Vector, each RM fills in its Contact Status Matrix and attempts to contact any unrecognized RMs in the vector.

[0233] Contact Status Matrix—Each Resource Manager (RM) maintains a Contact Status Matrix. Each row in this matrix is a Contact Status Vector representing the contact status as seen by each RM in the network. In short, the matrix maintains a complete relativistic view of the contact status of the RMs.

[0234] Determinant Contact Status—Determinant Contact Status is achieved when all active RMs (from the local RM's viewpoint) agree on the present state of all other RMs (i.e., each active column of the Contact Status Matrix has a determinant state, that is, every slot in the column has the same state value).

[0235] Pool Status Vector—A Pool Status Vector is an associative array of Resource Manager Service Addresses to pool status (state) mappings. Whenever the local synchronization status of a Resource Pool (RP) changes, the changed RP sends a Pool Status Vector to all interested RMs to notify them of the pool's state change. Upon reception of a Pool Status Vector, each RM forwards the message to the appropriate RP. The receiving RP fills in its Pool Status Matrix and synchronizes with any unrecognized RMs in the vector (Implicit Synchronization).

[0236] Pool Status Matrix—Each Resource Pool (RP) maintains a Pool Status Matrix. Each row in this matrix is a Pool Status Vector representing the Pool Status as seen by each RP in the network. In short, the matrix maintains a complete relativistic view of the pool status of the RPs.

[0237] Pool Synchronization—A Resource Pool is synchronized when all slots in its Pool Status Matrix indicate a synchronization complete state, that is, when all RMs have synchronized with the Resource Pool.

[0238] While there have been shown and described several embodiments of the present invention, it will be obvious to those skilled in the art that various changes and modifications may be made therein without departing from the scope of the invention as defined by the appending claims.