High availability asynchronous computer system

[0002] This is a continuation of application No. 08/996,171 filed Dec. 22, 1997 now U.S. Pat. No. 6,185,662, the contents of which are incorporated by reference herein.

[0003] The present invention relates to a high availability asynchronous computer system and is particularly concerned with a system employing redundant, shared memory.

[0004] The present invention relates to a high availability or fault tolerant memory system for a computer system. Such a computer system may be employed in telecommunications network switches, where high availability and fault tolerance is required.

[0005] In such memory systems, memory is often duplicated in shadow or mirror shared memory cards so that system performance is not halted or degraded by an error or fault in one shared memory card. It may be necessary in these systems to discover when a shared memory card has an error or fault and to diagnose the error or fault. As well, the systems need the capability to add or remove redundant shared memory cards, in order to enhance fault tolerance or eliminate excessive redundancy.

[0006] A common difficulty with systems having redundant shared memory is keeping track of how many redundant copies of data are stored and allowing a variable number of redundant copies to be stored. Many systems require a fixed number of responses to all memory requests. This reduces system flexibility and fault tolerance.

[0007] U.S. Pat. No. 4,995,040 issued Feb. 19, 1991 to Best et al. discloses a fault tolerant, fail-safe computer system including a management unit and buffer to manage asynchronous redundant digital messages, to sort the redundant messages and to compare redundant messages. The messages are compared by a majority voting scheme. The system disclosed by Best may provide high integrity but is unlikely to provide high availability or high reliability.

[0008] An object of the present invention is to provide a high availability asynchronous memory system. A benefit of the present invention is that it allows the mirror set of redundant logical memory modules to have a variable number of logical modules. This increases system flexibility and fault tolerance.

[0009] In one aspect there is provided a method for accessing mirrored shared logical memory modules for a read including: (a) sending a memory request for a read to an interface and then to the shared logical memory modules from a requester; (b) formulating a reply at each shared logical memory module indicating whether the shared logical memory module is in a transitional state; (c) sending each reply to the interface; (d) deriving at the interface, from one or more replies, a second message; and (e) forwarding the second message to the requester from the interface.

[0010] In another aspect there is provided a method for accessing mirrored shared logical memory modules for a read including (a) sending a memory request for a read to an interface from a requester and then to the shared logical memory modules; (b) formulating a reply at each shared logical memory module indicating a size of a mirror set, where the mirror set comprises the shared logical memory modules; (c) sending each reply to the interface; (d) deriving at the interface, from one or more replies, a second message; and, (e) forwarding the second message to the requester from the interface.

[0011] In another aspect there is provided a logical shared memory module including means for receiving a memory request from another device; means for determining whether the logical shared memory module is in a transitional state; and means for generating a reply to the other device indicating whether the logical shared memory module is in a transitional state.

[0012] In another aspect there is provided a method for accessing mirrored shared logical memory modules for a write including (a) formulating a memory request for a write containing a data component to be written to a logical memory address; (b) sending the memory request for a write to an interface from a requester; (c) formulating an acceptance message indicating the memory request for a write has been accepted by the interface; (d) sending the acceptance message to the requester from the interface; (e) sending the memory request for a write to the shared logical memory modules from the interface; (f) formulating a reply, at each shared logical memory module, to the memory request for a write indicating whether the shared logical memory module is in a transitional state; and (g) sending each reply to the interface.

[0013] In another aspect there is provided a method for accessing mirrored shared logical memory modules for a write including (a) formulating a memory request for a write containing a data component to be written to a logical memory address; (b) sending the memory request for a M rite to an interface from a requester; (c) formulating an acceptance message indicating the memory request for a write has been accepted by the interface; (d) sending the acceptance message to the requester from the interface; (e) sending the memory request for a write to the shared logical memory modules from the interface; (f) formulating a reply to the memory request for a write, at each shared logical memory module, indicating a size of a mirror set, where a mirror set comprises the shared logical memory modules; and (g) sending each reply to the interface.

[0014] In another aspect there is provided a logical shared memory module comprising: means for receiving a memory request from another device; means for determining whether said logical shared memory module is in a transitional state; and, means for generating a reply to said other device indicating whether said logical shared memory module is in a transitional state.

[0015] In another aspect there is provided a method for removing a shared logical memory module from a mirror set comprising the steps: changing the shared logical memory module to a transitional state; changing the remaining shared logical memory modules to a destination set size; and disabling the shared logical memory module in said transitional state.

[0016] In another aspect there is provided a method for accessing mirrored shared logical memory modules for a write comprising the steps of: (a) formulating a memory request for a write containing a data component to be written to a logical memory address; (b) sending said memory request for a write to an interface from a requester; (c) formulating an acceptance message indicating the memory request for a write has been accepted by said interface; (d) sending said acceptance message to said requester from said interface; (e) sending said memory request for a write to at least one shared logical memory module from said interface; (f) formulating a reply to said memory request for a write indicating whether said shared logical memory module is in a transitional state; and (g) sending a reply in response to said memory request for a write from said shared logical memory module to said interface element.

[0017] In another aspect there is provided a method for accessing mirrored shared logical memory modules for a write comprising the steps of: (a) formulating a memory request for a write containing a data component to be written to a logical memory address; (b) sending said memory request for a write to an interface from a requester; (c) formulating an acceptance message indicating the memory request for a write has been accepted by said interface; (d) sending said acceptance message to said requester from said interface; (e) sending said memory request for a write to at least one shared logical memory module from said interface; (f) formulating a reply to said memory request for a write indicating a mirror set size of the shared logical memory module; and (g) sending said reply in response to said memory request for a write from said shared logical memory module to said interface element.

[0018] The invention will be further understood with the following description with references to the drawings in which

[0019] FIG. 1 is a schematic view of a computer system in accordance with an embodiment of the present invention.

[0020] FIG. 2 contains Table 1 which lists set mode values in accordance with an embodiment the present invention.

[0021] FIG. 3 contains Table 2 which lists legal Mirror Mode Sequences in accordance with an embodiment of the present invention.

[0022] FIG. 4 contains Table 3 which describes a sequence of steps for adding two logical memory modules to a mirror set in accordance with an embodiment of the present invention.

[0023] The preferred embodiment of a fault-tolerant, asynchronous, computer system allows multiple copies of stored memory to be maintained, thus providing a back-up in case of failure of one copy. Such a system is illustrated in FIG. 1. In the preferred embodiment there are four main components: processing elements 102, 104; shared memory cards 128, 129, 132; input/output element 106; and interconnect 108. Processing elements 102, 104 include a processor 110, local memory 112 associated with the processor 110, and a processing element interface chip 114. The processing element interface chip 114 transmits or receives data in the form of frames from the interconnect 108. A frame is a structured message that contains well-defined information in particular locations within the frame. Data may be read from or written to shared memory cards 128, 129, 132 by processing elements or input/output elements. Shared memory cards have a memory interface chip 116, memory card register table 126 and physical memory modules 118, 120, 122, 124. The system shown in FIG. 1 is illustrative only and typically there will be a variable number of processing elements, shared memory cards, and input/output elements. As well, processing elements such as 102 and input/output elements such as 106 function in the same manner in respect of making a memory request, so that when the phrase processing element is used hereafter with respect to making a memory request it should be understood to encompass a memory request by either processing or input/output elements.

[0024] In the preferred embodiment, each shared memory card contains a number of logical memory modules. Each logical memory module has a programmable start and end address range. A single memory card will not have more than one logical memory module with the same address range. The memory card register table 126 of memory card 128 contains a mapping between logical memory address ranges and physical memory addresses used for accessing physical memory modules, such as physical memory module 118.

[0025] When a shared memory card is initially connected to the interconnect 108, the shared memory card is in a disabled state. In the preferred embodiment, the processing element is aware that the shared memory card has been connected to the interconnect 108.

[0026] A memory request is a message from a requester such as processor 110 requesting access of data, such as a read or write, of data that is located at a particular logical memory address. The requester could also be a pre-fetcher or additional pipelines. When memory requests are transmitted onto the interconnect 108, there can be multiple requests and responses occurring simultaneously. There may be overlap and misordering between the requests and responses. A transaction identifier is added by the processing element interface chip 114 to the frame containing the request in order to keep track of requests and responses. Contention between simultaneous requests must be managed. A system providing such contention management is described in U.S. patent application Ser. No. 08/774548 titled “Shared Memory Control Algorithm for Mutual Exclusion and Rollback”, applicants T. E. Newell et al., filed Dec. 30, 1996 which is hereby incorporated by reference for all purposes.

[0027] One aspect of the invention deals with the means by which memory requests are made from a processing to a shared memory card on the interconnect. In the preferred embodiment, the interconnect operates in a point to multipoint fashion. In other words, the processing element that sends out a memory request will be connected to each of the shared memory cards on the interconnect. Thus, it is possible that more than one shared memory card may respond to any given memory request. This is likely to happen, as there is mirror memory set up to help ensure system availability and fault tolerance. A mirror memory set comprises multiple copies of the same data, each copy having the same logical address. The mirror set size is the number of copies of the same data. A mirror set of the memory is set up on the system, so that if there is a fault in one shared memory card, the processing element can refer to other responses to the same memory request in order to attempt to continue processing, with minimal interruption or error, despite the fault.

[0028] The method by which the memory requests are made for a read from a logical shared memory module comprises the following steps:

[0029] Step 1

[0030] The requester such as processor 110 determines that a memory request is needed.

[0031] Step 2

[0032] The requester such as processor 110 signals to the processing element interface chip 114 that a memory request is required.

[0033] Step 3

[0034] The processing element interface chip 114 assembles the request by constructing a frame containing data showing the frame type (such as shared memory request), a transaction identifier, an identifier of the originating processing or input/output element, a logical memory address, a read code, and status flags. Processing element interface chip 114 saves information from the frame for later comparison with responses to this memory request.

[0035] Step 4

[0036] The memory request is placed on the interconnect 108 by the processing element interface chip 114 where it is transmitted to all the shared memory cards 128, 129, 132 on the interconnect 108. When the request is sent out, timers 113 are started in order to detect error conditions such as a failure to reply to the memory request.

[0037] Step 5

[0038] A shared memory interface chip 116 on the shared memory card 128 receives the request. The shared memory interface chip 116 checks the memory card register table 126, to see if a logical memory module 119 with the logical address contained in the request is on shared memory card 128. If the shared memory card 128 does have the logical address, the shared memory interface chip 116 queues the request for execution. This checking is performed at all memory cards connected to interconnect 108. If memory card register table 126 does not have the logical address, the shared memory interface chip 116 discards the request. When the request is executed, a data component is read from the logical memory address. After the request is executed, a reply frame is generated which contains a data component, which is data stored at the logical address specified in the memory request; a destination identifier which is the identifier of the processing element which made the memory request; the transaction identifier from the memory request frame; the set mode of the responding memory module taken from Table 1, wherein the valid values for the set mode in the preferred embodiment are one module, two modules, three modules, one-update, and two-update; the identifier of the shared memory card; and other status codes. The set mode is a type of status code that indicates when a shared memory card is in a transitional state and that also indicates how many responses the originating processing element should expect to receive (the mirror set size). The reply is then transmitted by the shared memory interface chip 116 to the interconnect 108.

[0039] Step 6

[0040] The replies are received by the processing element interface chip 114. As each accepted reply with a particular transaction identifier comes in, it is compared with the previous replies with the same transaction identifier and checked for errors. If the combination of set modes from the responses form a valid set, then all elements of the mirror set have replied and the memory access is complete.

[0041] The processing element interface chip 114 indicates an error if any of the following conditions are detected:

[0042] (a) The timers 113 expire before the replies are received from all members of the set. A timeout initiates an error handler. A timeout is a strong indication that there is a hardware fault with one of the shared memory cards 128, 129, 132.

[0043] (b) Those parts of the replies that are directly compared, such as the data component for the processor, do not match.

[0044] (c) An error indicator is contained in one or more of the memory replies. The errors can include the status of the shared memory cards 128, 129, 132 and other error indicators.

[0045] (d) The processing element interface chip 114 reads the transaction identifier returned as part of the reply, checks it against a table of valid transaction identifier numbers and concludes it is invalid.

[0046] (e) The processing element interface chip 114 checks the set mode and determines it is not a valid mode, as shown in Table 1.

[0047] (f) The combination of set modes from the replies with the same transaction identifier do not form a valid set. Table 2 shows the legal combinations of set modes.

[0048] Step 7

[0049] When the processing element interface chip 114 receives a reply it prepares a second message for forwarding to the requester such as processor 110. The second message is derived from the reply and may contain the data component located at the logical memory address in the memory reply, a status or error code, or some status code derived from the data or status codes or both.

[0050] The processing element interface chip 114 operates in two distinct modes, fast forward and safe mode, which are discussed below.

[0051] In safe mode, the processing element interface chip 114 collects all replies before forwarding the data component to the requester such as processor 110. This prevents the requester such as processor 10 from proceeding until it is certain that no errors were found in step 6.

[0052] In fast forward mode, the data component from the first reply that contains no error and is not in a transitional state is forwarded to the requester such as processor 110 immediately. The processing element interface chip 114 stores the reply, and ensures the set modes from all replies with the same transaction identifier form a valid combination and that there are no other errors.

[0053] The operation in fast forward mode can be significantly faster, but the requester such as processor 110 must be prepared to correct the problem if an error is detected after the data component from the first reply has been forwarded. The correction is controlled by the error handler and may be by roll-back, correction of the memory system if the error was in the memory system, or by withdrawing the incorrect data from the system.

[0054] A benefit of the present invention is that it allows the mirror set to have a variable number of logical modules. This increases system flexibility and fault tolerance.

[0055] The method by which the memory requests for a write are made comprises the following steps:

[0056] Step 1

[0057] The requester such as processor 110 determines that a memory request for a write is needed.

[0058] Step 2

[0059] The requester such as processor 10 signals to the processing element interface chip 114 that a memory request for a write is required.

[0060] Step 3

[0061] Processing element interface chip 114 sends an acceptance message back to requester such as processor 110 that it has accepted the write request.

[0062] Step 4

[0063] The processing element interface chip 114 assembles the memory request by constructing a frame containing data showing the frame type (such as shared memory request), a transaction identifier, an identifier of the originating processing or input/output element, a logical memory address, a write code, and status flags, and the data to be written to the logical memory address. Processing element interface chip 114 saves information from the request frame for later comparison with responses to the memory request.

[0064] Step 5

[0065] The memory request is placed on the interconnect 108 by the processing element interface chip 114 where it is transmitted to all the shared memory cards 128, 129, 132 on the interconnect 108. When the request is sent out, timers 113 are started in order to detect error conditions such as a failure to reply to the memory request.

[0066] Step 6

[0067] A shared memory interface chip 116 on the shared memory card 128 receives the request. The shared memory interface chip 116 checks the memory card register table 126, to see if a logical memory module 119 with the logical address contained in the request is on shared memory card 128. If the shared memory card 128 does have the logical address, the shared memory interface chip 116 queues the request for execution. This checking is performed at all memory cards connected to interconnect 108. If memory card register table 126 does not have the logical address, the shared memory interface chip 116 discards the request. When the request is executed the data is written to the logical memory address. After the request is executed, a reply frame is generated which contains a destination identifier which is the identifier of the processing element which made the memory request; the transaction identifier from the memory request frame; the set mode of the responding memory module taken from Table 1, wherein the valid values for the set mode in the preferred embodiment are one module, two modules, three modules, one-update, and two-update; the identifier of the shared memory card; and other status codes. The set mode is a type of status code that indicates when a shared memory card is in a transitional state and that also indicates how many responses the originating processing element should expect to receive (the mirror set size). The reply is then transmitted by the shared memory interface chip 116 to the interconnect 108.

[0068] Step 7

[0069] The replies are received by the processing element interface chip 114. As each accepted reply with a particular transaction identifier comes in, it is compared with the previous replies with the same transaction identifier and checked for errors. If the combination of set modes from the responses form a valid set, then all elements of the mirror set have replied and the memory access is complete.

[0070] The processing element interface chip 114 indicates an error if any of the following conditions are detected:

[0071] (d) The timers 113 expire before the replies are received from all members of the set. A timeout is a strong indication that there is a hardware fault with one of the shared memory cards 128, 129, 132.

[0072] (e) Those parts of the replies that are directly compared, do not match.

[0073] (f) An error indicator is contained in one or more of the memory replies. The errors can include the status of the shared memory cards 128, 129, 132 and other error indicators.

[0074] (d) The processing element interface chip 114 reads the transaction identifier returned as part of the reply, checks it against a table of valid transaction identifier numbers and concludes it is invalid,

[0075] (e) The processing element interface chip 114 checks the set mode and determines it is not a valid mode, as shown in Table 1.

[0076] (f) The combination of set modes from the replies with the same transaction identifier do not form a valid set. Table 2 shows the legal combinations of set modes.

[0077] If an error is indicated an error handler is initiated.

[0078] Another important aspect of the invention is the ability to add or remove mirror logical memory modules from the mirror set. The ability to add new mirror memory or remove it without impacting system operation is an important characteristic. It is also important to be able to move logical memory modules from one mirror set to another, so that mirror elements are not overly concentrated in a subset of logical addresses of the system. This provides the ability to reconfigure the system to the optimal state of reliability.

[0079] As described above, when the requester such as processor 110 sends out a memory request, it can receive replies from more than one of shared memory cards 128, 129, 132. Between the time when a mirror logical memory module 119 is disabled and not associated with a logical memory address range and the time when it is fully enabled and is properly and correctly mirroring the other logical memory modules, such as logical memory module 123 associated with the same logical memory address range, the logical memory module is in a transitional state. When a logical memory module is in the transitional state, it is necessary that the processing element interface chip 114 receives a response from that logical memory module but disregards the data generated by that logical memory module in response to a memory request. When mirror memory is added to or removed from association with a logical address range, it is necessary to understand or keep track of the status of the logical memory module being added or subtracted, so that the comparison, matching and error checking performed by the processing element interface chip 114 does not inadvertently compare with or attempt to match responses from a logical memory module that is in a transitional state.

[0080] When a logical memory module is added to a mirror set, the following steps occur:

[0081] Step 1

[0082] The set mode of the logical memory module being added is changed to an update state, such as 1-update or 2-update. The set mode of the logical memory module is changed to the 1-update state, when a simplex mirror memory set is being changed to a duplex set or a triplex set, or to the 2-update state, when a logical memory module is being added to an existing duplex set.

[0083] Step 2

[0084] The memory card register table 126 is set up with its logical memory address range of the mirror set. A signal is then generated to take the logical memory module 119 from the disabled state to an enabled state.

[0085] Step 3

[0086] The existing logical module(s) have the set mode changed to the final number of mirror modules in the mirror set. Thus, for a change from simplex to duplex, in other words, changing from one logical memory address range with a mirror set of size one to a mirror set of size two (one redundant), the existing logical module has its set mode changed from 1 to 2; from a simplex to triplex (two redundant), the existing logical module set mode is changed from 1 to 3 and for a duplex to triplex, the existing logical module set mode is changed from 2 to 3.

[0087] Step 4

[0088] The new logical module 119 has a range of logical addresses stored in table 126 which is the logical address range of the mirror set. The processor 110 initiates a read/write back march through that range of logical addresses. After this is complete the memory in the new logical module 119 is consistent with the memory in the existing logical module(s) in the mirror set.

[0089] Step 5

[0090] The set mode of the new logical module 119 is changed to the final number of modules in the mirror memory set.

[0091] It can be noted that a shortcut exists when moving from the simplex to triplex or, in other words, from one logical memory module in the mirror set to three logical modules in the mirror set. Both the new logical modules are added at that same time and, prior to the read/write back march both the new logical modules are in the 1-update state. The read/write back march then updates both the new logical modules at the same time. After the march, the set mode of one and then the other of the new logical modules is changed to the final number of modules, which in this case is three.

[0092] In the preferred embodiment, when moving the mirror set size of the original active module from a mirror set size of one to a mirror set size of three, it is necessary to move through a mirror set size of two to avoid creating a false error condition. This is illustrated in Table 3.

[0093] When removing shared logical memory modules from a mirror set it is necessary for the logical module to pass through a transitional state where its data component is not actively compared to other responses from other logical memory modules in the mirror set to a memory request. This transitional state is required to allow the various logical memory modules in the mirror set to be changed to the destination set size without reporting errors. Once the remaining logical modules have their set mode indicators set to the destination set size the logical module can be removed.

[0094] The preferred embodiment of this invention uses a maximum set size of three. This meets current requirements but the invention is not restricted to a mirror set size of three. The mechanism can be extended to an arbitrarily large mirror set size.

[0095] Extension of the maximum allowed set size to the general case of N modules per mirror set is generally straight forward. The extension to an arbitrarily large mirror set size can be done by following the following set of rules:

[0096] the set mode field must contain:

[0097] N-1 update modes to be included in the mirror mode field labelled 1-U through (N-1)-U according to the previous used naming convention;

[0098] codes for mirror set sizes 1 through N;

[0099] update mode M−U must be used when adding modules to a set size of M (M active cards in the set) to get any other set size, or when removing module from any set size to get to a size of M;

[0100] when adding modules to sets all non-update mode modules must have a mirror set size equal to the destination set size when the copy is performed;

[0101] adding modules to a set to go from set size A to set size B (i.e. B>A) requires B−A steps in preparation for copying data starting with A modules in mode A and B−A modules disabled, each step involves adding one to the set size of each module and enabling one more module in A−U update mode such that after the Mth step there are A modules with a set size of A+M, M modules in M−U update mode and B−A−M modules still disabled, after B−A steps (the point where the system is ready to perform the copy) there are A modules in mirror mode B and B−A modules in A−U update mode and a copy of the module can begin. Once the copy is complete the B−A modules in A−U update mode can be changed to set size B and the operation is complete;

[0102] when the copy is being performed to add modules to sets all active modules must respond with the set mode bits set to the new set size; and

[0103] removing modules to go from a set size of A to a set size of B (i.e. B<A) requires all modules in the set that are to be removed to be placed in update mode B then A−B steps must be performed starting with B modules in mode A and A−B modules in update mode, each step involves reducing the set size by one in each module not in update mode and then disabling one of the modules in update mode, such that after the (A−B) step there are B modules with set size B and A−B modules disabled.

[0104] The error checking used to determine which combinations of mirror set sizes and transitional states are valid is summarized by the following rules:

[0105] the normal running mode requires all shared logical memory modules in the mirror set to have a mirror set size equal to the number of shared logical memory modules in the mirror set;

[0106] a response from a shared logical memory module with the set mode indicating it is in transitional mode is counted as a response but the data is not compared to other responses;

[0107] when a response with set mode in update mode M−U is received, there must be at least M modules responding with set mode set to size M; and in all combinations there must be at least the number of responses coming back as the largest set size.

[0108] By defining the set mode sequences during the transitional states, it is possible to ensure that new logical memory modules can be brought into a state consistent with existing logical memory modules in a controlled and predictable fashion.

[0109] Numerous modifications, variations and adaptations may be made to particular embodiments of the invention described above without departing from the scope of the invention which is defined in the claims. For example, the number of shared memory cards mirroring a logical address space could be greater than three, causing a corresponding change in valid sets of set modes and a corresponding change in the method for adding or removing redundant memory modules.