Shared memory multiprocessor system and method with address translation between partitions and resetting of nodes included in other partitions

[0002] The present application relates to subject matter described in application Ser. No. 09/030,957 filed Feb. 26, 1998 entitled “Shared Memory Multiprocessor”, now U.S. Pat. No. 6,088,770, the disclosure of which is hereby incorporated by reference.

[0003] The present invention relates to a shared memory parallel processor system used in information processing apparatuses, especially used in personal computers (PCs), work stations (WSs), and server machines. In particular, the present invention relates to a control scheme of a shared memory between partitions.

[0004] In recent years, use of the architecture of the shared memory multiprocessor as a host module of the parallel processors has spread. In this architecture, such a configuration where several tens to several hundreds processors share a main memory is needed in some cases in order to improve the performance. As the configuration method of the shared memory multiprocessor, bus connection symmetrical multiprocessors (SMPs) used in personal computers are typical. Since the bus throughput forms a bottleneck in the bus connection SMPs, however, the number of processors which can be connected is limited to approximately four. Thus the bus connection SMP is not suitable for such a scheme as to connect a large number of processors.

[0005] In order to solve the above described problem, there has been proposed a method of connection bus connecting SMPs hierarchically by using a crossbar switch or the like. A typical example of the hierarchical SMP is found in “Gigaplane—XB: Extending the Ultra Enterprise Family”, HOT Interconnects V, pp. 97 to 112, August 1997. The crossbar switch or the like between nodes logically functions as a bus. Coherence of a CPU cache between nodes of the bus connection SMP having processors and main memories can be managed at high speed by using a bus snoop protocol.

[0006] As one of problems of the large scale shared memory multiprocessor as described above, there is reliability. In conventional shared memory multiprocessors, the whole system has one Operation System (OS). Since all processors of the system can be managed by one OS, this scheme has an advantage that flexible system operation (such as load distribution) can be conducted. However, this scheme has a drawback that the system reliability falls in the case where a large number of processors are connected in a shared memory multiprocessor configuration. In a server of a cluster configuration in which a plurality of processors are connected by a network, respective nodes have different OSs. Even if a fatal error such as a bug or the like of an OS or the like occurs, only the corresponding node suffers from a system down state. On the other hand, if a certain processor is brought into a down state by a system bug or the like in the case where the whole system is controlled by one OS in a shared memory multiprocessor, the OS is brought into a down state and consequently all processors are affected.

[0007] In order to solve this problem, there has been proposed such a scheme that the inside of a shared memory multiprocessor is divided into a plurality of partitions and a plurality of OSs are run independently. Each partition has an independent main memory. A processor of a certain partition basically accesses only the main memory of its own partition. As a result, it becomes possible to realize the fault containment between partitions and improve the system performance.

[0008] Furthermore, also for improving the operation performance and reducing the management cost using server consolidation, it is desired to integrate works which have been conducted by a plurality of servers into one highly multiplexed server. The above described partition technique is indispensable.

[0009] In the case where a shared memory multiprocessor is divided into partitions, how communication is conducted between partitions poses a problem. A scheme in which communication between partitions is conducted by making efficient use of a shared memory mechanism provided in a system before partitioning is at advantage in performance. Therefore, realization of a shared memory between partitions becomes necessary.

[0010] A partition technique of making a plurality of OSs run in one system has been used heretofore in mainframes, and it has been disclosed in U.S. Pat. No. 4,843,541. In this scheme, it is possible to make a plurality of guest OSs operate under the management of a host OS which manages the whole system. Respective guest OSs are independent systems having different address spaces. Access to a main memory in each partition is conducted according to the following procedure.

[0011] (1) A virtual address of a guest is translated to a real address.

[0012] (2) The above described guest real address is translated to a main memory address in the host.

[0013] (3) The main memory is accessed by using the main memory address in the host derived in (2).

[0014] The above described address translation of the two stages must be conducted between a CPU and the main memory.

[0015] In the partitions of the main frame, it is made possible for respective guest partitions to have different address spaces and the fault containment is realized by conducting the above described address translation of the two stages. By overlapping addresses of guests in the address translation of (3), the shared memory can be realized.

[0016] In realizing a partition mechanism and an inter-partition shared memory of a hierarchical bus connection SMP by using the above described conventional techniques, there are problems described hereafter.

[0017] The conventional inter-partition shared memory mechanism is premised on a concentrated main memory architecture having an address translation mechanism of two stages between each CPU and the main memory. Therefore, the conventional inter-partition shared memory mechanism is largely different in architecture from the hierarchical bus connection SMP. Accordingly, the conventional technique cannot be applied to the hierarchical bus connection SMP as it is. In particular, respective CPUS use standard components. As a result, the address translation of the two stages used in the conventional technique cannot be conducted in the CPU, and relocation of the address of each partition (guest) cannot be conducted.

[0018] Furthermore, in the hierarchical SMP, the CPU cache coherence is kept at a high speed by using the bus snoop protocol. Therefore, the inter-partition shared memory mechanism needs to be capable of supporting the bus snoop protocol.

[0019] Therefore, an object of the present invention is to realize a partition mechanism and an inter-partition shared memory mechanism suitable for the architecture of the hierarchical SMP.

[0020] In addition, a future parallel system must support a general purpose OS. Accordingly, the partition system needs to have a general purpose architecture which does not depend on a specific OS. It is necessary to make it possible for each partition to have a free address space. In addition, it is necessary to realize dynamic generation and erasing of a partition in order to deal with a large number of applications and improve the reliability of the system by using dynamic reconfiguration of partitions.

[0021] Another object of the present invention is to flexibly manage the configuration of the inter-partition shared memory.

[0022] In addition, the partition system needs to realize high reliability at a low cost. Thus it is indispensable for partitions to back up each other. Therefore, a third object of the present invention is to facilitate recovery from an error from another partition in the case where the OS of a certain partition suffers from system down.

[0023] In order to achieve the above described first and second objects, such a hierarchical SMP that nodes each having CPUs coupled by a bus and a main memory are connected by a switch and cache coherence control is conducted through the switch, at the gateway of the switch from each node when the inside of the system is divided into partitions in each of which a different OS operates, with means for mutually translating an address of an access command for an area shared between partitions, between a real address used in a partition and an address used in common between partitions. As a result, the address of a local area of each partition is freely set. In addition, cache coherence control of the shared area can be conducted at high speed by using a snoop command of the hierarchical SMP.

[0024] Furthermore, in another preferred aspect of the present invention, conformity between the address of the access command issued from another partition and the configuration of the shared area is checked at the gateway of each node. As a result, fault containment can be realized between partitions.

[0025] Furthermore, in another preferred aspect of the present invention, there is provided apparatus for the system software to dynamically modify the configuration information of the shared area between partitions. As a result, flexible management of the shared area becomes possible.

[0026] In addition, in order to achieve the above described third object, each partition is provided with a function of resetting CPUs of other partitions. In the case where a certain partition suffers from system down, it is possible to reset and re-initialize the partitions which have suffered from system down.

[0027] FIG. 1 shows a shared memory multiprocessor having a partition mechanism according to the present invention;

[0028] FIG. 2 shows an example of partitioning in a shared memory multiprocessor according to the present invention;

[0029] FIG. 3 shows an example of address spaces of respective partitions in a shared memory multiprocessor according to the present invention;

[0030] FIG. 4 shows a summary table of relations between access addresses and operations of address translation circuits of respective nodes;

[0031] FIG. 5 shows the configuration of an address translation table of each node;

[0032] FIG. 6 shows a detailed diagram of an output side translator of each node;

[0033] FIG. 7 shows the configuration of partition configuration information of respective nodes;

[0034] FIG. 8 shows the configuration of main memory configuration information in partition of respective nodes;

[0035] FIG. 9 shows a detailed diagram of a shared area detect circuit of each node;

[0036] FIG 10 shows a table describing relations between access commands and a destination indicate and global bit addition circuit of respective nodes;

[0037] FIG. 11 shows a detailed diagram of a shared area detect and address check circuit of each node;

[0038] FIG. 12 shows a flow diagram showing a process conducted when a home partition ensures a sa in memory in a multiprocessor system of the present invention;

[0039] FIG. 13 shows a flow diagram showing a process conducted on an added partition side when a shared partition is added in a multiprocessor system of the present invention;

[0040] FIG. 14 shows a flow diagram showing a process conducted on a home partition side when a shared partition is added in a multiprocessor system of the present invention;

[0041] FIG. 15 shows a flow diagram showing a process conducted on a home partition side when a shared area is erased in a multiprocessor system of the present invention;

[0042] FIG. 16 shows a flow diagram showing a process conducted on a side of a partition which imports the shared area when a shared area is erased in a multiprocessor system of the present invention; and

[0043] FIG. 17 shows the details of a reset circuit in a multiprocessor system of the present invention.

[0044] FIG. 1 is a block diagram of a shared memory multiprocessor according to the present invention. The present system is an 8-node system. A plurality of nodes 100 to 800 (which are sometimes referred to as nodes 0 to 7) are connected by a bus or internode connection switch 900. Six nodes other than the nodes 100 and 800 are omitted in FIG. 1 in order to avoid the complexity. However, every node has the same structure. By taking the node 0 as a representative example, the structure will be described. The node has a main memory 180 and four processors, i.e., Central Processor Units (CPUs) 110, 111, 112 and 113. These CPUs are coupled to a node controller 120 via a CPU bus 190. The node controller 120 includes a main memory access circuit 130, an address translation circuit 140, partition configuration information 160, and main memory configuration information partition 165. The main memory 180 forms a part of a main memory common to this system, and holds a part of a program executed by each node and data. This system is the so-called parallel processor system of distributed shared memory type.

[0045] The cache coherence between CPUs in the node is managed by a bus snoop protocol. As the bus snoop technique, a known technique is employed. In FIG. 1, the CPUS in the node are connected by the bus. As for the hardware, however, a coupling scheme other than the bus, such as one-to-one coupling or coupling using a switch, may also be used. As for these connection methods in the nodes, various known techniques can be applied.

[0046] The cache coherence between the CPUs of the nodes is also managed by using the bus snoop protocol. The inter-node connection switch 900 logically functions in the same way as the bus.

[0047] The address translation circuit 140 is hardware for translation between an address in the node (an address used by the CPUs 110 to 113, the CPU bus 190, and the main memory 180) and an address used by the inter-node connection switch 900 (an address outside the node).

[0048] Within the address translation circuit 140, the side for outputting a command from the inside of the node includes a shared area detect circuit 141 for determining whether the accessed area is a shared area, an output side translator 144 for translating the address of the shared area, and a destination indicate and global bit addition circuit 142 for sending out a command to proper destination. The side for inputting a command from the outside of the node includes a shared area detect and address check circuit 143 for determining whether a command from another node is a command directed to a shared area and checking the address, an input side translator 145 for translating the address of a shared area, a DND acknowledge circuit 146 for issuing an acknowledge in response to an access request for a shared area which is not shared by its own node, and an acknowledge wait circuit 147 for waiting an acknowledge of a snoop command from another node. These circuits performs operations specific to the present invention.

[0049] Each node has an inter-OS communication function 170 to be used for system initialization process, configuration control, debug and so on. Without using the shared memory, system software of each node can perform communication. Physically, an inter-OS communication path 910 can also share the same hardware as the inter-node connection switch. Alternatively, the inter-OS communication path 910 can also use a typical network such as a LAN.

[0050] FIG. 2 shows a configuration example of partitioning in a hierarchical SMP of the present invention. Partitioning is conducted by taking a SMP node as the unit, and an arbitrary combination of nodes can be adopted. In the example shown in FIG. 2, the inside of the system is divided into three partitions. A first partition 1000 includes nodes 0 to 3 (100, 200, 300 and 400). A second partition 1001 includes nodes 4 and 5 (500 and 600). A third partition 1002 includes nodes 6 and 7 (700 and 800). Respective partitions are independent systems in that different OSs operate. Respective partitions have independent address spaces as hereafter described. In addition, partitions have a shared memory which is a part of the main memory and which is shared by partitions.

[0051] In the case where there is not a memory shared by partitions, exchange of data between partitions must be conducted by using a message communication mechanism or a typical network (LAN). Since the latency of message communication requires approximately at least 10 μs, it is difficult to realize high speed communication. If a hierarchical bus connection shared memory mechanism is utilized, however, communication between nodes can be realized in several hundreds ns or less. Therefore, the inter-partition shared memory is indispensable for realizing faster communication between partitions, faster resource lock processing between partitions, and faster fail over between partitions.

[0052] FIG. 3 shows an example of address maps of respective partitions in the case where a shared memory mechanism of partitions is used. This example corresponds to the partitioning exemplified in FIG. 2.

[0053] Respective partitions (partitions 0 to 2) have real address spaces which are independent from partition to partition (as shown on the left hand side of FIG. 3). Respective nodes in the same partition have the same real address space. The real address of the above described partition is the address used in the CPU bus 190 in nodes included in the partition. It is a real address to be used by the CPUs 110 to 113, the main memory 180 and the main memory access circuit 130 which are included in the node. In addition to the above described real address spaces of respective partitions, the system has a shared memory space (shown on the right hand side of FIG. 3) common to the whole system. An area shared by partitions have different addresses within the partitions, but has a common address in the shared memory space.

[0054] Access from each partition to a shared area is conveyed to another partition via a shared memory space address. In other words, an access command such as a cache coherent check command (CCC command) issued to a shared area by each node is transformed from the real address of its own partition to a shared memory space address when the access command is outputted from the accessing node. When the access command is inputted to an accessed node, the access command is transformed from the shared memory space address to a real address of an accessed partition. These address transformation operations are conducted by the address translation circuit 140 located at the gateway of each node. Owing to this address translation function, it is possible in each partition to freely determine a real address space which exceeds the range of the physical main memory of a node in its own partition.

[0055] It should be noted that in the case where the bus snoop protocol is used there is a possibility that a CCC command will be issued by an arbitrary CPU which shares a certain cache line and the CCC command might be broadcasted in some cases to all CPUS which may possibly share the pertinent line. Therefore, address translation conducted in each partition between the local real address of a partition and the shared space address must be translatable in both directions. Furthermore, also when accessing a shared area which has a physical main memory in the same partition (such as when one of the nodes 0 to 3 accesses a shared area A), an address in the CCC command must be translated to an address of the shared memory space once in order to correctly convey the CCC command to nodes located outside the partition as well. For access to a not-shared area (an area which is accessed by only nodes included in the partition), address translation is not conducted at all (the CCC command is sent to only the nodes included in the partition). At that time, it is impossible to determine on the basis of the address alone whether a command inputted from the switch is access to a not-shared space or access to a space shared by partitions. Therefore, each command in the switch has a global bit whereby it can be determined whether the command is access to a shared space.

[0056] In accordance with a feature of the present invention, the address translation circuit 140 is provided in the gateway of each node to the switch, and mutual translation between the address of a shared area in each node and its shared memory space address is conducted.

[0057] Hereafter, an address management scheme between partitions will be described in detail by referring to the example shown in FIG. 3. In FIG. 3, the address space of the first partition 1000 has local main memories 10, 20, 30 and 40 respectively of the nodes 0 to 3. The address space of the second partition 1001 has local main memories 50 and 60 respectively of the nodes 4 and 5 and a shared area 1a.The address space of the third partition 1002 has local main memories 70 and 80 respectively of the nodes 6 and 7 and shared areas 1band 2a.All of the nodes included in each partition have the same address space. For example, both the nodes 4 and 5 can access the areas 50, 60 and 1a.

[0058] As for shared areas, there are two shared areas A and B. The shared area A is allocated onto the local main memory of the node 0, and the shared area B is allocated onto the local main memory of the node 4. They are represented as shared area A (physical main memory) 1 and shared area B (physical main memory) 2, respectively. Partitions having main memories corresponding to these shared areas are hereafter referred to as partitions of export side. On the other hand, in the shared memory space, the shared area A and the shared area B are located in 1zand 2z,respectively. Therefore, the partition 0 maps (exports) the shared area A from the real address space 1 in the partition to the shared memory space 1z,and the partition 1 maps (exports) the shared area B from the real address space 2 in the partition to the shared memory space 2z.

[0059] Furthermore, the shared area A is shared by the partition 1, the partition 2 and the partition 3. The shared area B is shared by the partition 2 and the partition 3. Like the partitions 2 and 3 for the shared area A and the partition 3 for the shared area B, a partition which has not a main memory of the pertinent area therein, i.e., a partition which accesses a physical main memory of the shared area included in another partition, is referred to as partition of import side. In the partition of the import side, an area (window) for accessing the main memory area included in another partition is formed in a portion of its own partition where a main memory of a real address is not mounted. Therefore, the partition 1 maps (imports) the shared area A from the shared memory space 1zto the real address space 1ain the partition. The partition 2 maps (imports) the shared area A from the shared memory space 1zto the real address space 1bin the partition, and maps (imports) the shared area B from the shared memory space 2zto the real address space 2ain the partition.

[0060] The foregoing description is summarized as follows. In the example of FIG. 3, the following (bidirectional) address translation operations are required at gateways of respective nodes.

[0061] The shared area A is exported from the main memory 1 in the node 0 to the shared memory space 1z.

[0062] Nodes 4 and 5 (the Second Partition 1001)

[0063] The shared area B is exported from the main memory 2 in the node 4 to the shared memory space 2z.

[0064] The shared area A is imported from the shared memory space 1zto the area 1ain its own partition.

[0065] Nodes 6 and 7 (the third partition 1002)

[0066] The shared area A is imported from the shared memory space 1zto the area 1bin its own partition.

[0067] The shared area B is imported from the shared memory space 2zto the area 2ain its own partition.

[0068] The address of each of not-shared areas which are not mentioned in the foregoing description is not translated even when a CCC command is outputted from a node.

[0069] FIG. 4 collectively shows address translation functions required at respective gateways of respective nodes in the present invention. Real addresses are divided into two classes, i.e., not-shared address (area which can be accessed from only the inside of a partition) and shared address (address shared by partitions). In addition, the shared addresses are divided into two classes, i.e., the case where there is a main memory in its own partition, and the case where there is not a main memory in its own partition.

[0070] (1) Not-shared Area

[0071] As for the not-shared area which is accessed from only the inside of the partition, the address in partition is used in the switch as well. Therefore, address translation at the gateway of the node is not conducted. The snoop command is multicast only to nodes in the partition.

[0072] (2) Shared Area (Export Side)

[0073] As for a shared area having a physical main memory in the partition, translation between the address of the main memory in the own partition and the address of the shared memory space in the switch is required. The snoop command is sent to all nodes. Such optimization that the snoop command is sent to only nodes of sharing partitions is also possible.

[0074] (3) Shared Area (Import Side)

[0075] As for a shared area having no physical main memory in the partition, translation between the address of a window for accessing a shared area in the own partition and the address of a shared memory space in the switch is required. The snoop command is sent to all nodes. In this case as well, such optimization that the snoop command is sent to only nodes of sharing partitions is also possible.

[0076] Furthermore, in order to realize the fault containment between partitions, unauthorized access between partitions is checked. In ordinary highly multiplexed SMP, access authority check is conducted at the time of address translation when the CPU translates a virtual address to a real address. Assuming that the system software of an OS or the like of another partition runs away, however, access authority check on the CPU side which issues the access is not enough, but it is necessary to conduct access authority check in the accessed partition. Furthermore, since the cache coherence management using the bus snoop protocol is conducted between CPUs, not only data in the main memory but also data cached in the cache must be protected from unauthorized access. In order to realize the above described check, the following check is conducted at the entrance of each node on a CCC command which has arrived from another node.

[0077] (1) Not-shared Area

[0078] Commands from nodes located outside the partition are not authorized.

[0079] (2) Shared Area (Both the Export Side and the Import Side)

[0080] In the address translation hardware, there is provided means for storing nodes authorized to share each shared area (i.e., nodes sharing the area) in a bit map form. Commands from nodes which are not authorized to share the shared area are not accepted.

[0081] The address translation heretofore described is dynamically set by system software such as an OS or middle software when the shared memory is allocated. The scheme of setting will be described later.

[0082] Hereafter, the configuration of a shared memory multiprocessor which realizes the inter-partition shared memory mechanism and the inter-partition address translation heretofore described will be described in detail.

[0083] With reference to FIG. 1, the address translation circuit 140 is disposed at the gateway of each node, to be concrete, between the main memory access circuit 130 and the inter-node connection switch 900. The address translation circuit 140 is a circuit for conducting address translation inside and outside the node, destination specification of a CCC command sent to the outside of the node, and error check for a CCC command sent from the outside of the node, on the basis of information written in the address translation table 150, the partition configuration information 160, and the main memory configuration information in partition 165.

[0084] The address translation table 150, the partition configuration information 160, and the main memory configuration information in partition 165 have been subjected to memory mapping. The processors in the partitions can access them.

[0085] FIG. 5 shows the format of an entry of the address translation table 150. In order to conduct address translation on the shared area, the address translation table has one entry shown in FIG. 5 per shared area. The address translation circuit conducts full-associative bidirectional address translation. Therefore, the number of shared areas exported and imported by a certain node is limited by the number of entries of the address translation table. As a result, the number of entries of the address translation table must be sufficiently large. Each entry has the following information.

[0086] “Enable bit” (E) 151

[0087] The entry is made valid.

[0088] “Address in partition” 152

[0089] “Address in shared memory space” 153

[0090] (These two addresses are respective start addresses.)

[0091] “Size” 154

[0092] Size of shared space

[0093] “Home node number” 155

[0094] Number of a node having the pertinent shared area in the main memory

[0095] “Nodes to be authorized to share” 156

[0096] A list of node numbers authorized to share the pertinent area (stored in a bit map form).

[0097] (Whether nodes are authorized to share is determined by taking a partition as the unit. With due regard to the processing efficiency, storage is conducted in the present table by taking a node number as the unit.)

[0098] Here, the shared memory is handled every certain fixed unit (such as 1 MB). In that case, fields 152 to 154 become multiples of 1 MB. Therefore, twenty low-order bits of the fields 152 to 154 are fixed to 0. Bits of the table are not mounted.

[0099] FIG. 7 shows details of the partition configuration information 160. The partition configuration information 160 shows how the system is partitioned. The partition configuration information 160 is formed of a bit map 161 showing which nodes are included in which partition, and a flag 162 showing which partition its own node is housed. The number of entries of the table is the total number of nodes of the system. Therefore, up to the case of one node/one partition can be supported. FIG. 7 shows values of the partition configuration information 160 in the nodes 0 to 3 illustrated in FIG. 2. In other words, in this example, the partition 0 is its own partition, and the partitions 3 to 7 are not used. In other nodes as well, the same information is stored except the local partition flag 162.

[0100] FIG. 8 shows details of the main memory configuration information in partition 165. The main memory configuration information in partition indicates the address range the local main memory of each of the nodes included in the own partition takes charge of. The main memory configuration information in partition does not have information of other partitions. The number of entries is the total number of nodes of the system. Up to the case where the whole system is formed of one partition can be supported. The main memory configuration information in partition includes a start address 166 and an end address 167 of the main memory each node takes charge of, and a V bit 168 indicating that the entry is valid.

[0101] Commands exchanged by the inter-node connection switch 900 will now be described. In the hierarchical SMP, data reading/writing (cache coherent check: CCC) of each CPU is managed by the bus snoop protocol. As for the bus protocol in the node, a known technique is adopted. As for the CCC between nodes as well, a known technique is used. Here, as an example of the CCC between nodes, outline of a procedure of the bus snoop using a fetch command between nodes will now be described. For the purpose of description, it is now assumed that partitioning is not conducted. In other words, it is now assumed that the whole system is formed of one partition.

[0102] (1) In the case where a certain CPU reads data, the fetch command is issued to all other nodes (all CPUs and the main memory of the home node).

[0103] (2) If there is modified data (latest data) in the cache in the own node, the node which has received the fetch command returns that data. Otherwise, the node which has received the fetch command replies that there are no modified data. If there are no modified data in the cache, the home node returns data in the main memory.

[0104] (3) In the accessing node, replies from other nodes are totalized. If modified data in the cache is sent from another node, the data is returned to the accessing CPU. If every node does not have modified data, data returned from the main memory (data of the main memory in the own node in the case where the own node is the home node) is returned to the accessing CPU. For determining whether data returned from the main memory should be used, it is necessary to wait the reply “there are no data” from all nodes.

[0105] Hereafter, CCC commands used on the bus within the nodes of the hierarchical SMP will be described. Characters enclosed in parentheses are abbreviations used in the embodiment.

[0106] Fetch (F)

[0107] It requests line transfer of data.

[0108] It is issued in the case where a read command of the CPU has failed.

[0109] Fetch&Invalidate (FI)

[0110] It requests invalidation of data on other caches simultaneously with the line transfer of data.

[0111] It is issued in the case where a read command of the CPU has failed.

[0112] Invalidate (I)

[0113] It requests invalidation of data on other caches. It is issued in the case where the CPU has issued a write request to a cache line shared with other caches.

[0114] WriteBack (WB)

[0115] It requests writeback of a cache line.

[0116] It is issued when data has been turned out by replacement.

[0117] Data (D)

[0118] It is a command for returning data in response to the F or FI command in the case where there is modified (latest) data in the cache.

[0119] DataMem (DM)

[0120] It is a command for returning data in the main memory of the home node in response to the F or FI command. In the home node, it is returned in the case where there are no modified (latest) data in the cache within the node. In the case where a D command has come from any other node, data returned by the DM command is ignored. (Data in the cache is given priority.)

[0121] It is also used for the reply to UncachedRead.

[0122] NoData (ND)

[0123] It is returned in response to the F or FI command in the case where there are no modified (latest) data in the cache of the pertinent node (except the home node).

[0124] DummyNoDatä (DND)

[0125] A reply returned from a node located outside the partition in response to the F or FI command.

[0126] (DND is not an ordinary CCC command, but it is a command required in implementation specific to the present embodiment. Its meaning is the same as that of ND.)

[0127] UncachedRead (UR)

[0128] UncachedWrite (UW)

[0129] It is a command for directly accessing the main memory in cache off.

[0130] A command in the inter-node connection switch has the following fields specific to the present invention besides fields required for CCC, such as command, address, and data.

[0131] (1) Destination Node Number

[0132] The destination node number is represented by a bit map. By virtue of representation of destination using a bit map, it is possible to easily realize multicast directed to a plurality of specific nodes, such as nodes in the partition, by setting a plurality of bits and further broadcast directed to all processors of the system by setting all bits.

[0133] (2) Global Bit

[0134] It is a bit for determining whether the pertinent access command is a command for a shared area (hence the address is an address of the shared memory space) or a command for a not-shared area (hence the address is a local address in the partition).

[0135] Hereafter, operations of the address translation circuit in the case where a command is issued to another node, and in the case where a command has been received from another node will be described separately for respective cases and successively.

[0136] (A) Processing in the Case where a Command is Issued from a Node

[0137] If a CCC command is delivered from the main memory access circuit 130 to the address translation circuit, it is first inputted to the shared area detect circuit 141.

[0138] FIG. 9 shows details of the shared area detect circuit 141. From an inputted command 141c,a part of an access address 1410bis taken out. The access address 1410bis inputted to circuits 1410 and 1411.

[0139] In the circuit 1410, it is detected by using the main memory configuration information in partition 165awhether the access address 1410bis included in any of main memories within the local partition or not. In other words, it is checked for every bit having the valid bit 168 of FIG. 8 set equal to 1 whether the access address 1410bis located between the start address 166 and the end address 167. If the access address is included in any of the main memories, then “1” is outputted to output 1410aand the access address 1410bis judged to be an address having a main memory in the partition. Since the internal circuit is the same as that of FIG. 6, detailed description thereof will be omitted.

[0140] In the circuit 1411, it is detected by using the information 150aof the address translation table whether the access address 1410bis included in any of partitions' shared areas defined in the address translation table or not. In other words, it is checked for every bit having the E bit of FIG. 5 set equal to “1” whether the access address 1410bis included between the “address in partition” 152 and the “size” 154. If the access address is included in any of the partitions' shared areas, then “1” is outputted to output 1411a,and the access address 1410bis judged to be one of shared areas exported from the inside of the partition or imported from the inside of the partition.

[0141] If a gate 1412 outputs “1”, i.e., if the access address 1410bhas a main memory in the partition, but it is not a shared area exported or imported, then the access command is distributed to a not-shared side 141bby a gate 1413. Otherwise (i.e., if the output of the gate 1412 is “0”), i.e., the access address 1410bdoes not have a main memory in the partition, or it is an exported or imported shared area, then the access command is distributed to a shared side 141a.

[0142] On the not-shared side 141b,the access command is sent to the destination indicate and global bit addition circuit 142 without being subject to address translation. As a result, address translation is not conducted on the access command sent to the not-shared area.

[0143] On the other hand, on the shared side 141a,the access command is inputted to the output side translator 144. FIG. 6 shows details of the output side translator 144. From an inputted command 141a,an access address 1442ais separated. It is determined by a range detect circuit 1440 whether the access address 1442ais included in the “address in partition” 152 to the “size” 154 of a valid entry of the address translation table. The range detect circuit is provided at the rate of one per entry of the address translation table (1440 to 1440′). If the access address 1442ais included in the range of address in the partition of the address translation table, then a signal 1440ais outputted. If the signal 1440ais “1”, then the “address in partition” 152 of a pertinent address translation table is outputted to an output A 1441a,and the “address in shared memory space” 153 is outputted to an output B 1441b.In an address translation circuit 1442, a new address 1442bis calculated according the following equation by using an old base address 1441aoutputted from the signal A (corresponding to the address in partition in the address translation table), a new base address 1441boutputted from the signal B (corresponding to the shared memory area address in the address translation table), and the input address 1442a.

[0144] New Address=Input Address−Old Base Address+New Base Address

[0145] By virtue of this calculation, the input address issued from the inside of the partition (local real address of the partition) can be relocated to an address of the shared memory space. Also in the case where the access address has been included in another entry, similar processing is advanced via the range detect circuit 1440′. Together with a part of the command other than the address, the output address 1442bis sent to the destination indicate and global bit addition circuit 142 via a gate 1445 (enabled by an OR 1443aof selection signals). If the input address 1442adoes not match with any of entries of the address translation table, then an error is detected.

[0146] The destination indicate and global bit addition circuit 142 determines which node a command outputted from the inside of the node should be outputted to, and in addition determines the value of the global bit in the command. FIG. 10 shows operation of the destination indicate and global bit addition circuit 142. The circuit conducts predetermined operation according to the kind of the command and whether the command is a command directed to a not-shared area (in the case where the command is inputted from the signal 141bside) or a command directed to a shared area (in the case where the command is inputted from the signal 144aside).

[0147] (1) Snoop Request (F, FI, I) Command on not-shared Address

[0148] After the node configuration in the partition is derived from the partition configuration 160, the command is multicast to nodes in the partition.

[0149] (2) Snoop Request (F, FI, I) Command on Shared Address

[0150] The command is broadcasted to all nodes of the system.

[0151] The global bit is set equal to “1”.

[0152] (3) Main Memory Access (WB, UW, UR) Command on not-shared Address

[0153] On the basis of the main memory configuration information in partition 165, comparison is conducted to determine which node takes charge of the address range the access address belongs to. The home node (in the local partition) of the access address is derived, and thereafter the command is sent to the home node.

[0154] The global bit is set to “0”.

[0155] (4) Main Memory Access (WB, UW, UR) Command on Shared Address

[0156] From the address translation table, the “home node number” 155 of the access address is derived. The command is sent to the home node.

[0157] The global bit is set to “1”.

[0158] (5) Acknowledge (D, DM, ND) Command on not-shared Address

[0159] The command is returned to the requesting node. (DND to a not-shared area is not generated.)

[0160] The global bit is set to “0”.

[0161] (6) Acknowledge (D, DM, ND, DND) Command on Shared Address

[0162] The command is returned to the requesting node.

[0163] The global bit is set to “1”.

[0164] Owing to the processing heretofore described, the command outputted from the node can be transmitted to suitable destination via the inter-node connection switch 900.

[0165] (B) Processing Conducted when a Node has Received a Command

[0166] If a command inputted from the inter-node connection switch 900 is inputted to the address translation circuit 140, then the command is first inputted to the shared area detect and address check circuit 143. In the shared area detect and address check circuit 143, classification of an inputted command and error check are conducted. FIG. 11 shows details of the shared area detect and address check circuit 143. A command 143dsent from another node is first inputted to a circuit 1430. The value of the global bit is examined.

[0167] If the global bit is “0”, i.e., in the case of access to a not-shared area, the command is outputted to a signal 1430a.Thereafter, in circuit 1432, it is checked on the basis of the partition configuration information 160 whether the source node belongs to the local partition. Thereafter, in a circuit 1433, it is checked on the basis of the main memory configuration information in partition 165 whether the access address is included in main memories of any node in the local partition. A command which has not posed a problem in both checks in the circuits 147 through a signal 143b.In this case, address translation is not conducted. A command which has posed a problem in the check of either 1432 or 1433, i.e., access from the outside of the partition or access to the outside of the main memory in the partition is detected as an error. As a result, unauthorized access to a not-shared area in the partition can be prevented.

[0168] Processing conducted in the case where the global bit is “1”, i.e., in the case of access to a shared area will now be described. The access command is sent to a circuit 1431 through a signal 1430b.In the circuit 1431, it is checked whether the access address is included in the range of the “address in shared memory space” 153 to the “size” 154 of any of entries of the address translation table 150 (i.e., whether the access address is included in any of shared areas exported or imported by the partition this node belongs to).

[0169] If there is the pertinent entry in the address translation table, i.e., if the access address is an address shared by a partition this node belongs to, the command is conveyed to a circuit 1434 through a signal 1431a.The DND is returned from a not-shared node. On the other hand, in the case of a command other than the DND, it is checked in the circuit 1434 whether the source node of the access command is included in the bit map 156 of the address translation table representing nodes to be authorized to share the pertinent entry. If there is no problem as a result of the check, the command is sent to the input side translator 145 through a signal 143a.In other words, translation from the shared area address to the real address in the partition is conducted.

[0170] If there is a problem as a result of the check in the circuit 1434, i.e., if access is conducted from a node other than partition authorized to share the pertinent shared area (except the DND), then it is detected as an error. As a result, unauthorized access to a shared area in the partition can be prevented.

[0171] If there is the pertinent entry in the address translation table, the access address is not included in the shared main memory shared by the partition this node belongs to. This case occurs because in the destination indicate and global bit addition circuit 142 which has transmitted the command, the snoop request command (F, FI, I) on the shared area is broadcasted uniformly to all nodes. Therefore, the command is also sent to nodes which are not included in the sharing nodes. In this case, a DND command indicating that there is no pertinent data in the node must be returned to the access node in order to correctly wait acknowledge to the snoop request. (The access node anticipates acknowledges from all nodes to which the command has been broadcasted.) After being checked in a circuit 1435 that the access command is a snoop request, therefore, the access command is conveyed to the DND acknowledge circuit 146 through a signal 143c.(if the access command is not a snoop request, an error is reported.) In the DND acknowledge circuit 146, the DND command is issued to the source node. The DND command is returned to the accessing node through the destination indicate and global bit addition circuit 142.

[0172] Operation of the input side translator 145 will now be described. In the input side translator, the shared memory space address in the command is translated to a real address in the partition on the basis of information of the address translation table 150. The input side translator has the same internal configuration as that of the output side translator. The input side translator is different only in the translation direction from the output side translator.

[0173] Finally, operation of the acknowledge wait circuit 147 will now be described. The acknowledge wait circuit 147 is a circuit for totalizing acknowledges (D, DM, ND, DND) to F and FI commands which request readout of data from another node. The acknowledge wait circuit 147 does not influence other commands. The acknowledge wait circuit 147 waits until acknowledges (D, DM, ND, or DND) come from all nodes which have issued F or FI commands, i.e., all nodes of the system in the case of shared areas, and all nodes in the partition (indicated by the partition configuration information 160) in the case of not-shared areas. Then the acknowledge wait circuit 147 conducts the following judgments.

[0174] (1) In the case where the D command has been returned:

[0175] If the D command has been returned from one node, and ND, DND, or DM has been returned from another node, then the latest data read out by the D command, i.e., data modified in the cache of another node is returned.

[0176] (2) In the case where the DM command has been returned:

[0177] If the DM command has been returned from one node, and ND or DND has been returned from another node, then data in the main memory read out by the DM command is returned.

[0178] (3) In the case where the ND command has been returned:

[0179] If ND or DND has been returned from another node, then the own node is the home node. A reply that there are no data is returned. Thereafter, the main memory access circuit 130 reads out data in the main memory 180 and returns the data.

[0180] The operation of the acknowledge wait circuit 147 heretofore described is the same as that of the cache coherent check circuit of the conventional hierarchical SMP. Therefore, details of the internal configuration will be omitted.

[0181] How access to another node is conducted in the shared memory multiprocessor of the present invention heretofore described will hereafter be described by taking the F command as an example.

[0182] (1) F Command on Not-shared Area

[0183] In a node which has issued the F command, the F command outputted from the inside of the node is sent from the shared area detect circuit 141 to the destination indicate and global bit addition circuit 142 through the signal 141b(address translation is not conducted), and multicast to nodes in the partition (global bit=0).

[0184] In the node which has received the F command, error check is conducted in the shared area detect and address check circuit 143, and thereafter the F command is conveyed to the inside of the node through the signal 143band the acknowledge wait circuit 147 (address translation is not conducted).

[0185] The acknowledge (D, DM, ND) for the F command is sent from the shared area detect circuit 141 to the destination indicate and global bit addition circuit 142 through the signal 141b(address translation is not conducted), and returned to the accessing node (global bit=0).

[0186] In the accessing node which has received the acknowledge for the F command, error check is conducted in the shared area detect and address check circuit 143, and thereafter the acknowledge is conveyed to the acknowledge wait circuit 147 through the signal 143b(address translation is not conducted). In the acknowledge wait circuit 147, replies from all nodes in the partition are waited, and are returned to the accessing CPU.

[0187] (2) F Command on Shared Area

[0188] In the node which has issued the F command, the F command outputted from the inside of the node is sent from the shared area detect circuit 141 to the destination indicate and global bit addition circuit 142 through the output side translator 144. In this process, the access address is translated from the real address of the accessing partition to the address of the shared memory space. The F command is broadcasted to all nodes of the system (global bit=1).

[0189] If the node which has received the F command is included in a partition sharing the accessed shared area, then the F command is subjected to error check in the shared area detect and address check circuit 143, and thereafter the F command is conveyed to the acknowledge wait circuit 147 through the input side translator. In this process, the access address is translated from the address of the shared memory space to the real address of the accessed partition. Thereafter, the F command is conveyed to the inside of the node. Snoop in the node is conducted by using the local real address of the accessed partition.

[0190] The acknowledge (D, DM, ND) for the F command generated as a result of the above described process is sent from the shared area detect circuit 141 to the destination indicate and global bit addition circuit 142 through the output side translator 144. In this process, the access address is translated from the real address of the accessed partition to the address of the shared memory space again. The command is returned to the accessing node (global bit=1).

[0191] If the node which has received the F command is not included in the partition sharing the accessed shared area, then the F command is conveyed to the DND acknowledge circuit 146, and a DND command is returned to the accessing node (global bit=1).

[0192] In the accessing node which has received the acknowledge (D, DM, ND, DND) for the F command, error check is conducted in the shared area detect and address check circuit 143, and thereafter the acknowledge is conveyed to the acknowledge wait circuit 147 through the input side translator 145. In this process, the access address is restored from the address of the shared memory space to the real address of the accessing partition. In the acknowledge wait circuit 147, replies from all nodes of the system are waited, and the replies are returned to the accessing CPU. Answering is conducted by using the local real address of the accessing partition.

[0193] As heretofore described, owing to the address translation mechanism of the present invention, data can be accessed by using the local real address in the partition in both the accessing node and the accessed node.

[0194] How the system software such as the OS manages the shared memory among partitions of the shared memory multiprocessor in the present invention will now be described. The OS itself may manage the shared memory among partitions, or system software for managing the shared memory among partitions, such as middleware other than the OS, may manage the shared memory among partitions.

[0195] In the present system, the partitions do not have a shared memory at all when the system is initialized. Communication between partitions is conducted by using inter-OS communication means (170 and 910). Every inter-partition communication for shared memory management hereafter described is conducted by using this inter-OS communication means.

[0196] Hereafter, a node having a physical main memory of a shared memory (a node which exports the shared memory is referred to as home node. A partition the home node belongs to is referred to as home partition. Hereafter, with reference to FIGS. 12 to 16, the procedure of the operation of the system software will be described in detail by dividing it into allocation of a shared area in the main memory, addition of a shared partition, and erasing of a shared area.

[0197] As for the management of the shared area, it is hereafter assumed that the home partition basically conducts concentrated management. Also in the case where some unit other than the home partition conducts the management, the management can be realized by using the similar technique.

[0198] (A) Allocation of the main memory of the shared area Allocation of the main memory of the shared area is conducted in the home partition (FIG. 12). A main memory of a requested size is allocated to be used as a shared area, and necessary data are written into the address translation table. The shared area can be dynamically allocated.

[0199] First, an unused entry (an entry having E=0) in the address translation table of each node in the partition is allocated (step 2001). A real memory of the requested size is allocated in the local main memory (step 2002). Thereafter, the system software communicates with system software of other partitions, and allocate an area of the requested size in the shared memory space (step 2003). In all partitions, the shared memory must have the same address map.

[0200] Subsequently, the following data concerning the shared area are written into the address translation table of each of nodes in the partition (step 2004).

[0201] “Real address in partition” 152 (start address)

[0202] “Address in shared memory space” 153 (start address) “Size of shared area” 154

[0203] “Home node number” 155 (number of node which has secured a main memory)

[0204] “Bit map of node to be authorized to share” 156 (initialize it to 0)

[0205] At this time point, access authority to the shared area is not given to any partition. Thereafter, “1” is written into E bit of the address translation table of each bode in the partition, and the entry is made valid (step 2005).

[0206] By the processing heretofore described, an area in the main memory can be exported. It becomes possible to access the shared area allocated in the step 2002 from the address in the shared memory space allocated in the step 2003.

[0207] It should be noted here that the same content must be written into address translation tables of all nodes in the partition.

[0208] (B) Addition of Sharing Partition

[0209] For making it possible for other partitions to access the shared area allocated in (A) (i.e., to import the shared area), the following procedure is necessary. The import processing of the shared area can be dynamically conducted at arbitrary time after the shared area is allocated in the home partition.

[0210] (B1) Partition which Newly Imports Shared Area

[0211] In a partition of import side which is going to newly share a certain shared area, the following processing is necessary (FIG. 13).

[0212] First, an unused entry (an entry having E=0) in the address translation table of each node in the partition is allocated (step 2101). A real space, to which a physical main memory is not mapped, of a requested size is allocated (step 2102). This real space is used as a window for accessing a shared area having a physical main memory in another partition. Thereafter, access permission of a pertinent shared area is requested from the system software of the home partition (step 2103). Here, in a partition which newly imports the shared area, the home node conducts processing of steps 2201 to 2206 (which will be described later) and waits return of a reply of access authorization.

[0213] Thereafter, if access authorization comes from the home partition, an entry concerning the shared area of the address translation table sent together with access authorization is received (step 2104). Contents of the entry of the address translation table received in the step 2104 (except the E bit and address in the partition) are written into the address translation table of each of nodes in the partition, and the real address allocated in step 2102 is written into the “address in partition” field 152 of the address translation table of each of nodes in the partition (step 2105). Finally, “1” is written into the E bit of the address translation table of each of nodes in the partition.

[0214] By the processing heretofore described, import of 10 the shared area is completed, and it is possible to access the shared area in the home node from the real address allocated in the step 2102.

[0215] (B2) Home Partition

[0216] In the system software of the partition requested 15 to permit access to the shared area in the step 2103, processing described hereafter becomes necessary (FIG. 14). First the access request is checked, and it is decided whether the requesting partition is permitted to share the pertinent shared area (step 2201). Hereafter, a procedure in the case where the access is permitted will be described.

[0217] Subsequently, an old value of the “nodes to be authorized to share” field 156 of an entry corresponding to the shared area of the address translation table is stored into a work variable W (step 2202). In the “nodes to be authorized to share” field 156 of an entry corresponding to the shared area of the address translation table possessed by each of nodes in the partition, bits corresponding to all nodes which belong to the requesting partition are set (step 2203). As a result, the requesting partition is authorized to share the pertinent shared area.

[0218] Furthermore, the new value of the “nodes to be authorized to share” field 156 is sent to each of partitions (except the own partition) which includes nodes having W set equal to 1, and each of the partitions is requested to modify the field (step 2204). It should be here noted that the system software which manages shared areas is one in number every partition and each partition needs only to be requested once. Arrival of ACK telling that modification of the “nodes to be authorized to share” field 156 has been completed, from every partition requested in the step 2204 is waited (step 2205).

[0219] Finally, the entry of the shared area of the address translation table is sent to the requesting partition together with the access authorization of the shared area (step 2206).

[0220] By virtue of the processing heretofore described, the “nodes to be authorized to share” field 156 in the address translation table of every node which shares the pertinent shared area is updated, and the requesting partition can share the area.

[0221] (B3) Partitions which have Shared the Pertinent Area Until then (Except Home Partition)

[0222] In the partitions which have shared the pertinent area until then (except the home partition), the request issued by the home partition in the step 2204 is received, and the “nodes to be authorized to share” field 156 of an entry corresponding to the shared area of the address translation table possessed by each of nodes in the partition is updated. Upon completion of updating, the ACK is returned to the home partition.

[0223] (C) Erasing of Shared Area

[0224] Erasing of a shared area is started by the home partition, and all partitions which have shared the shared area execute a procedure hereafter described, in cooperation. Erasing of a shared area can also be conducted dynamically. After the shared area has been erased once, resources, such as the resource address translation table, the main memory, and the shared address space, which have been used until then can be utilized again.

[0225] It is assumed in the processing hereafter described that the application program has finished the use of the pertinent shared area. Therefore, access to the shared area to be opened is not caused.

[0226] (C1) Processing in Home Partition

[0227] The home partition makes all other partitions sharing the pertinent area erase the shared area, and thereafter erases the shared area of its own partition.

[0228] First, the home partition requests every partition which includes nodes corresponding to 1 in the “nodes to be authorized to share” field 156 of the address translation table (except its own partition) to erase the shared area (step 2301). And the home partition waits for every requested partition to return the ACK (step 2302). As a result, it is assured that the shared area has been erased in the partitions which imported the shared area.

[0229] Subsequently, the home partition purges all data of the shared area to be erased, in caches of all processors in its own partition (step 2303). Old data in the cache are thus driven out. Thereafter, the home partition invalidates the entry (i.e., writes “0” in the E bit) corresponding to the shared area to be erased, in the address translation table of every node in the partition (step 2304). The home partition releases the area in the shared memory space and the physical memory of the shared area (step 2305). Finally, the home partition purges the TLB of every processor in the partition (step 2306). By virtue of the processing heretofore described, resources used to access the shared area can be completely released.

[0230] (C2) Processing in Partitions which Import Shared Area

[0231] In each partition requested to erase the shared area by the home partition in the step 2301, processing heretofore described becomes necessary.

[0232] First, in all processors in its own partition, data of the shared area to be erased, in caches of all processors in its own partition are purged (step 2401). Thereafter, the pertinent entry in the address translation table of every node in the partition is invalidated (step 2402). The shared memory space, and the area (window area) allocated in the real address space in the partition to access the shared memory are released (step 2403). The TLB of every processor in the partition is purged (step 2404).

[0233] As a result, resources used to access the shared area can be completely released. Finally, the ACK is returned to the home partition (step 2405) to inform the home partition that the shared area has been completely released in its own partition.

[0234] By virtue of the procedure heretofore described, it is possible in the shared memory multiprocessor of the present invention to dynamically manage the memory shared among partitions.

[0235] By virtue of the configuration heretofore described, it is possible in the hierarchical bus connection SMP to realize a memory shared by partitions.

[0236] The function of resetting the partitions will now be described in detail. FIG. 17 shows a reset circuit according to the present invention. FIG. 17 shows only node 0. Other nodes also have absolutely the same configuration. Hereafter, the configuration of the node 0 will be described in detail. The CPU, the main memory access circuit, and the address translation circuit in each node are driven by a reset signal 921. Respective nodes have independent reset signals 921 to 928. Respective reset signals are driven by a reset transfer circuit 920. In addition, each node has a reset register 175 which can be accessed from CPUS, and each node can request resetting of other nodes.

[0237] First, if an external reset signal (reset at the time of power on, a signal from an ordinary reset button) 929 is driven, then reset signals (921 to 928) of all nodes are made active and all nodes are reset as usual.

[0238] The reset register of each node includes a node bit map 1751 indicating nodes to be reset, and an enable bit 1750. If the enable bit 1750 is set to “1”, the reset signal is sent to nodes corresponding to “1” in the bit map 1751 through the circuit 920.

[0239] By virtue of the circuit heretofore described, it becomes possible for software of each node to reset arbitrary nodes.

[0240] In the case where a certain partition (hereafter referred to as partition A) suffers from system down, the following operations are conducted by software of one (hereafter referred to as partition B) of partitions which does not suffer from system down.

[0241] (1) The partition B senses that the partition A has suffered from system down. (It can be known by a heart beat function or the like via the shared memory.)

[0242] (2) The partition B conducts investigation of error cause and removal of an error cause via the memory shared by partitions, and prepare information so as to allow the partition A to resume its task.

[0243] (Since arbitrary addresses including the system area can be shared in the shared memory function of the present invention, error analysis including the OS is possible.)

[0244] (3) The partition B sets bits of the bit map 1751 corresponding to nodes of the partition A to “1” (and sets other bits to “0”.)

[0245] (4) The partition B sets the enable bit 1750 to “1” and then restores it to “0”. (As a result, a reset signal is conveyed to respective nodes of the partition A through the reset transfer circuit 920.)

[0246] (5) Owing to the processing heretofore described, the partition A can resume the execution. (By using the information of (2), application is resumed.)

[0247] By virtue of the reset function heretofore described, it becomes possible for partitions to back up each other, and high reliability can be realized.

[0248] The present invention is not limited to the embodiment heretofore described, but can be applied to various variations.

[0249] (1) In the foregoing description, the snoop command on the shared area is broadcasted to all nodes in the system.

[0250] In contrast thereto, there is also possible such a scheme as to broadcast only to nodes in partitions sharing the accessed shared area in order to reduce the traffic imposed on the switch.

[0251] The following points are changed.

[0252] When the destination of a command outputted from the inside of a node is decided in the destination indicate and global bit addition circuit 142 (see FIG. 5), the snoop (F, FI, I) command on a shared area is multicast to only nodes corresponding to “1” in the “partitions to be authorized to share” field 156 of the address translation table corresponding to the shared area. (As a result, the snoop command is not sent to partitions which do not share the area.)

[0253] The DND command and the DND acknowledge circuit are not used.

[0254] When acknowledges of the command on the shared area are waited in the acknowledge wait circuit 147, as many acknowledges as nodes corresponding to “1”s in the “partitions to be authorized to share” field 156 of the address translation table corresponding to the shared area are waited. (As a result, acknowledges of the command in the sharing partitions are waited.)

[0255] (2) In the foregoing description, nodes to be authorized to share are stored in the bit map 156 corresponding to each node. However, it is also possible to store nodes to be authorized to share in a bit map corresponding to each partition.

[0256] (3) In the foregoing description, the main memory access circuit 130 and the main memory 180 in the node are inserted between the address translation circuit 140 and the CPU bus 190. However, there is also possible such a scheme that the address translation circuit 140 is directly coupled to the CPU bus 190 (but is not directly coupled to the main memory access circuit 130). Apart from the address translation circuit 140, the main memory access circuit 130 is connected to the CPU bus 190. In this case, access from another node to the main memory 180 is conducted via the CPU bus 190. In this case as well, the present invention having the address translation means 140 at the entrance of each node can be utilized as it is.

[0257] (4) In the foregoing description, the components 141 to 145 and 147 are shown as separate circuits. By making the circuits of the output side (141,142 and 144) and the circuits of the input side (143, 145 and 147) common respectively, however, duplicated circuits such as the search of the address translation table can be reduced.

[0258] (5) In the foregoing description, addresses of the shared memory space can be taken in arbitrary positions. In order to realize arbitrary translation, address translation is realize in a full associative table.

[0259] In contrast thereto, by representing the node number by high-order bits of address of the shared memory space and representing the real address in the partition by lower-order bits, the address translation on the export side in the home partition can be significantly reduced. In this case, a real address in the partition can be translated to an address of the shared memory space by simply adding a node number as high-order bits. An address of the shared memory space can be translated to a real address in the partition by simply removing the node number in the high-order bits.

[0260] In this case, in order to store, in each node, the management information of shared areas with respect to the whole main memory space in the partition, there is needed a table for storing, every managed unit (having, for example, 1 MB) of shared area, a bit indicating whether sharing is possible (i.e., a bit for storing information as to whether the 1-MB space is a shared area or a not-shared area) and a bit map of nodes to be authorized to share (which is the same information as the field 156 of the address translation table).

[0261] As a result, the address translation and access check on the export side (home partition) can be significantly simplified and the hardware can be reduced.

[0262] (6) In the foregoing description, the address translation table of the export side is common to the address translation table of the import side. On the import side, however, a table different from that of the export side may be used. In this case, the “home node number” 155 on the export side (which is redundant information) can be eliminated.

[0263] (7) In the foregoing description, it is possible to freely read and write data with respect to the shared area. By making the shared area read only from the outside of a partition, however, the hardware can be simplified. In this case, check of the address and the node number (check in the circuits 1432 to 1434) in the shared area detect and address check circuit 143 is unnecessary. Furthermore, the “nodes to be authorized to share” field 156 needs only to be stored in the home node alone. If a write command such as FI, I, WB or UW comes from the outside of the partition, the shared area detect and address check circuit 143 reports an error.

[0264] Furthermore, by combining (5) to (7), the hardware quantity can be significantly reduced.

[0265] (8) By adding a read only bit to the address translation table, it is also possible to set only a specific shared area to read only from the outside of the partition. If in that case a write command such as FI, I, WB or UW comes from the outside of the partition to a shared area having a set read only bit, the shared area detect and address check circuit 143 reports an error.

[0266] (9) In the foregoing description, the destination indicate and global bit addition circuit 142 located at the exit of each node specifies the destination of the snoop (F, FI, I) command issued from the node. In contrast thereto, there is possible to adopt such a scheme that the inter-node connection switch 900 has information equivalent to the partition configuration information 160 and the destination is specified in the inter-node connection switch 900. In that case, if the global bit in the command is “1” (i.e., in the case of access to a shared area), the command is broadcasted to all nodes in the system is conducted. If the global bit in the command is “0” (i.e., in the case of access to a not-shared area), the command is multicast only to nodes in the same partition as the destination node.

[0267] (10) In the foregoing description, CPUs in the node are connected by the bus 190. However, other connection forms (such as connection using a switch, or one-to-one connection to the main memory access circuit 130) may also be used.

[0268] (11) In the above described embodiment, the main memory configuration information in partition 165 separately stores the ranges of the main memory respective nodes take charge of, by using pairs of the start address 166 and the end address 167. However, by using, for example, such a technique as to make the end address of the node n common to the start address of the node n+1, the hardware can be reduced. Furthermore, each node may take charge of a plurality of real address areas by providing a plurality of pairs of the start address 166 and the end address 167 as the areas each node takes charge of.

[0269] (12) In the foregoing description, CPUS (110 to 113) in the node have independent caches. However, an external cache (level-3 cache) shared by a plurality of CPUs may be provided. It is also possible to provide the main memory access circuit 130 of each node with a copy of the cache TAG of the CPU, and filter the cache coherent transaction coming from another node.

[0270] (13) In the foregoing description, the inter-node connection switch 900 is connected by a crossbar network. However, the inter-node connection switch 900 may also be connected by using a network of a different form (such as complete coupling or a multi-stage network).

[0271] (14) In the above described embodiment, broadcasting or multicasting is ordered by specifying the destination node in the bit map when issuing a network command to other nodes. However, broadcasting or multicasting may be realized by issuing a plurality of commands for each of the destination nodes from the destination indicate and global bit addition circuit 142.

[0272] According to the present invention, the inside of the hierarchical bus connection SMP is divided into a plurality of partitions. There is provided, at the gateway of each node, means for bidirectionally translating an address of a shared area between an address in a partition and a shared memory space address common to partitions, when realizing a shared memory between partitions. As a result, each partition has a free address space, and data in a shared area can be managed at high speed by a snoop protocol. In addition, an inter-partition shared memory mechanism making possible fault containment between partitions can be realized. Furthermore, by providing means for dynamically generating and erasing the above described address translation information, the shared memory between partitions can be managed flexibly.