Method and system for supporting software partitions and dynamic reconfiguration within a non-uniform memory access system

[0002] 1. Technical Field

[0003] The present invention relates to a method and system for data processing in general, and in particular to a method and system for providing support for software partitioning. Still more particularly, the present invention relates to a method and system for supporting software partition and dynamic reconfiguration within a non-uniform memory access system.

[0004] 2. Description of the Prior Art

[0005] With the ever increasing demand for computing power, computer architectures are driven from uniprocessor designs towards multiprocessor designs. While uniprocessor systems are typically limited to processing only a few instructions simultaneously, a multiprocessor system can execute more instructions in parallel with a higher processing speed.

[0006] Within a multiprocessor system, a group of processors is typically defined as a node where each processor within the node may concurrently execute instructions from one or more processes to enable efficient parallel processing of those processes. Some advanced multiprocessor systems may even assign processes to different nodes within the multiprocessor system in order to provide more efficient parallel processing of multiple processes.

[0007] In a tightly coupled multiprocessor system, all processors, which share a single memory system, are typically interconnected by a high-speed, circuit-switched inter-connection network. The processors are also controlled by the same control program and can communicate directly with each other. When an application program spawns several tasks, the operating system may assign these tasks to different processors. For processes that do not generate subprocesses, a multi-tasking operating system can regard all processors of a multiprocessor system as a simple set of computational resources.

[0008] An emerging memory architecture stemming from the class of tightly coupled multiprocessor systems is known as a non-uniform memory access (NUMA) architecture. The NUMA architecture provides overall speed advantages that are not seen in the prior art multiprocessor systems. Also, the NUMA architecture can combine massive scalability of hundreds of processors with the simplified programming model of symmetric multiprocessor (SMP) technology. Generally speaking, a NUMA computer system is a set of SMP nodes interconnected with a high bandwidth interconnection that allows all processors to access any of the main memory within the NUMA computer system. Each processor node shares the same addressable main storage, which is distributed among all the local memory of all the processor nodes. The access time to the local memory within a processor node is the same for all processors within the processor node. Access to a memory on another processor node, however, has a much greater access latency than a similar access to a local memory. Given this greater latency of accesses to the non-local memory, system performance could be enhanced if the memory management facility of the operating system is capable of managing the use of data storage such that the percentage of memory accesses to non-local memories are minimized.

[0009] More precisely, a NUMA computer system can be characterized as an interconnection of a set of SMP nodes, with each SMP node containing:

[0010] a) 0 to N processors;

[0011] b) cache memories, connected individually to each processor and/or to subsets of a node's processors;

[0012] c) a main memory; and

[0013] d) potentially one or more connections to I/O busses and devices.

[0014] The contents of every node's main storage is accessible by all processors within the NUMA computer system. The contents of the main storage in processor caches (or caches of processor subsets) is capable of remaining coherent with all changes made to the contents of any local memory. The term “local” is defined to mean those processors and main memory that are on the same processor node and the term “non-local” as main memory and processors that are on different nodes. The access time (cache linefill latency for example) for a processor to read or write the contents of a main memory that is local tends to be faster than the access time to a non-local main memory. Storage ordering and atomicity can also be maintained.

[0015] The present disclosure is related to a method and system for supporting software partition and dynamic reconfiguration within a NUMA computer system.

[0016] In accordance with the method and system of the present invention, a NUMA computer system includes multiple nodes coupled to an interconnect. Each of the nodes includes a NUMA bridge, a local system memory, and at least one processor having at least a local cache memory. Multiple groups of software partitions are formed within the NUMA computer system, and each of the software partitions is formed by a subset of the nodes. A destination map table is provided in a NUMA bridge of each of the nodes for keeping track of the nodes within a software partition. A command is forwarded to only the nodes within a software partition.

[0017] All objects, features, and advantages of the present invention will become apparent in the following detailed written description.

[0018] The invention itself, as well as a preferred mode of use, further objects, and advantages thereof, will best be understood by reference to the following detailed description of an illustrative embodiment when read in conjunction with the accompanying drawings, wherein:

[0019] FIG. 1 is a block diagram of a multiprocessor system having a non-uniform memory access architecture, in which a preferred embodiment of the present invention may be utilized;

[0020] FIG. 2 is a block diagram of a Destination Map Table, in accordance with a preferred embodiment of the present invention;

[0021] FIG. 3 is a high-level logic flow diagram for supporting software partition within a NUMA computer system, in accordance with a preferred embodiment of the present invention; and

[0022] FIG. 4 is a high-level logic flow diagram for supporting dynamic reconfiguration within a NUMA computer system, in accordance with a preferred embodiment of the present invention.

[0023] Referring now to the drawings and in particular to FIG. 1, there is depicted a block diagram of a shared-memory multiprocessor system having a non-uniform memory access (NUMA) architecture, in which a preferred embodiment of the present invention may be utilized. As shown, a NUMA multiprocessor system 10 includes four nodes, namely, node 11a, node 11b, node 11c, and node 11d. Each of nodes 11a-11dhas at least one processor connected to a local memory within the node via an intra-node connection mechanisms such as a special bus or a crossbar switch. For example, multi-processor node 11acontains processors P₁-P_Nalong with their respective cache memory connected to a main memory 13athat is local to processors P₁-P_Nvia an intra-node bus 12. Each of nodes 11a-11dalso contain an I/O unit for supporting connection to I/O peripherals such as printers, communication links, workstations, or direct access storage devices (DASD). For example, node 11acontains an I/O unit 14a.

[0024] All nodes 11a-11dare interconnected by a Scalable Coherent Interconnect (SCI) 16. SCI 16 is a high-bandwidth interconnection network capable of providing cache coherence throughout NUMA multiprocessor system 10. Each of nodes 11a-11dhas a NUMA bridge, such as a NUMA bridge 15ain node 11a, to provide connections to SCI 16 in order to maintain inter-nodal connection among nodes 11a-11d.

[0025] All processors within NUMA multiprocessor system 10 share an identical addressable main memory, which is distributed among nodes 11a-11das local main memories 13a-13d. Because all local main memories 13a-13dare accessible by all the processors within NUMA multiprocessor system 10, the total addressable main memory space within NUMA multiprocessor system 10 includes the combination of all local main memories 13a-13d. Each byte of system main memory can be addressable by a unique real address. The bus logic for each of nodes 11a-11dmonitors all memory accesses by the processors and the I/O unit within a node and then directs each local memory access to the node's local main memory. Remote accesses to a non-local main memory are sent to SCI 16 via a NUMA bridge 15 within the requesting node.

[0026] Within the NUMA architecture, various multiprocessor nodes can be grouped into different software partitions by an operating system via a process known as software partitioning, as it is well-known to those skilled in the relevant art. Details of the software partitioning process can be found in “System 390 PR/SM Planning Guide” (Order Number: GA22-7236-3), the pertinent of which is incorporated herein by reference.

[0027] Although the exact number of nodes contained within each software partition is application specific, there is typically a maximum number of nodes that can be included within a software partition. For example, a NUMA multiprocessor system may be able to support only up to 256 nodes and only a maximum of 16 nodes can be included in a software partition. Thus, a NUMA bridge of each node needs a method (or hardware logic) to determine which nodes within the NUMA multiprocessor system belongs to its software partition for instructions such as Sync and TLBSync and for command forwarding and error indication forwarding. In addition, a coherency directory within the NUMA bridge also needs a method (or hardware logic) to determine which nodes within the NUMA multiprocessor system are valid to maintain cache coherency after a dynamic reconfiguration. Dynamic reconfiguration is a process of adding or removing a node to a NUMA multiprocessor system while other operations are being performed.

[0028] As a preferred embodiment of the present invention, a destination map table is implemented in each NUMA bridge within a NUMA multiprocessor system for supporting software partitioning and dynamic reconfiguration. The destination map table is preferably initialized by software such as the operating system.

[0029] With reference now to FIG. 2, there is illustrated a block diagram of a destination map table, in accordance with a preferred embodiment of the present invention. As shown, a destination map table 20 includes three fields, namely, a destination identification (ID) field 21, an S field 22, and an E field 23. Destination ID field 21 contains actual node identifications (node IDs) used by a SCI, such as SCI 16 in FIG. 1, connected to all nodes within the NUMA multiprocessor system. The length of the node IDs depends on the total number of nodes a NUMA multiprocessor system is capable of supporting. For example, a NUMA multiprocessor system having 8-bit of node IDs is capable of supporting up to 256 nodes. S field 22 contains a bit for indicating that a destination node (i.e., a remote node) is valid for broadcasts and coherency operations, such as a Sync operation. E field 23 contain a bit for indicating that a destination (i.e., a node) is valid for Machine Check and Checkstop broadcasts to indicate error conditions. The number of entries of destination map table 20 defines a maximum number of nodes within a software partition. For example, a destination map table having 16 entries defines the maximum number of nodes within a software partition to be 16.

[0030] Destination map table 20 is used differently by various logic sections of a NUMA bridge in a node within a NUMA multiprocessor system as follows.

[0031] A. Address Decoding and Routing

[0032] When a command is received by a NUMA bridge from a remote node across an SCI, the Source node ID of the command is sent to a coherency manager of the NUMA bridge, where the command is placed on a local intra-node address bus within a local node. The coherency manager will then compare the Source node ID with each destination ID within destination field 21 of destination map table 20 in order to determine an index that can be utilized as a pseudo node ID for indexing to an entry within a coherency directory of the NUMA bridge. The index of destination map table 20 is the relative position of an entry having a destination ID that matches the Source node ID. For example, if the fifth entry of destination map table 20 contains a destination ID that matches with the Source node ID of the command, then the index that can be utilized as a pseudo node ID for indexing to an entry within the coherency directory of the NUMA bridge is “5.”

[0033] As for routing, when the NUMA bridge determines that an operation must be forwarded to a remote node, the address range decode will include the index (i.e., the pseudo node ID) for indexing into destination map table 20. The command will subsequently be forwarded to a remote node that is identified by the destination ID according to the index to an entry within destination map table 20.

[0034] B. Software Partitioning

[0035] As mentioned above, each NUMA bridge within the NUMA multiprocessor system contains a destination map table; and each destination map table keeps track of all nodes within its software partition by setting the S field associated with the destination ID field of the nodes valid. When a Sync or TLBSync command is received from the local interconnect, the NUMA bridge must interrogate each S bit in S field 22 within destination map table 20. An Output Request Manager for the SCI will then forward the command only to those remote nodes that have their respective S bit set in S field 22 within destination map table 20 of the sending node.

[0036] If the S bit in S field 22 is not set for the corresponding node, or if the Source node ID does not match any of the Destination ID in destination ID field 21 of destination map table 20, then the cache coherency directory will not maintain cache information for that node. Remote nodes that are not within the software partition may access the local memory and receive the latest coherent data, however, their access of the data will not be recorded in the cache coherency directory of the NUMA bridge.

[0037] Similarly, if a forwardable Machine Check or Checkstop condition is received, the error handling logic within the NUMA bridge must also interrogate each E bit in E field 23 within destination map table 20. The Output Request Manager for the SCI will then forward the error indications only to those destination nodes that have respective E bit set in E field 23 within destination map table 20 of the sending node.

[0038] Referring now to FIG. 3, there is illustrated a high-level logic flow diagram for supporting software partitions within a NUMA computer system, in accordance with a preferred embodiment of the present invention. Starting at block 30, a command is received by a node from the SCI, as shown in block 31. The received command can be a request or an error condition. The received command is then forward to a local bus (such as intra-node 12 in FIG. 1), as depicted in block 32. Subsequently, necessary coherency operations are performed to allow the received command to proceed, as illustrated in block 33. Each destination ID with the destination ID field of the destination map table (from FIG. 2) is then interrogated to determine whether or not there is a “match” with the node ID(s) of the received command and whether or not the S bit of the matched node ID(s) is set, as depicted in block 34. If there is a match with the node ID(s) of the received command and the S bit is set for the matched node ID, the coherency directory is update to indicate ownership of the corresponding cache line for the matched node ID(s), as illustrated in block 35; otherwise, the process is completed.

[0039] C. Dynamic Reconfiguration

[0040] Dynamic reconfiguration support requires the ability to remove or add a node to a NUMA multiprocessor system during the performance of other operations. When a node is removed, the coherency directories at other nodes within the NUMA multiprocessor system may contain erroneous information indicating that the removed node has modified or shared data. When removing a node, the cache memories of that node must be “flushed,” by the operating system, and the S bit of that node must be turned off, also by the operating system, in destination map table 20 of all the other nodes within its software partition. Incidentally, when the coherency manager on another node reads an entry in the cache coherency directory, the coherency manager will disregard and clear any node information for nodes that do not have their respective S bit set. Otherwise, coherency operations may be forwarded to a non-existent node, forcing an error condition.

[0041] With reference now to FIG. 4, there is illustrated a high-level logic flow diagram for supporting dynamic reconfiguration within a NUMA computer system, in accordance with a preferred embodiment of the present invention. Starting at block 40, a determination is made as to whether or not a node is being removed from or added to a software partition, as shown in block 41. If a node is being removed from the software partition, the S bit of the entry of the corresponding destination ID in the destination map table of each node in the same software partition is turned off, as depicted in block 42. If a node is being added to the software partition, a destination ID of the multiprocessor mode is added to the destination map table of each node in the same software partition and the corresponding S bit is set, as illustrated in block 43.

[0042] As has been described, the present invention provides an improved method and system for supporting software partition and dynamic reconfiguration within a NUMA multiprocessor system.

[0043] It is also important to note that although the present invention has been described in the context of a fully functional computer system, those skilled in the art will appreciate that the mechanisms of the present invention are capable of being distributed as a program product in a variety of forms, and that the present invention applies equally regardless of the particular type of signal bearing media utilized to actually carry out the distribution. Examples of signal bearing media include, without limitation, recordable type media such as floppy disks or CD ROMs and transmission type media such as analog or digital communications links.

[0044] While the invention has been particularly shown and described with reference to a preferred embodiment, it will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the invention.