Parallel processor system including a cache memory subsystem that has independently addressable local and remote data areas

[0002] This is a continuation application of U.S. Ser. No. 08/497,751, filed Jul. 3, 1995, now U.S. Pat. No. 5,778,429.

[0003] 1. Field of the Invention

[0004] The present invention relates to a parallel processor system having a plurality of cache memories and a shared memory, and more particularly to such a parallel processor system in which the shared memory is constituted by plural distributed shared memories.

[0005] 2. Description of the Related Art

[0006] Increasingly, improvements in parallel processor technology are focusing on communication among the processors to improve computer performance. Two primary communication means are the message passing method, in which processors exchange messages via a network, and the shared memory method, in which each processor accesses a shared memory.

[0007] Processors employing the message passing method generally exchange messages by starting an operating system. However, starting the operating system constitutes a large overhead, especially for communicating short messages.

[0008] In the shared memory method, on the other hand, communication may take place without starting the operating system. Therefore, the shared memory method alleviates this burden on communication.

[0009] A distributed shared memory method, which divides and distributes the shared memory, has proven to be an effective method for large-scale parallel processor systems. Distributing the shared memory allows simultaneous access to the shared memory by a plurality of processors, realizing a high level of parallel processing. An example of a distributed shared memory, in which nodes comprising a processor and a part of the shared memory are interconnected by a network, is disclosed in Japan Patent Laid-Open No. 89056/1993.

[0010] In this distributed shared memory method, when processors access the shared memory, data are transferred with high probability through the network. However, compared to the fast processing speed of the processors, the network speed is slow. Hence, the delay time through the network poses a problem for overall access speed.

[0011] A technique for speeding the access to a shared memory over a network provides a cache memory at each node. The cache memory is typically a small-capacity, high-speed buffer for registering the contents of part of the shared memory. Examples of distributed shared memories that use cache memory include Baylor et al, U.S. Pat. No. 5,313,609, and Lenoski et al, “The Stanford Multiprocessor,” Computer(March 1992), pp. 63-79.

[0012] These distributed shared memories that use cache memory are particularly characterized in that data in the shared memory in the same node (local data) and data in the shared memory in other nodes (remote data) are recorded in the same cache memory.

[0013] The distributed shared memory system divides large-scale array data and distributes them among the shared memory in each node. In array computations, each processor uses local data and remote data and performs calculations in parallel. The amount of remote data used in the array computation is generally enormous. Thus, when the large-scale array computation is performed by the distributed shared memory system that has the cache memory, the remote data used for the calculation cannot be fully accommodated in the cache memory, and are flushed out.

[0014] The local data includes, aside from the array data, those data which, though limited in quantity, are used the most often, such as a variety of system variables used by the operating system. Such data are desirably registered in the cache memory at all times to optimize overall system performance.

[0015] As stated above, remote data may be flushed from the cache memory when the capacity of the cache memory is insufficient for handling large-scale array computations. Similarly, the local data that is most frequently used may also be flushed out during large-scale array calculations due to the cache overflow resulting from accessing data at remote nodes. Loss of these frequently-used local data degrades the system performance.

[0016] The present invention solves this problem by controlling access to the distributed shared memory and to the plural cache memories to prevent frequently-used local data from being flushed out of a cache memory, which would otherwise occur due to cache overflow with remote data according to the systems described above.

[0017] The present invention includes, in a parallel processor system, a plurality of nodes each including a processor and a shared memory in a distributed shared memory arrangement, and a local-remote divided cache memory system, wherein local data and remote data are controlled separately. Each local-remote divided cache memory system includes a local data area, a remote data area, and a cache memory controller by which either the local data area or the remote data area is accessed according to the contents of an access request other constituent parts of the parallel processor system constructed according to the teachings of the present invention will be described in greater detail below.

[0018] FIG. 1 is a block diagram showing a configuration of a parallel processor system constructed according to the teachings of the present invention, including a local-remote divided cache memory subsystem;

[0019] FIG. 2A is a block diagram showing details of the local-remote divided cache memory subsystem of FIG. 1;

[0020] FIG. 2B illustrates a method of merging addresses as performed by the address merge circuit contained in the embodiment shown in FIG. 1;

[0021] FIG. 3 describes the address space employed according to the teachings of the invention;

[0022] FIG. 4 illustrates a method of determining a local data access according to the teachings of the invention;

[0023] FIG. 5 is a block diagram of another embodiment of a parallel processor system according to the present invention, wherein local and remote data cache memory subsystems are separately employed;

[0024] FIG. 6 is a block diagram of an access discriminating circuit employed by the embodiments shown in FIGS. 5A and 5B;

[0025] FIG. 7 is a block diagram of a local data cache memory subsystem that may be employed in either of the embodiments shown in FIGS. 5A and 5B;

[0026] FIG. 8 is a block diagram illustrating a remote data cache memory subsystem that may be employed in either of the embodiments shown in FIGS. 5A and 5B;

[0027] FIG. 9 illustrates another configuration of a parallel processor system according to the present invention;

[0028] FIG. 10 shows a local data cache memory subsystem according to the embodiment shown in FIG. 9; and

[0029] FIG. 11 shows a remote data cache memory subsystem according to the embodiment shown in FIG. 9.

[0030] The parallel processor system of FIG. 1 includes a plurality of nodes 10, 110, . . . , and a network 200 that joins them. For the purposes of this description, each node is considered to have the same components, although variations that do not deviate from the basic teachings of the invention may be made.

[0031] Each node 10, 100 has a processor 20, 120 and a shared memory 60, 160 connected to the network 200 via remote access request circuit 80, 180 and remote access reply circuit 90, 190, respectively. The processor 20, 120 is connected to its shared memory 60, 160 via a local-remote divided cache memory subsystem 30, 130, which contains a local data area 40, 140 separately from a remote data area 70, 170. As the names suggest, local data area 40, 140 stores local data, and remote data area 70, 170 stores remote data. Access to the local data area 40, 140 is controlled separately from access to the remote data area 70, 170.

[0032] FIG. 2A shows a preferred embodiment of the local-remote divided cache memory subsystem 30 of node 10. As noted above, the components of the other nodes, such as node 110, are desirably the same as those of node 10.

[0033] A data signal entering the local-remote divided cache memory subsystem 30 on data line 520 is directed to a node number register 330, a local memory access circuit 390, and a remote memory access circuit 400. As described more fully below, the data signal contains local node number information which, in combination with a mask 310, and a comparator 320, is used to determine whether the local memory access circuit 390 or the remote memory access circuit 400 is to be activated. If the data are to be transmitted to or from the local memory, then the local memory access circuit 390 communicates directly with the shared memory 60; however, if the data are to be transferred to or from a remote memory, then the remote memory access circuit 400 is activated to connect to a remote access request circuit 80, as shown in FIG. 1.

[0034] The local-remote divided cache memory subsystem 30 decides whether the local or a remote memory is to be accessed. An address control signal enters the local-remote divided cache memory subsystem 30 on address line 510, and is sent to the node number register 330, mask 310, an address merge circuit 370, local memory access 390, and remote memory access 400. A comparator 320 receives outputs from the mask 310 and from the node number register 330, from which it decides whether local or remote access is required. The decision is output on local-remote decision line 540 to the address merge circuit 370, and to either the local memory access 390 or the remote memory access 400. The signal received by the local memory access 390 or the remote memory access 400 is an activation signal that determines a source or destination of the data on data line 520.

[0035] If the signal on decision line 540 indicates that local memory access is required, the cache controller 340 is controlled by the merged address on line 570 to access local data area 40 in the divided cache memory via an address array 350. If remote data are concerned, then the cache controller 340 accesses the remote data area 70 via the address array 350 and the cache control signal line 550.

[0036] In the event that the accessed data area of the divided cache does not contain the data sought (i.e., if the data are not registered in the accessed data area), the cache controller 340 issues a memory access request signal on line 560 to the local memory access circuit 390 or the remote memory access circuit 400, depending on whether local or remote data are concerned.

[0037] FIG. 2B illustrates the preferred address merge process performed by the address merge circuit 370. Assuming a 1-megabyte direct-map cache memory addressed by 31 bits, the system illustratively assigns 19 lower-order bits for the address sought in the address array 350, and 12 higher-order bits as the address of the data area (local or remote) containing the requested data. The address merge circuit 370 embeds a single bit between the 19 lower-order and the 12 higher-order bits as a flag indicating whether the local data area 40 or the remote data area 70 is to be addressed.

[0038] An example of accessing the local data area 40 is shown in FIG. 2B. The embedded bit is a 1 bit, and as shown is embedded between the 12 higher-order bits and the 19 lower-order bits. The 32-bit merged address thus generated is output on the merged address line 570 to the cache controller 340, which then puts the merged address 572 onto the cache memory control signal line 550 to the address array 350.

[0039] Since the embedded bit is a logic 1, the lower-order 20 bits (including the embedded bit) of the merged address 572 represents the location sought in the address array 350. If, as a result of the search, the system determines that the requested data is not registered in the local-remote divided data cache memory constituted by the local data area 40 and the remote data area 70, the cache memory controller 340 is notified via the cache memory control signal line 550.

[0040] Then, the cache memory controller 340 issues a load request for memory to the local memory access circuit 390 and the remote memory access circuit 400 through a memory access request signal line 560. Whichever one of the local memory access circuit 390 and the remote memory access circuit 400 has been activated by the signal on the local-remote decision signal line 540 outputs the load request address received via t he address signal line 510. If the local-remote decision signal line 540 contains the bit indicating that local memory is to be accessed, a signal output from the activated local memory access circuit 390 is sent to the shared memory 60 through a selector 50. However, if the signal on the local-remote decision signal line 540 indicates that remote memory is to be accessed, then the remote memory access circuit 400 outputs the load request address to the remote access request circuit 80. Each of the scenarios will be discussed in turn.

[0041] In the former example, the shared memory 60 returns reply data for the load request to the local memory access circuit 390. The reply data are sent to the processor 20 via the data line 520 . At the same time, the cache memory controller 340 outputs a request to register the reply data to the address array 350 via the cache memory control signal line 550.

[0042] When the address array 350 receives the registration request, it erases part of the data in the local data area 40 (i.e., one of the entries in the local data area 40 is erased), newly registers the reply data sent via the data line 530 in the local data area 40, and updates information in the address array 350 reflecting the registration. In this way, the loading of local data that are not registered in the local data area 40 corresponding to the higher-order 512 kilobytes of cache memory does not affect the remote data area 70 that corresponds to the lower-order 512 kilobytes of cache memory.

[0043] FIG. 3 illustrates an example of the address space employed by the invention. As shown, eight nodes, each having 256 megabytes of shared memory, are interconnected by a network. In this example, each processor that employs the distributed shared memory system can access the shared memory of any node from two gigabytes of area 580 having an address in the range 00000000 to 7FFFFFFF. The address space contains an I/O access area 590 in a higher order region, in which exists an address 600 for accessing the node number register 330. The node number register 330, as shown in FIG. 2A, illustratively resides in the local-remote divided cache memory subsystem 30, 130 and stores the local node number.

[0044] FIG. 4 illustrates more particularly the properties of different blocks of address bits. The most significant bit is used to determine whether the access is a memory access or an I/O access, and the following three bits represent the node number of the access destination if the access request is for memory access. Thus, a logical conjunction of the load request address 610 and the information 620 stored in the mask 310 (FIG. 2A) is used to derive the node number 630. The node number 630 is compared with information 640 contained in the node number register 330 by the comparator 320. When the numbers agree, the load request is recognized as a request for local data access. When the numbers do not agree, the system recognizes the load request as being for remote data access.

[0045] Another embodiment of the present invention is shown in FIG. 5. FIG. 5 illustrates a plural processor system that utilizes a distributed shared memory system in conjunction with separate local and remote data cache memory subsystems that are independently addressable but do not form part of the same memory as in the embodiment disclosed above.

[0046] In the instant embodiment, a plurality of nodes 15, 115, . . . , are connected through a network 205 much as the nodes of the embodiment discussed above are connected. Again, only node 15 will be described, although it is assumed that each node will be similarly constituted to the end of operating similarly to achieve the objectives of the invention.

[0047] Thus, each node 15, 115 has a processor 25, 125, a shared memory 65, 165, a local data cache memory subsystem 45, 145, and a remote data cache memory subsystem 75, 175. Illustratively, the network 205 joins a total of eight nodes having such a configuration.

[0048] FIG. 6 explains in more detail the functional operation of the access discriminate circuit 35, 135, which includes a node number register 335 for storing a local node number, much as the node number register 330 stores a local node number in the previous embodiment.

[0049] When a data signal is received on data line 525 by the access discriminate circuit 35, the data signal is sent to the node number register 335, and to local data latch 395 and remote data latch 405. Broadly, latches 395 and 405 operate similarly to the local and remote memory access circuits 390 and 400 of the previous embodiment shown in FIG. 2A in that, when activated, the respective latch 395 or 405 transfers data to or from a location external to the access discriminate circuit 35. However, because the local and remote data cache memory subsystems are separately disposed in this embodiment, latches 395 and 405 are respectively connected to local data cache 45 and remote data cache 75 to selectively permit data transfer from one of the cache memory subsystems.

[0050] The node number originally set by data obtained from data line 525 is compared with the same three address bits, illustratively, contained in the address received on address line 515, employing a mask 315, comparator 325 and the node number register 335 in a similar manner as in the previous embodiment. However, according to the present embodiment, the result of the comparison is output on remote-local decision signal line 545 directly to local latch 395 and remote latch 405 to selectively activate one or the other (note that latch 405 illustratively inverts the remote-local decision signal at an input). If the local latch 395 is activated (i.e., if the remote-local decision signal indicates that local data access is necessary), then data are transferred to or from local data cache memory subsystem 45. If remote latch 405 receives the activated signal, then data are transferred to or from remote data cache memory subsystem 75.

[0051] FIG. 7 shows an example of the operation of local data cache 45. Local data cache memory subsystem 45 preferably includes a cache memory controller 343, address array 353, local cache memory 363, and memory access circuit 393 by which the shared memory 65 is accessed if the data required by the access request are not stored in the local cache memory 363. Thus, similarly to the previous embodiment, an address is received on address line 513 by the cache memory controller 343, which outputs a cache memory control signal on line 553 to address array 353. The address array 353 provides the location in local cache memory 363 that is sought by the address signal. If the data are found not to be registered in cache memory 363, a signal is returned to the cache memory controller 343 along cache memory control signal 553, and a memory access request signal is output on line 563 to the memory access circuit 393, which in turn accesses the shared memory 65. In a preferred embodiment, cache memory controller 343, address array 353, and memory access circuit 393, along with their interconnecting lines, are provided on a single chip.

[0052] FIG. 8 illustrates an example of the operation of a remote data cache memory subsystem 75. The operation is similar to that of FIG. 7, except that the address received on line 516 is routed by cache memory controller 346 to address array 356 along cache memory control signal line 556, to selectively access a location in remote cache memory 366. If the data are not registered in remote cache memory 366, a signal is returned to the cache memory controller 346, which outputs a signal to remote memory access circuit 396 along remote memory access request signal line 566, so as to connect to remote access request circuit 85.

[0053] FIG. 9 shows another embodiment of the invention that is functionally similar to the functional diagram of FIG. 5, but which can use a noncustomizied, currently-available processor which has only one data pin.

[0054] In the embodiment shown in FIG. 9, node 15′ contains processor 25′, which is connected to address line 515 and data line 525 by a single wireing line each. In turn, local data cache subsystem 45′ and remote data cache subsystem 75′ are each connected to the address and data lines, for transferring data between the processor and the shared memory 65 and between the processor and a remote node, respectively. The respective connections between the local data cache subsystem 45′ and the shared memory 65, and between the remote data cache subsystem 75′ and a remote node (via a network, not shown), are the same as shown in FIG. 5.

[0055] However, because the processor 25′ has a single data pin, the instant embodiment incorporates an access discriminate circuit 35 into each of the local data cache subsystem 45′ and remote data cache subsystem 75′. See FIGS. 10 and 11, respectively. The access discriminate circuits 35 and each of the cache subsystems can be the same as that shown in FIG. 6, discussed above, except that only one latch is necessary in each access discriminate circuit 35′. In other words, each of the access discriminate circuits 35′ determines, from the input address and data, whether to activate its latch. If local data are to be accessed, then the latch permitting transfer to or from the local data cache 45′ is activated. If remote data are to be accessed, then the latch permitting transfer to or from the remote data cache 75′ is activated.

[0056] To further illustrate the operation of a parallel processor system constructed according to the teachings disclosed above, the following examples are proposed.

[0057] For the system embodiments shown in FIGS. 1 and 2A, when the processor 20 loads local data present in its node 10, the following cases are considered.

[0058] (1-1)

[0059] In this example, the local data to be loaded are not registered in the local data area 40 in the local-remote divided cache memory 30.

[0060] When a load request is output from the processor 20 to the local-remote divided cache memory subsystem 30 shown in FIG. 2A, the local-remote divided cache memory subsystem 30 checks the load request address to determine whether the request is for local data or remote data. This decision is made in accordance with the operations of the mask 310, comparator 320, and node number register 330.

[0061] With further reference to FIG. 4, the load request address 610 and the mask information 620 are logically ANDed to derive a node number 630 from the load request address 610. The node number 630 derived from the address is compared with the node number information 640 stored in the node number register 330, using the comparator 320. In this example, local data are to be loaded, so the two node numbers agree, and the system recognizes the request as being for local data access.

[0062] Thus, the comparator 320 outputs a logic 1 to the address merge circuit 370 via the local-remote decision signal line 540, and at the same time activates the local memory access circuit 390. The load request is also input by the address merge circuit 370 and the local memory access circuit 390 via the address signal line 510.

[0063] In the address merge circuit 370, the logic 1 bit output on line 540 is embedded at bit position 20 in the merged address 572, as shown in FIG. 2B. The merged address is output on line 570 to the cache controller 340, which places the merged address 572 onto the cache control signal line 550 to the address array 350.

[0064] Because the embedded bit is a logic 1, the addressed range is 80000-FFFFF in hexadecimal, defining the address range corresponding to the memory locations in local data area 40. In other words, the higher-order 512 kilobytes of the 1-megabyte direct map cache memory are assigned to the local data area 40, while the lower-order 512 kilobytes are assigned to the remote data area 70.

[0065] Since in this example, the requested data are not registered in the local data area 40, the cache memory controller 340 is notified via the cache memory control signal line 550, and outputs a load request for memory to the local memory access circuit 390 via the memory access request signal line 560. The remote memory access 400 also receives the request, but since only the local memory access circuit 390 is activated by the local-remote decision bit on line 540, no action is taken by the remote memory access circuit 400.

[0066] Instead, the local memory access circuit 390 outputs the load request address presented on the address signal line 510 to the shared memory 60 via the selector 50. See FIG. 1. Then, the shared memory 60 ret urns the reply data corresponding to the load request to the local memory access circuit 390, which relays the reply data to the processor 20 on the data line 520. At the same time, the cache controller 340 outputs a request for registering the reply data to the address array 350 through the cache memory control signal line 550.

[0067] Upon receipt of the registration request, the address array 350 erases data in the corresponding part of local data area 40 (i.e., erases one of the entries in the local data area 40), newly registers the reply data sent via the data line 530 i n the local data area 40, and updates the address array 350. In this way, the loading of local data that was not previously registered in the local data area 40 corresponding to the higher-order 512 kilobytes of cache memory does not affect the remote data area 70 corresponding to the lower-order 512 kilobytes of cache memory.

[0068] (1-2)

[0069] In this example, the local data to be loaded are registered in the local data area 40 in the local-remote divided cache memory 30.

[0070] When a load request is output from the processor 20 to the local-remote divided cache memory subsystem 30 shown in FIG. 2A, the local-remote divided cache memory subsystem 30 checks the load request address to determine whether the request is for local data or remote data. This decision is made in the same manner as for example (1-1) above.

[0071] When the comparator 320 outputs a logic 1 to the address merge circuit 370 via the local-remote decision signal line 540 (indicating that the request concerns local data), it also activates the local memory access circuit 390. As before, the load request is also input by the address merge circuit 370 and the local memory access circuit 390 via the address signal line 510.

[0072] In the address merge circuit 370, the logic 1 bit output on line 540 is again embedded at bit position 20 in the merged address 572, as shown in FIG. 2B. The merged address is output on line 570 to the cache controller 340, which places the merged address 572 onto the cache control signal line 550 to the address array 350. Because the embedded bit is a logic 1, the address is defined in the address range corresponding to the memory locations in local data area 40. Thus, the cache memory controller 340 searches through the address array 350 corresponding to those memory locations.

[0073] Since in this example, the requested data are registered in the local data area 40, the cache memory controller 340 is notified via the cache memory control signal line 550, and the registered data are transferred to the processor 20 through the data lines 530, 520. In this way, the loading of the local data registered in the local data area 40 does not affect the remote data area 70.

[0074] For the system embodiments shown in FIGS. 1 and 2A, when the processor 20 stores local data present in its node 10, the following cases are considered.

[0075] (2-1)

[0076] In this example, the local data to be stored are not registered in the local data area 40 in the local-remote divided cache memory 30.

[0077] When a store request is output from the processor 20 to the local-remote divided cache memory subsystem 30 shown in FIG. 2A, the local-remote divided cache memory subsystem 30 checks the store request address to determine whether the request concerns local data or remote data. As for example (1-1), this decision is made in accordance with the operations of the mask 310, comparator 320, and node number register 330.

[0078] When the comparator 320 outputs a logic 1 to the address merge circuit 370 via the local-remote decision signal line 540, it also activates the local memory access circuit 390. The store request is also input by the address merge circuit 370 and the local memory access circuit 390 via the address signal line 510.

[0079] The merged address 572 is formed as described above, and is output on line 570 to the cache controller 340, which places the merged address 572 onto the cache control signal line 550 to the address array 350. Because the embedded bit is a logic 1, the address is defined in the address range corresponding to the memory locations in local data area 40. Thus, the cache memory controller 340 searches through the address array 350 corresponding to those memory locations.

[0080] Since in this example, the requested data are not registered in the local data area 40, the cache memory controller 340 is notified via the cache memory control signal line 550, and outputs a store request for memory to the local memory access circuit 390 via the memory access request signal line 560. The remote memory access 400 also receives the request, but since only the local memory access circuit 390 is activated by the local-remote decision bit on line 540, no action is taken by the remote memory access circuit 400.

[0081] Instead, the local memory access circuit 390 outputs the store request address presented on the address signal line 510, along with the data received on line 520, to the shared memory 60 via the selector 50, as described above with respect to Example (1-1). However, instead of returning the data for storage in the cache memory, the data are stored in the shared memory 60. In this way, the storing of local data that were not previously registered in the local data area 40 does not affect the remote data area 70.

[0082] (2-2)

[0083] In this example, the local data to be stored are registered in the local data area 40 in the local-remote divided cache memory 30.

[0084] When a store request is output from the processor 20 to the local-remote divided cache memory subsystem 30 shown in FIG. 2A, the local-remote divided cache memory subsystem 30 checks the store request address to determine whether the request concerns local data or remote data. This decision is made in the same manner as for example (1-1) above.

[0085] When the comparator 320 outputs a logic 1 to the address merge circuit 370 via the local-remote decision signal line 540 (indicating that the request concerns local data), it also activates the local memory access circuit 390. As before, the store request is also input by the address merge circuit 370 and the local memory access circuit 390 via the address signal line 510.

[0086] The merged address 572 is again formed in the address merge circuit 370 and output on line 570 to the cache controller 340, which places the merged address 572 onto the cache control signal line 550 to the address array 350. Because the embedded bit is a logic 1, the address is defined in the address range corresponding to the memory locations in local data area 40. Thus, the cache memory controller 340 searches through the address array 350 corresponding to those memory locations.

[0087] Since in this example, the requested data are registered in the local data area 40, the cache memory controller 340 is notified via the cache memory control signal line 550. The cache memory controller 340 then issues a request for eliminating registration of the store-requested data to the address array 350. In response to the request, the address array 350 deletes the registration information on the stored data from the local data area 40.

[0088] Further, the cache memory controller 340 sends a store request for memory to the local memory access circuit 390 and the remote memory access circuit 400 through the memory access request signal line 560. The remote memory access circuit 400 is not activated, but the local memory access circuit 390 outputs the store request address presented on the address signal line 510 and the store data presented on line 520 to the shared memory 60 via selector 50. The shared memory 60 updates data according to the request. In this way, the storing of the local data registered in the local data area 40 does not affect the remote data area 70.

[0089] For the system embodiments shown in FIGS. 1 and 2A, when the processor 20 loads remote data present in another node, the following cases are considered.

[0090] (3-1)

[0091] In this example, the remote data to be loaded are not registered in the remote data area 70 in the local-remote divided cache memory 30.

[0092] When a load request is output from the processor 20 to the local-remote divided cache memory subsystem 30 shown in FIG. 2A, the local-remote divided cache memory subsystem 30 checks the load request address to determine whether the request is for local data or remote data. This decision is made in accordance with the operations of the mask 310, comparator 320, and node number register 330.

[0093] However, unlike the previous Example (1-1), the comparison of node numbers 630 and 640 respectively derived from the address and the node number register 330 yields a false result. In this example, then, remote data are to be loaded, and the system recognizes the request as being for remote data access as a result of the comparison.

[0094] Thus, the comparator 320 outputs a logic 0 to the address merge circuit 370 via the local-remote decision signal line 540, and at the same time activates the remote memory access circuit 400. The load request is also input by the address merge circuit 370 and the remote memory access circuit 390 via the address signal line 510.

[0095] In the address merge circuit 370, the logic 0 bit output on line 540 is embedded at bit position 20 in the merged address 572. The merged address is output on line 570 to the cache controller 340, which places the merged address 572 onto the cache control signal line 550 to the address array 350.

[0096] Because the embedded bit is a logic 0, the addressed range is 00000-7FFFF in hexadecimal, defining the address range corresponding to the memory locations in remote data area 70. In other words, the lower-order 512 kilobytes of the 1-megabyte direct map cache memory are assigned to the remote data area 70, as previously mentioned.

[0097] Since in this example, the requested data are not registered in the remote data area 70, the cache memory controller 340 is notified via the cache memory control signal line 550, and outputs a load request for memory to the remote memory access circuit 400 via the memory access request signal line 560. The local memory access 390 also receives the request, but since only the remote memory access circuit 400 is activated by the local-remote decision bit on line 540, no action is taken by the local memory access circuit 390.

[0098] Instead, the remote memory access circuit 400 outputs the load request address presented on the address signal line 510 to the remote access request circuit 80. See FIG. 1. Then, the remote access request circuit 80 generates a remote load request message. It further derives a node number from the load request address by using a method similar to that shown in FIG. 4, and sends the remote load request message to a remote access reply circuit of the node having the derived node number via the network 200.

[0099] The remote access reply circuit (such as circuit 190 in node 110) loads the data from the shared memory of the remote node, generates a remote load reply message and returns the message to the remote access request circuit 80 through the network 200.

[0100] Back at the requesting node 10, the remote access request circuit 80 receives the reply message, derives the reply data corresponding to the load request, and returns the reply data to the remote memory access circuit 400. The reply data are transferred to the processor 20 through the data line 520, and the cache memory controller 340 outputs a reply data registration request to the address array 350 via the cache memory control signal line 550.

[0101] Upon receiving the registration request, the address array 350 erases part of the data in the remote data area 70, newly registers the reply data obtained from the data line 530 in the remote data area 70, and updates the address array 350. In this way, the loading of remote data not registered in the remote data area 70 does not affect the local data area 40.

[0102] (3-2)

[0103] In this example, the remote data to be loaded are registered in the remote data area 70 in the local-remote divided cache memory 30.

[0104] When a load request is output from the processor 20 to the local-remote divided cache memory subsystem 30 shown in FIG. 2A, the local-remote divided cache memory subsystem 30 checks the load request address to determine whether the request is for local data or remote data. This decision is made in the same manner as for example (3-1) above.

[0105] When the comparator 320 outputs a logic 0 to the address merge circuit 370 via the local-remote decision signal line 540 (indicating that the request concerns remote data), it also activates the remote memory access circuit 400. As before, the load request is also input by the address merge circuit 370 and the remote memory access circuit 400 via the address signal line 510.

[0106] In the address merge circuit 370, the logic 0 bit output on line 540 is again embedded at bit position 20 in the merged address 572. The merged address is output on line 570 to the cache controller 340, which places the merged address 572 onto the cache control signal line 550 to the address array 350. Because the embedded bit is a logic 0, the address is defined in the address range corresponding to the memory locations in remote data area 70. Thus, the cache memory controller 340 searches through the address array 350 corresponding to those memory locations.

[0107] Since in this example, the requested data are registered in the remote data area 70, the cache memory controller 340 is notified via the cache memory control signal line 550, and the registered data are transferred to the processor 20 through the data lines 530, 520. In this way, the loading of the remote data registered in the remote data area 70 does not affect the local data area 40.

[0108] For the system embodiments shown in FIGS. 1 and 2A, when the processor 20 stores remote data present in another node, the following cases are considered.

[0109] (4-1)

[0110] In this example, the remote data to be stored are not registered in the remote data area 70 in the local-remote divided cache memory 30.

[0111] When a store request is output from the processor 20 to the local-remote divided cache memory subsystem 30 shown in FIG. 2A, the local-remote divided cache memory subsystem 30 checks the store request address to determine whether the request concerns local data or remote data. As for example (3-1), this decision is made in accordance with the operations of the mask 310, comparator 320, and node number register 330.

[0112] When the comparator 320 outputs a logic 0 to the address merge circuit 370 via the local-remote decision signal line 540, it also activates the remote memory access circuit 400. The store request is also input by the address merge circuit 370 and the remote memory access circuit 400 via the address signal line 510.

[0113] The merged address 572 is formed as described above, and is output on line 570 to the cache controller 340, which places the merged address 572 onto the cache control signal line 550 to the address array 350. Because the embedded bit is a logic 0, the address is defined in the address range corresponding to the memory locations in remote data area 70. Thus, the cache memory controller 340 searches through the address array 350 corresponding to those memory locations.

[0114] Since in this example, the requested data are not registered in the remote data area 40, the cache memory controller 340 is notified via the cache memory control signal line 550, and outputs a store request for memory to the remote memory access circuit 400 via the memory access request signal line 560. The local memory access 390 also receives the request, but since only the remote memory access circuit 400 is activated by the local-remote decision bit on line 540, no action is taken by the local memory access circuit 390.

[0115] Instead, the remote memory access circuit 400 outputs the store request address presented on the address signal line 510, along with the data received on line 520, to the remote access request circuit 80. The remote access request circuit 80, in response to the request from the remote memory access circuit 400, generates a remote store request message that contains the store request address and data, and further derives a node number from the store request address by using a method similar to that shown in FIG. 4. Then, the remote access request circuit 80 sends the remote store request message to a remote access reply circuit (such as circuit 190) of the node having the node number via the network 200.

[0116] At the remote node, the remote access reply circuit stores the data into the shared memory of the remote node, generates a remote load reply message and returns the message to the remote access request circuit 80 through the network 200. In this way, the storing of remote data not registered in the remote data area 70 does not affect the local data area 40.

[0117] (4-2)

[0118] In this example, the remote data to be stored are registered in the remote data area 70 in the local-remote divided cache memory 30.

[0119] When a store request is output from the processor 20 to the local-remote divided cache memory subsystem 30 shown in FIG. 2A, the local-remote divided cache memory subsystem 30 checks the store request address to determine whether the request concerns local data or remote data. This decision is made in the same manner as for example (4-1) above.

[0120] When the comparator 320 outputs a logic 0 to the address merge circuit 370 via the local-remote decision signal line 540 (indicating that the request concerns remote data), it also activates the remote memory access circuit 400. As before, the store request is also input by the address merge circuit 370 and the remote memory access circuit 400 via the address signal line 510.

[0121] The merged address 572 is again formed in the address merge circuit 370 and output on line 570 to the cache controller 340, which places the merged address 572 onto the cache control signal line 550 to the address array 350. Because the embedded bit is a logic 0, the address is defined in the address range corresponding to the memory locations in remote data area 70. Thus, the cache memory controller 340 searches through the address array 350 corresponding to those memory locations.

[0122] Since in this example, the requested data are registered in the remote data area 70, the cache memory controller 340 is notified via the cache memory control signal line 550. The cache memory controller 340 then issues a request for eliminating registration of the store-requested data to the address array 350. In response to the request, the address array 350 deletes the registration information on the stored data from the local data area 40.

[0123] Further, the cache memory controller 340 sends a store request for memory to the local memory access circuit 390 and the remote memory access circuit 400 through the memory access request signal line 560. The local memory access circuit 390 is not activated, but the remote memory access circuit 400 outputs the store request address presented on the address signal line 510 and the store data presented on line 520 to the remote access request circuit 80. The remote access request circuit 80, in response to the request, generates a remote store request message that contains the store request address and the data. It further derives a node number from the store request address by using a method similar to that shown in FIG. 4, and sends the remote store request message to the remote access reply circuit of the remote node having the derived node number. The remote access reply circuit then outputs the received address and data to its shared memory, which is updated according to the request. In this way, the storing of the remote data registered in the remote data area 70 does not affect the local data area 40.

[0124] For the system embodiments shown in FIGS. 5-8, when the processor 25 loads local data present in its node 15, the following cases are considered.

[0125] (5-1)

[0126] In this example, the local data to be loaded are not registered in the local data cache 45.

[0127] When a load request is output from the processor 25 to the access discriminate circuit 35 shown in FIG. 6, the access discriminate circuit 35 checks the load request address to determine whether the request is for local data or remote data. As for Example 1, this decision is made in accordance with the operations of the mask 310, comparator 320, and node number register 330.

[0128] With further reference to FIG. 4, the load request address 610 and the mask information 620 are logically ANDed to derive a node number 630 from the load request address 610. The node number 630 derived from the address is compared with the node number information 640 stored in the node number register 330, using the comparator 320. In this example, local data are to be loaded, so the two node numbers agree, and the system recognizes the request as being for local data access.

[0129] Thus, the comparator 325 activates the latch 395 through the local-remote decision signal line 545. As a result, the load request is output to the local data cache 45 shown in FIG. 7, and enters the cache memory controller 343 through the address signal line 513. The cache memory controller 343 then outputs the load request address to the address array 353 through the cache memory control signal line 553.

[0130] The address array 353 that stores the addresses of information registered in the cache memory 363 checks whether the data, which are load-requested from the cache memory controller 343, are registered in the cache memory 363, and gives information representing the absence of registration to the cache memory controller 343 through the cache memory control signal line 553.

[0131] Upon receiving the information, the cache memory controller 343 issues a load request for memory to the memory access circuit 393 through a memory access request signal line 563. The memory access circuit 393 outputs the load request address presented on the address signal line 513 to the shared memory 65 through the selector 55. The shared memory 65 returns reply data for the load request to the memory access circuit 393.

[0132] The reply data are then sent via the data line 523 to the processor 25. At the same time, the cache memory controller 343 outputs a reply data registration request to the address array 353 through the cache memory control signal line 553. Upon receiving the registration request, the address array 353 erases part of the data in the cache memory 363, newly registers the reply data obtained through the data line 533 in the cache memory 363, and updates the information in the address array 353. In this way, the loading of local data not registered in the local data cache 45 does not affect the remote data cache 75.

[0133] (5-2)

[0134] In this example, the local data to be loaded are registered in the local data cache 45.

[0135] When a load request is output from the processor 25 to the access discriminate circuit 35 shown in FIG. 6, the access discriminate circuit 35 checks the load request address to determine whether the request is for local data or remote data. This decision is made in the same manner as for Example (5-1).

[0136] When the comparator 325 determines that the request concerns local data, it activates the latch 395 through the local-remote decision signal line 545. As a result, the load request is output to the local data cache 45 shown in FIG. 7, and enters the cache memory controller 343 through the address signal line 513. The cache memory controller 343 then outputs the load request address to the address array 353 through the cache memory control signal line 553.

[0137] The address array 353 that stores the addresses of information registered in the cache memory 363 checks whether the data, which are load-requested from the cache memory controller 343, are registered in the cache memory 363, and gives information representing the presence of registration to the cache memory controller 343 through the cache memory control signal line 553. At the same time, the registered data are placed on the data lines 533, 523. In this way, the loading of local data registered in the local data cache 45 does not affect the remote data cache 75.

[0138] When the processor 25 stores local data present in its node 15, the following cases are considered.

[0139] (6-1)

[0140] In this example, the local data to be stored are not registered in the local data cache 45.

[0141] When a store request is output from the processor 25 to the access discriminate circuit 35 shown in FIG. 6, the access discriminate circuit 35 checks the store request address to determine whether the request is for local data or remote data. This decision is made in the same manner as for Example (5-1).

[0142] When the comparator 325 decides that the request pertains to local data, it activates the latch 395 through the local-remote decision signal line 545. As a result, the store request is output to the local data cache 45 shown in FIG. 7, and enters the cache memory controller 343 through the address signal line 513. The cache memory controller 343 then outputs the store request address to the address array 353 through the cache memory control signal line 553.

[0143] The address array 353 that stores the addresses of information registered in the cache memory 363 checks whether the data, which are store-requested from the cache memory controller 343, are registered in the cache memory 363, and gives information representing the absence of registration to the cache memory controller 343 through the cache memory control signal line 553.

[0144] Upon receiving the information, the cache memory controller 343 issues a store request for memory to the memory access circuit 393 through a memory access request signal line 563. The memory access circuit 393 outputs the store request address presented on the address signal line 513 and the store data on the data signal line 523 to the shared memory 65 through the selector 55. The shared memory 65 updates data according to the request. In this way, the storing of local data not registered in the local data cache 45 does not affect the remote data cache 75.

[0145] (6-2)

[0146] In this example, the local data to be stored are registered in the local data cache 45.

[0147] When a store request is output from the processor 25 to the access discriminate circuit 35 shown in FIG. 6, the access discriminate circuit 35 checks the store request address to determine whether the request is for local data or remote data. This decision is made in the same manner as for Example (5-1).

[0148] When the comparator 325 determines that the request concerns local data, it activates the latch 395 through the local-remote decision signal line 545. As a result, the store request is output to the local data cache 45 shown in FIG. 7, and enters the cache memory controller 343 through the address signal line 513. The cache memory controller 343 then outputs the store request address to the address array 353 through the cache memory control signal line 553.

[0149] The address array 353 that stores the addresses of information registered in the cache memory 363 checks whether the data, which are store-requested from the cache memory controller 343, are registered in the cache memory 363, and gives information representing the presence of registration to the cache memory controller 343 through the cache memory control signal line 553.

[0150] Upon receiving the registration presence information, the cache memory controller 343 issues a request for eliminating the registration of the store-requested data to the address array 353. The address array 353, in response to the registration cancel request, erases the registration information concerned from the cache memory 363 and updates the information in the address array 353.

[0151] Further, the cache memory controller 343 outputs a store request for memory to the memory access circuit 393 through the memory access request signal line 563. The memory access circuit 393 sends the store request address presented on the address signal line 513 and the store data on the data signal line 523 to the shared memory 65 through the selector 55. The shared memory 65 updates the data according to the request. In this way, the storing of local data registered in the local data cache 45 does not affect the remote data cache 75.

[0152] When the processor 25 loads remote data present in another node such as node 115, the following cases are considered.

[0153] (7-1)

[0154] In this example, the local data to be loaded are not registered in the remote data cache 45.

[0155] When a load request is output from the processor 25 to the access discriminate circuit 35 shown in FIG. 6, the access discriminate circuit 35 checks the load request address to determine whether the request is for local data or remote data. The procedure is the same as that for Example (5-1).

[0156] When the comparator 325 has determined that the request concerns remote data, it activates the latch 405 through the local-remote decision signal line 545. As a result, the load request is output to the remote data cache 75 shown in FIG. 8.

[0157] The load request that was output to the remote data cache 75 is entered into a cache memory controller 346 through an address signal line 516. The cache memory controller 346 puts out the load request address to the address array 356 through the cache memory control signal line 556.

[0158] The address array 356 that stores the addresses of information registered in the cache memory 366 checks whether the data, which are load-requested from the cache memory controller 346, are registered in the cache memory 366, and gives information representing the absence of registration to the cache memory controller 346 through the cache memory control signal line 556.

[0159] Upon receiving the information, the cache memory controller 346 issues a load request for memory to the memory access circuit 396 through a memory access request signal line 566. The memory access circuit 396 outputs the load request address presented on the address signal line 516 to the remote access request circuit 85.

[0160] The remote access request circuit 85 generates a remote load request message according to the request from the memory access circuit 296. Further, by using the method similar to the one shown in FIG. 4, the remote access request circuit 85 derives a node number from the load request address and passes the remote load request message to a remote access reply circuit 195 of the node having the node number through the network 205.

[0161] The remote access reply circuit 195 that has received the remote load request message loads the data from the shared memory 165, generates a remote load reply message containing the load data, and returns it to the remote access request circuit 85 through the network 205. The remote access request circuit 85 then forwards the reply data to the memory access circuit 396.

[0162] The memory access circuit 396 then sends the reply data to the processor 25 through the data line 526. At the same time, the cache memory controller 346 issues a request for registering the reply data to the address array 356 through the cache memory control signal line 556. In response to the request, the address array 356 erases part of the data in the cache memory 366, newly registers the reply data in the cache memory 366, and updates the address array 356.

[0163] In this way, the loading of remote data not registered in the remote data cache 75 does not affect the local data area 40.

[0164] (7-2)

[0165] In this example, the remote data to be loaded are registered in the remote data cache 75.

[0166] When a load request is output from the processor 25 to the access discriminate circuit 35 shown in FIG. 6, the access discriminate circuit 35 checks the load request address to determine whether the request is for local data or remote data. This decision is made in the same manner as for Example (5-1).

[0167] When the comparator 325 determines that the request concerns remote data, it activates the latch 405 through the local-remote decision signal line 545. As a result, the load request is output to the remote data cache 75 shown in FIG. 8.

[0168] The load request that was output to the remote data cache 75 is entered into a cache memory controller 346 through an address signal line 516. The cache memory controller 346 outputs the load request address to an address array 356 through a cache memory control signal line 556.

[0169] The address array 356 that stores the addresses of information registered in the cache memory 366 checks whether the data, which are load-requested from the cache memory controller 346, are registered in the cache memory 366, and gives information representing the presence of registration to the cache memory controller 346 through the cache memory control signal line 556. At the same time, the registered data are placed on the data lines 536, 526. In this way, the loading of remote data registered in the remote data cache 75 does not affect the local data cache 45.

[0170] When the processor 25 stores remote data present in its node 15, the following cases are considered.

[0171] (8-1)

[0172] In this example, the remote data to be stored are not registered in the remote data cache 75.

[0173] When a store request is output from the processor 25 to the access discriminate circuit 35 shown in FIG. 6, the access discriminate circuit 35 checks the store request address to determine whether the request is for local data or remote data. This decision is made in the same manner as for Example (5-1).

[0174] When the comparator 325 decides that the request pertains to remote data, it activates the latch 405 through the localremote decision signal line 545. As a result, the store request is output to the remote data cache 75 shown in FIG. 8.

[0175] The store request that was output to the remote data cache 75 is entered into a cache memory controller 346 through an address signal line 516. The cache memory controller 346 outputs the store request address to an address array 356 through a cache memory control signal line 556.

[0176] The address array 356 t hat stores the addresses of information registered in the cache memory 366 checks whether the data, which are store-requested from the ache memory controller 346, are registered in the cache memory 366, and gives information representing the absence of registration to the cache memory controller 346 through the cache memory control signal line 556.

[0177] Upon receiving the information, the cache memory controller 346 issues a store request for memory to the memory access circuit 396 through a memory access request signal line 566. The memory access circuit 396 outputs the store request address presented on the address signal line 516 and the store data on the data signal line 526 to the remote access request circuit 85.

[0178] The remote access request circuit 85 generates a remote store request message containing the store request address and data according to the request from the memory access circuit 396. Further, by using a method similar to that shown in FIG. 4, the remote access request circuit 85 derives a node number from the store request address and passes the remote store request message to the remote access reply circuit 195 of the derived node number through the network 205.

[0179] The remote access reply circuit 195 that has received the remote store request message forwards the remote store address and data from the message and outputs them to the shared memory 165. The shared memory 165 updates data according to the request. In this way, the storing of remote data not registered in the remote data cache 75 does not affect the local data cache 45.

[0180] (8-2)

[0181] In this example, the remote data to be stored are registered in the remote data cache 75.

[0182] When a store request is output from the processor 25 to the access discriminate circuit 35 shown in FIG. 6, the access discriminate circuit 35 checks the store request address to determine whether the request is for local data or remote data. This decision is made in the same manner as for Example (5-1).

[0183] When the comparator 325 determines that the request concerns remote data, it activates the latch 405 through the local-remote decision signal line 545. As a result, the store request is output to the remote data cache 75 shown in FIG. 8.

[0184] The store request that was output to the remote data cache 75 is entered into a cache memory controller 346 through an address signal line 516. The cache memory controller 346 outputs the store request address to an address array 356 through a cache memory control signal line 556.

[0185] The address array 356 that stores the addresses of information registered in the cache memory 366 checks whether the data, which are store-requested from the cache memory controller 346, are registered in the cache memory 366, and gives information representing the presence of registration to the cache memory controller 346 through the cache memory control signal line 556.

[0186] Upon receiving the registration presence information, the cache memory controller 346 issues a request for eliminating the registration of the store-requested data to the address array 356. The address array 356, in response to the registration cancel request, erases the registration information concerned from the cache memory 366 and updates the information in the address array 356.

[0187] Further, the cache memory controller 346 outputs a store request for memory to the memory access circuit 396 through the memory access request signal line 566. The memory access circuit 396 sends the store request address presented on the address signal line 516 and the store data on the data signal line 526 to the remote access request circuit 85.

[0188] The remote access request circuit 85 generates a remote store request message containing the store request address and data according to the request from the memory access circuit 396. Further, by using a method similar to that shown in FIG. 4, the remote access request circuit 85 derives a node number from the store request address and passes the remote store request message to the remote access reply circuit 195 of the derived node number through the network 205.

[0189] The remote access reply circuit 195 that has received the remote store request message forwards the remote store address and data from the message and outputs them to the shared memory 165. The shared memory 165 updates data according to the request. In this way, the storing of remote data registered in the remote data cache 75 does not affect the local data cache 45.

[0190] For the system embodiments shown in FIGS. 9-11, Examples 5-8 are representative. The primary differences between the embodiments stem from the restructuring of the access discriminate circuit and the local and remote caches, so that the latter embodiment can take advantage of the ability to use market processors having a single data pin. Functionally, the embodiments are the same, and thus specific examples for FIGS. 9-11 along the above lines would be substantially redundant.

[0191] Various modifications of the invention will become apparent to those of ordinary skill in the art. All such modifications that basically rely upon the teachings through which the invention has advanced the state of the art are properly considered within the spirit and scope of the invention.