Hierarchy Structured Memory System Contained in a Multiprocessor System

1. Field of the Invention

The present invention relates to a multiprocessor system having a shared memory and, more particularly, to a multiprocessor system, in which each processor element has a dedicated arbiter and a shared memory between itself and its upper processor element, and a method therefor.

2. Description of the Prior Art

In a prior art, the multiprocessor system having the shared memory is known as exemplified by "Nikkei Electronics, Vol. 407 (issued on Nov. 3, 1986)", pp. 119 to 129. FIG. 4 is a block diagram showing the schematic structure of the aforementioned system of the prior art.

In the microprocessor system having the shared memory according to the prior art, as shown in FIG. 4, one shared memory is connected with a common bus between two or more microprocessor units CPU 1, CPU 2, - - -, and so on, and a bus arbiter is also connected for arbitrating the conflict of the access to the shared memory.

In the above-specified conventional system, the capacity of the shared memory is limited to the address space (i.e., the address width capable of being expressed in the bit width of the shared bus) of the shared bus, thus raising a problem of shortage of its expandability. In order to expand and increase the size of the system thereby to improve the processing performance, more specifically, it is conceivable to expand and augment the memory capacity and to increase the number of the processor elements (i.e., the CPU 1, the CPU 2, - - -, and so on). In the system of the prior art, however, a shared memory in an external position is accessed from each processor element through the common arbiter and the common external bus so that any expanded system cannot be realized due to the bottleneck of the common bus even if the number of CPUs or the capacity of the memory is merely increased. In other words, the accesses of the individual CPUs conflict one another so that the processing performance (or throughput) matching the number of the CPUs cannot be attained even if only the number of the CPUs is increased while leaving the common bus as it is. It is also conceivable to reform the arbiter and the bus wiring together with the increases in the number of the CPUs and in the memory capacity so that the performance of the system in its entirety may be enhanced. However, this concept requires much labor and high cost, and it is uneconomical for preparing system for expansion in the future by disposing a bus having a wide address in advance.

Thus, in the existing multiprocessor system of the shared memory type, the maximum access space of the shared memory is equal to the address space of the common bus, which has its size limited thus making it difficult to subsequently increase the real memory capacity. In this system, moreover, one OS shared among a plurality of processors is arranged over the shared memory so as to centralize the management of the shared memory. As a result, the bus accessing frequency is increased to limit the number of processors to be added so as to eliminate the aforementioned bus bottleneck, thus limiting realization of a new function.

On the other hand, range of the data to be accessed by each processor is not efficient despite many parts proper to each processor, if the centralized shared memory is always used for such data. It is, therefore, desirable to distribute the functions of the individual processors to eliminate the centralization of the data while retaining the merits of the centralized system of the prior art.

It is, therefore, an object of the present invention to solve the above-specified technological problems of the prior art and to provide a multiprocessor system and a method therefor, which is free from having its real memory capacity limited by the bottleneck of the address space of a bus and from any data centralization by distributing the functions of individual processors and further which permits expansion of the system by connecting the individual processors sequentially by only a memory bus connection without any drastic reconstruction of the memory bus or the arbiter when the functions of the system are to be expanded and large-scaled.

In order to achieve the above-specified object, according to the present invention, there is provided a microprocessor system or a multiprocessor system of the type in which each of processor elements has a memory built therein. Each processor element comprises in addition to the aforementioned memory: a CPU; an address translator for translating a logical address outputted from the CPU into a physical address and for outputting it to an internal memory bus or an external memory bus; and a bus arbiter/selector (i.e., an arbiter for arbitrating a memory access between upper and lower ranks, and a bus selector for switching the bus in accordance with the output of the arbiter) for arbitrating access between the bus master of the memory bus of the upper processor element and the CPU to select and switch the bus so as to share said memory between the two (i.e., the bus master of the upper processor element and the CPU, that is, the upper processor element and the lower processor element). These processor elements can be connected sequentially in a multistage manner (e.g., in a tree structure having upper/lower relations) by only a memory bus connection (i.e., the connection by the aforementioned external memory bus, which is composed of an address bus, a data bus and a control bus (e.g., an address strobe, a read strobe, a write strobe or a weight control signal)).

Incidentally, the multiprocessor system of the present invention should not be restricted to the narrow one of the prior art named so but can cover a multiprocessor system widely, in which a plurality of processor elements are connected by a network having an arbitrary structure.

Each processor element has a virtual address space and accesses a memory built in itself by accessing in the virtual address space, when the target real memory area is in the memory built therein, and a memory built in a lower (than itself) processor element when the target real memory area is in the memory built in the lower processor element. In these ways, the memory built in each processor element is shared between itself and an upper (than itself) processor element (however, the real memory built in itself need not be shared with its lower processor element).

Moreover, the whole system can be realized in a small size and at a reasonable cost merely by packaging the aforementioned processor element in one chip to connect a plurality of chips.

Still moreover, a physically and logically remote processor element is equipped with both an active remote address & data communication controller having a general purpose communication interface and a passive remote address & data communication controller having a general purpose communication interface, so that an access can be accomplished from each processor element as if a memory interface were used.

Incidentally, in order to realize an arbitrary function distribution, there is disposed in a portion of the shared memory of each processor element a register which can register the function of each processor element from its upper processor element or a processor in processor element, so that the processing function of each processor element can be dynamically changed or added.

When an abnormality of an adjoining upper processor element is informed to a lower processor element through the shared memory by a self diagnosis of each processor element between itself and an adjoining upper processor, the function to be changed is registered in the aforementioned register so that the processor element itself informed of the abnormality may promptly change its own function. Thus, a group of processor elements lower than the processor element can be controlled independently of the upper processor by disconnecting said upper processor.

The operations of the aforementioned structure will be described in the following.

In a multiprocessor including a plurality of processor elements each having a memory stored therein for storing data, according to the present invention, there are provided: an arbiter which is disposed in a lower one of vertically adjacent processors for arbitrating a memory access between each processor element and an upper processor element so as to share the memory between itself and the adjacent processors; and a bus selector for switching a bus in accordance with the output of said arbiter. These processor elements are connected in a multistage (e.g., in a tree structure) by a memory bus connection only. As a result, the shared memory in the lower processor element can be accessed from the upper processor element by only the external bus which is composed of the address bus, the data bus and the control bus (e.g., the address strobe, the read strobe, the write strobe or the weight control signal).

Moreover, each processor element has a virtual address space proper to itself so that it accesses the shared memory through the virtual address space. More specifically, the external specifications, as seen from the upper processor element, are such that the processor element acts as a memory, while for the lower processor element, it acts as a computing unit using its lower processor element as the memory. As a result, by connecting the vertically adjacent processor elements of an identical structure sequentially in the multistage through the memory bus only, the whole system is not restricted by the address space of each processor and the bus, even if the address space is finite, so that the real memory capacity can be limitlessly expanded in a manner to correspont to the internal memory of each processor element. Still moreover, all the processor elements look as if they were stored memories of the same kind, as seen from the upper processor element, so that the memories existing in the virtual memory space can be unified. As a result, the whole system can be easily constructed merely by wiring the individual processor elements directly or through a driver/receiver without preparing any special external logic such as a bus arbitration control.

Since each processor element has an internal memory shared with its upper processor element, the functions can be easily distributed from the upper processor element to the lower processor element by registering the functions to be borne.

The foregoing and other objects, advantages, manner of operation and novel features of the present invention will be understood from the following detailed description when read in connection with the accompanying drawings.

FIG. 1 is a diagram showing one embodiment of a microprocessor system according to the present invention;

FIG. 2 is a diagram showing an address area assignment for the memory areas of each processor element according to the present invention;

FIG. 3 is a diagram showing the structure of one embodiment of the microprocessor system having a connection with a remote processor in accordance with the present invention;

FIG. 4 is a diagram showing the structure of an existing example of the multiprocessor having the shared memory;

FIG. 5 is a diagram showing an example of an address format of each processor element in the present invention;

FIG. 6 is a diagram showing one example of a memory space of the processor element according to the present invention;

FIG. 7 is a block diagram showing a processor structure for indirectly accessing a two-stage lower processor from an upper processor;

FIG. 8 is a diagram for explaining an address translation in the case of a direct access in the present invention;

FIG. 9 is a diagram for explaining an address translation method when the lower processor is accessed from the upper processor in the present invention;

FIG. 10 is a diagram for explaining the indirect access from the upper processor in the present invention;

FIG. 11 is a diagram for explaining the operations of a remote access in the present invention;

FIG. 12 is a diagram showing a data format for remote sending and reception in the present invention;

FIG. 13 is a diagram showing one embodiment of the processor for the indirect access from the upper processor element to a two-stage lower processor element by means of a hardware in the present invention;

FIG. 14 is a diagram showing the memory space of the uppermost processor element PE.sub.0 in the structure of FIG. 13;

FIG. 15 is a diagram showing the memory space of an intermediate processor element PE.sub.1 in the structure of FIG. 13; and

FIG. 16 is a diagram showing the memory space of the lowermost processor element PE.sub.2 in the structure of FIG. 13.

The present invention will be described in the following in connection with the embodiments thereof with reference to the accompanying drawings.

FIG. 1 is a diagram showing the structure of one embodiment of a microprocessor system to which is applied the present invention. Two processor elements (PEa) 1a and (PEb) 1b are arranged at upper and lower ranks, respectively. Each PE 1 (1a or 1b) is constructed of: an internal processor MPU 2 (2a or 2b); a shared memory 3 (3a or 3b) shared with the upper processor element; a bus arbiter and selector 4 (4a or 4b) for arbitrating a conflict with the bus master of the memory bus or the like of the upper processor element with respect to an access to the aforementioned shared memory 3; a control register 5 (5a or 5b) which can be accessed by both the upper processor element and the internal processor element through the bus arbiter and selector 4 (4a or 4b); and an address translator 6 (6a or 6b) for translating a logical address 7 (7a or 7b) outputted from the MPU 2 into a physical address 8 (8a or 8b). Incidentally, reference characters 10a and 10b designate external buses for coupling the individual processor elements, and characters 10a' and 10b' designate internal buses in the individual processor elements. In the present embodiment those internal and external buses are directly connected.

In FIG. 1, the internal processor MPUa (2a) of the PEa (1a) shares the shared memory 3b built in the PEb (1b) with the internal processor MPUb (2b) of the PEb (1b) through the bus arbiter and selector 4b of the PEb (1b). The PEa (1a) assigns the memory 3b, which is shared with the PEb (1b), to a portion of a virtual address space of the MPUa (2a) and translates the logical address 7a of the MPUa (2a) to the physical address 8a of the MPUb (2b) through the address translator 6a. A data line 9a of the MPUa (2a) is directly inputted to and outputted from the external bus 10a of the PEa (1a) and is connected to the bus arbiter and selector 4a so that it is inputted to and outputted from the shared memory 3a or the control register 5a in accordance with the arbitration of said arbiter. In order to register the lower processor PEb (1b), the upper processor PEa (1a) a function code indicating a function to be shared, in either the control register 5b in the shared area in the PEb (1b) or a buffer memory. The PEb (1b) executes a predetermined function corresponding to a function code in the control register 5b with reference to the function code. In other words, in case a number of aforementioned processor elements are used, the PEs such as the PEa (1) or PEb (1b) of FIG. 1 are cascaded in a multistage so that real memories can be arranged in the logical address space of the PEa (1a) and logical address space of the PEb (1b). These real memories are exemplified by processor-attached memories having a structure similar to that of the PEa (1a) and the PEb (1b). Thus, the real memory capacity can be arbitrarily expanded by expanding the logical address space sequentially in the multistage.

FIG. 2 shows the address assignments to the memory areas of the individual processor elements according to the present invention. In the present embodiment, the two processor elements are connected in juxtaposition to a common hierarchy, and two processor elements are connected in juxtaposition to the lower hierarchy of one of the processor elements. Specifically, processor elements PEa.sub.1 and PEa.sub.2 are connected in juxtaposition to the external bus 10 of the upper processor element (although not shown), which can access common address areas 30a (30a.sub.1 and 30a.sub.2). However, the PEa.sub.1 and PEa.sub.2 at an identical rank are independent so that they cannot access the address area 30a common with each other processor element. The memory of each of the processor elements PEa.sub.1 and PEa.sub.2 is constructed of: the address area 30a (30a.sub.1 or 30a.sub.2) common with its upper processor element; an address area 31a (31a.sub.1 or 31a.sub.2) common with its lower processor element; and a local address area 32a (32a.sub.1 or 32a.sub.2) for its own processor element. Next, the PEa.sub.1 has the PEb.sub.1 and PEb.sub.2 as its lower processor elements and has the address area 31a.sub.1 common with those lower processor elements. What should be noted here is that the area 31a.sub.1 is not the real memory of its own processor element unlike the area 30a or 32a, and that the real memory exists in the address areas 30b.sub.1 and 30b.sub.2 of the lower processor elements PEb.sub.1 and PEb.sub.2 common with their upper processor elements. Thus, the processor element PEa.sub.1 can make an access through the area 31a.sub.1 to the address area 30b.sub.1 or 30b.sub.2 common with its lower processor element PEb.sub.1 or PEb.sub.2. However, the PEb.sub.1 and PEb.sub.2 cannot access their mutual common address areas. Thus, the PEb.sub.1 and PEb.sub.2 take a memory structure similar to that of the PEa.sub.1, and the total system can be constructed of only the memory bus interface by cascading the processor elements having a similar structure sequentially in the multistage. As is apparent from the description thus far made, in FIG. 1 the address areas 30 to 32 of the individual PEs indicate the memory areas which can be accessed by the individual PEs when a view is made downward from the input sides of the address translators 6 (6a, 6b, - - - , and so on) of the individual PEs. It is apparent that the functions can be distributed for fitting applications by registering the functions of the individual processor elements in conformity to the related forms of the functions and by connecting the processor elements in the form of a tree structure conforming to the related forms of the functions through the aforementioned memory bus, because the individual processor elements are equipped with the control registers for registering the function codes indicating the function shares of the processor elements.

Next, it will be described with reference to FIG. 3 that accesses are made to processor elements which are physically and logically apart from each other. In FIG. 3, the parts identical to those of FIG. 1 are designated at identical reference characters, and their descriptions will be omitted. FIG. 3 shows the logical address outputs in case the upper processor element (PEa) 1a and the lower processor elements (PEb) 1b and (PEc) 1c are so physically and logically apart from each other so that they cannot access the buses directly. The PEa (1a) has an active address and data communication controller 11a for detecting that the logical address outputted to the external bus of the PEa (1a) is one assigned to a physically and logically remote processor element, to transmit said logical address and data to the PEb (1b). The active address and data communication controller 11a has a general purpose communication interface such as a MODEM or LAN for connecting the processor elements. On the other hand, the lower processor element PEb (1b) receives as I/O data the addresses and data, which are outputted from the upper processor element PEa (1a) by the active address and data communication controller 11a having the aforementioned general purpose communication interface, through a passive address and data communication controller 12b. The logical addresses thus received are translated to and outputted as the own logical addresses of said processor by the judgment of the same. Likewise, the PEb (1b) can access a physically and logically apart lower processor element PEc (1c) through an active address and data communication controller 11b and a passive address and data communication controller 12c.

The multiprocessor according to the present invention will be described in detail with reference to FIGS. 5 to 10. Each of the processor elements PEs of the multiprocessor has an address format, as shown in FIG. 5.

In FIG. 5, letter S (201) designates a segment number, and letter P (202) designates a page number, both of which are used as the logical address or the physical address. Letters RA (203) designate a lower address of the segment and page, i.e., the physical address.

Next, FIG. 6 shows an address space of one processor element (as designated at PEo). FIG. 7 shows a processor structure corresponding to FIG. 6.

The address space of the PEo is composed of an m number of segments So, S.sub.1, S.sub.2, - - - , S.sub.i, - - - , and S.sub.m-1, each of which S.sub.i is composed of an n number of pages Po, P.sub.1, P.sub.2, - - - , and P.sub.n-1. Here, the segment So (i.e., the segment in the column of S=0 in FIG. 6) is belongs to the address space which can be directly accessed by the processor element PEo (i.e., 100a of FIG. 7). Likewise, the segments S.sub.1 to S.sub.m-1 (i.e., the segments in the column of S=1.about.m-1 in FIG. 6) respectively belong to the address spaces which can be directly accessed by the processor elements PE.sub.1 to PE.sub.m-1 (i.e., 100b.sub.1 to 100b.sub.m-1 of FIG. 7).

The page Po of the segment So (i.e., the segment in the column of P=0 in FIG. 6) is not only the internal memory of the processor element PEo but also the common area with the upper processor element (although not shown) of the processor element PEo. This area corresponds to the area 30 of FIG. 2. Likewise, the pages P.sub.1 to P.sub.n-1 (i.e., the columns of S=0 and P=1 to n-1 in FIG. 6) are not only the real memory area which can be directly accessed by the PE.sub.0 but also the common memory area with the lower processor elements PE.sub.1 to PE.sub.n-1 (i.e., 100b.sub.1 to 100b.sub.m-1 of FIG. 7) of the PE.sub.0 (i.e., 100a of FIG. 7). This area corresponds to the area 31 of FIG. 2. Incidentally, the present embodiment is not equipped with any one corresponding to the local area 32 for its own process in FIG. 2.

The segments S.sub.1 to S.sub.m-1 are the logical address areas of the PE.sub.0 so that they can be accessed not directly from the PE.sub.0 but through the PE.sub.1 to PE.sub.m-1. In FIG. 7, letter I designates the processor element numbers 0, 1, 2, - - - , and (m-1) in the space of the segment S.sub.0, and letter i designates an arbitrary one of them. Letter J designates the process element numbers 0, 1, 2, - - - , and (m-1) in the space of the segment S.sub.i, and letter j designates an arbitrary one of them. The PE.sub.1 to PE.sub.m-1 (100b.sub.1 to 100b.sub.m-1) are at a mutually equal rank and just lower than the PE.sub.0 (100a). Moreover, the PE.sub.i1 to PE.sub.i(n-1) (100c.sub.1 to 100c.sub.n-1) are at a mutually equal rank and just lower than the PE.sub.i (100b.sub.i). The PE.sub.i is shared by the spaces S.sub.0 and S.sub.i. Specifically, in FIG. 6, the area of S=0, P=i and the area of S=i and P= are actually identical.

According to this structure, in case the processor element PE.sub.0 (100a) accesses the page P.sub.j of the segment S.sub.i, the logical address outputted by the PE.sub.0 (100a) is once stored in the address register for the interface with an upper processor, which is built in a processor element PE.sub.i (100b.sub.i) (i.e., the processor element capable of directly accessing a page in the segment S.sub.i) and is referred to by a software until it is outputted as the logical address of the processor element PE.sub.i to lower processor elements PE.sub.i1 to PE.sub.i(n-1) (100c.sub.1 to 100c.sub.(n-1)) of the PE.sub.i. Thus, the PE.sub.i (100b.sub.i) accesses the shared memory with the corresponding processor element PE.sub.ij (100c.sub.j) of the PE.sub.i1 to PE.sub.i(n-1).

Next, FIG. 8 presents a diagram for explaining an address translation in the case of the direct access. The explanation is started from the case in which the processor element PE.sub.i accesses the shared memory with its upper one, i.e., its own internal memory. The MPU.sub.i 2 in the PE.sub.i outputs the logical address 7 in accordance with the address format of FIG. 5, translates its logical address through the address translator 6 into the physical address 8 and accesses it as the internal memory area (as assumed for S.sub.i =0 and P.sub.i =0) of its own processor MPU.sub.i 2. The physical address 8 or the output of the address translator 6 is inputted to a decoder 303, which in turn detects the assumption of S.sub.i =0 and P.sub.i =0 to output a bus request signal to an arbiter 304. This arbiter 304 accords to the mode, which is registered in advance in the MPU.sub.i 2, to arbitrates the access coming from the processor element (e.g., PE.sub.o in this case) just upper the PE.sub.i and the access of its own processor element. The arbitration mode can be exemplified by the preference of the processor element owned by itself, the preference of its upper processor element, the equivalent access of both processors or the single access of its own processor element. This mode is set by the registration of a mode register 310 by the MPU.sub.1 2. In other words, the arbitration mode can be dynamically switched.

In accordance with the arbitration result of the arbiter 304, a bus selection signal SEL and a memory select signal MSEL are outputted. When the SEL is at "0", for example, its own processor element is selected. For the SEL="1", the upper processor element is selected. When the shared memory is selected in response to the memory select signal MSEL outputted from the arbiter, the physical address RA of its own processor MPU.sub.i or the upper processor element PE.sub.o is selected by a selector 305 so that said address in the shared memory 3 can be accessed. In case the shared memory 3 is to be accessed from the upper processor element PE.sub.o, the address from the external bus of the PE.sub.o is latched with an address strobe AS indicating the validity of the address when in an external bus cycle and is stored in the address register 307. If the address from the PE.sub.o is expressed at S.sub.o, P.sub.o and RA, as indicated in an address register 307 of FIG. 8, the outputs are S.sub.o =0 and P.sub.o =i so as to access the shared memory 3 with the PE.sub.i so that the shared memory 3 can be accessed through the arbiter 304 in accordance of the decoded result of a decoder 308 for decoding the output of the address register 307. At this time, the page number P used as the decode signal indicating the address of its own processor element PE.sub.i is different for each PE so that it is set in advance under the upper processor. If an external circuit such as an internal switch is required, the cost is raised by the increase of an external device. Thus, there is provided a processor number register (PNR) 309 which is built in the processor element.

Next, referring to FIG. 9, an address translation method will be described when the shared memory in the lower processor element is to be accessed from the processor element PEo. It is assumed for simplicity that the physical address of the internal memory in each processor element have the segment S=0 and the page P=0 when each processor element itself makes the access. The logical addresses to be outputted from the processor element PEo are the So, Po and RA in accordance with the address format of FIG. 5. Let it be assumed that So=io and Po=jo (wherein io and jo designate the values of the segment So and the page Po). First of all, if the internal memory is to be accessed, the relations of io=jo=0 are detected by decoder 308 so that the memory owned by the PEo itself is accessed through the arbiter.

In another case in which the memory of the processor element at the segment S=0 is to be accessed, the relations of io=0 and jo.noteq.0 are detected by the decoder 308 so that the entry of the reference jo of the address translation table DAT 311 in the PEo is called up. The content 312 of a page table 311 is composed of a real page corresponding to each logical page and a page invalid bit (IB) indicating the validity or invalidity of said real page. For IB=0, for example, said real page is valid and can be directly accessed. If the segment S=0 is outputted from the processor element PE.sub.o and if the lower processor elements of the PE.sub.o are less than (n-1), IB=1 may result. For IB=1, said real page is invalid, and the so-called "page fault interruption" is caused as the address invalid interruption by the processor MPU.sub.o in the PE.sub.o. In response to the page fault, the MPU.sub.o cuts out the corresponding area from the secondary memory and loads or pages it in the real memory corresponding to said real page j.sub.o ' to set the page invalid bit (IB) at IB=0 and to access a predetermined address. When the page invalid bit (IB) is at "0", the entry page of the page table 311 indicates the leading address (i.e., j.sub.o ' in FIG. 9) of the page to be accessed. Since i.sub.o =0, as a matter of fact, the address of S=0, P=j.sub.o ' and RA is outputted as the external address of the PE.sub.o.

In another case of i.sub.o .noteq.0, it is indicated that the segment S.sub.10 managed by the other processor element PE.sub.10 is indirectly accessed. In the aforementioned case, the number i.sub.o is outputted as it is. Since i.sub.o .noteq.0, the external output page j.sub.o " of the PE.sub.o is the page j.sub.o outputted from the MPU.sub.o. For i.sub.o =0 and j.sub.o .noteq.0, j.sub.o "=j.sub.o ', as has been described hereinbefore. Specifically, the selecting direction of a selector 313 is determined according to the value of the segment number i.sub.o so that the page address j.sub.o " outputted from the processor element PE.sub.o is selected to have either the number j.sub.o or j.sub.o '.

Although the address outputting method of the upper processor element has been described hereinbefore, an address outputting method of the lower processor element PE.sub.i to be indirectly accessed from the upper processor element PE.sub.o will be described with reference to FIG. 10. The lower processor element PE.sub.i once buffers the output address of the upper processor element PE.sub.o in the input address register 307 in response to the address strobe AS, to decodes the segment S.sub.o and the page P.sub.o with the decoder 308. Since S.sub.o .noteq.0 in the case of the indirect access, the relation i.noteq.0 is detected, and a wait signal is returned to the upper processor element PE.sub.o if (S.sub.o, P.sub.o)=(i, j.sub.o). For the MPU.sub.i 2, a page index flag PX is set in a page index interruption control status register PXR 315 so as to indicate the indirect access. The PXR 315 has a PXI interruption enable bit PXIE. For PXIE=1, the MPU.sub.i 2 receives the input of the page index interruption PXI by taking a logical product of the PXIE and the PX with an AND operator 316. In response to the PXI, the MPU.sub.i 2 refers to the input address register 307 and outputs the logical address (S.sub.i, P.sub.i)=(0, j.sub.o) of the MPU.sub.i 2 so as to access the logical page j.sub.o in the segment for managing the PE.sub.i. By the dynamic address translator DAT 6 in the PE.sub.i, the address (S.sub.i, P.sub.i)=(0, j.sub.o) is translated into the real address (S.sub.i ', P.sub.i ')=(0, j) of the PE.sub.i. This (S.sub.i ', P.sub.i ')=(0, j) and the displacement physical address RA are outputted to the lower processor PE.sub.ij of the PE.sub.i. Since S.sub.i '=0, the aforementioned address presents an address area which can be directly accessed by the PE.sub.i. If PXIE=0, the interruption is disabled, and the PXI control status register PXR 315 is referred with the software of the PE.sub.i so that similar operations as those for the PXIE=1 are accomplished. Incidentally, the wait signal is continuously issued till the MPU.sub.i finishes its access to the data.

Next, the case, in which a physically and logically removed processor is to be accessed (which will be called the "remote access"), will be described with reference to FIGS. 11 and 12. FIG. 11 is a diagram for explaining the operations of the remote access. FIG. 12 is a format showing the remote address output data. A processor element to perform the remote access is designated at PE.sub.i, and a processor element to be subjected to the remote access is designated at PE.sub.ij.

If the processor MPU.sub.i 2 in the PE.sub.i outputs an address through the address translator DAT 6, whether or not it is the remote access is decided with a remote address access table 321 in an active address I/O controller 320. If not the remote access, the address is outputted from the external bus as usual. If the remote access, a remote access signal or the decision result of the remote address access table 321 is outputted to a remote bus controller 322 and a remote sending buffer 323. In response to the remote access signal, the remote bus controller 322 outputs a wait signal to the MPU.sub.i 2 and an inhibit signal of the external bus output to a bus driver 324. In response to the remote access signal, moreover, the remote bus controller 322 a bus control signal to an FC & DL generator 325 and the remote sending buffer 322 so as to generate data to be set in the remote sending buffer 323. The format of the data to be set in the remote sending buffer 323 is composed, for example, a function code FC, a data length DL, the addresses S, P and RA, and data D, as shown in FIG. 12. The function code FC sets a function code in which an R/W (read/write) direction of the remote access, a data width and so on are coded. The data length DL indicates a sending data length sent from a general purpose adapter. The addresses latch the segment S, the page P and the physical address RA after having passed through the address translator DAT 302. Finally, output data corresponding to the data size of the write instruction, if so, and dummy data corresponding to the data size of the read instruction, if so. In short, the write cycle is completed by outputting the data in the case of the write operations by the remote access. In the case of the read operation, however, a wait is made till the data are returned from the general purpose communication interface. The wait signal is returned to the memory bus of the PE.sub.i from the active address I/O controller 320. The wait signal is returned from the active address I/O controller 320 to the memory bus of the PE.sub.i. When the data of the format shown in FIG. 12 are latched in the remote sending buffer 323, a DMA (i.e., Direct Memory Access) controller 326 initialized in advance for the sending and receiving operations operates to perform the DMA transfer of the data of the remote sending buffer 323 to a general purpose sending adapter 327 and to output the remote access address and data from the active address I/O controller 320 of the PE.sub.i.

The sending and receiving DMA controller 326 performs the same operations as those of an ordinary DMA controller. Specifically, when the bus mastership is requested of the processor MPU.sub.i 2 in the PE.sub.i and is acquired, the DMA transfer is accomplished by using the bus between the remote sending buffer 323 and the sending adapter 327. During this DMA transfer, the sending and receiving DMA controller 326 outputs the read address to the remote sending buffer 323 to sequentially read out the sending data latched in accordance with the format of FIG. 12 from the remote sending buffer. At the same time, in order that the aforementioned sending data may be directly written in the sending adapter 327 through the data bus, the sending strobe is outputted from the sending and receiving controller 326 to the sending adapter 327 so that the aforementioned sending data are sequentially outputted from the sending adapter 327.

On the other hand, the lower processor PE.sub.i j for receiving the remote access of the PE.sub.i receives the aforementioned remote access data at a receiving adapter 421 so that the remote access data are latched in a receiving buffer 423 by a receiving DMA controller 422 which has been set in advance like the sending case. When the input to the receiving buffer 423 is completed, a remote address reception complete signal is outputted from the receiving buffer 423 to a bus timing generator 423 so that the bus control signals of the remote access such as an address strobe AS', a read strobe RD' or a write strobe WR' are generated from the bus timing generator 424. In response to the address strobe AS' of the remote access, the data of the receiving buffer are latched in an address register 501 for the upper processor of the processor element PE.sub.ij. Like the case of the direct access from the upper processor, the memory select signal MSEL is outputted through a decoder and arbiter 502 so that the write data are latched in the shared memory 3 or so that the read data are read out from the shared memory 3. In the write case, the access is completed at the write stage. In the read case, the read data are set in a return buffer 425, and a remote access flag RMF is outputted from the bus timing generator 424 and inputted to a returning FC & DL generator 426. Moreover, the memory select signal is also inputted to the FC & DL generator so that the FC and DL are generated and latched in the return buffer 425 by using the bus control signal such as the address strobe AS' or the read strobe RD' as the fundamental signal. The return data of the format, as shown in FIG. 12, are returned from the return buffer 425 to the PE.sub.i via sending adapter 427 by the sending and receiving DMA controller 422 in the PE.sub.ij. In the PE.sub.i, the return data are received in a remote receiving buffer 329 by a receiving adapter 328 and the sending and receiving DMA controller 326. It is detected by an FC coincidence detector 330 that the FC and DL of the remote receiving buffer 329 and the FC and DL of the sending FC & DL generator 325 are coincident. The reception completion is returned to the remote bus controller 322, if coincident, and an error is returned to the remote bus controller 322 if incoincident. The MPU.sub.i 2 reads the data of the remote receiving buffer 329 to complete the data transfer through the remote bus controller 322 turning off the wait signal instantly. If the FC or DL is not coincident, the error signal is outputted from the FC coincidence detector 330, and a read error interruption is issued in the MPU.sub.i 2 through the remote bus controller 322 to release the wait signal, thus completing the read cycle instantly.

FIG. 13 shows an embodiment in case an access from an upper processor element is relayed to realize the indirect access to the shared memory in the lower processor element with a hardware. As shown in FIG. 13, the processor elements PEo (PEa), PE.sub.1 (PEb) and PE.sub.2 (PEc) are individually connected in a sequential multistage such that the PE.sub.0 is underlaid by the PE.sub.1 and that the PE.sub.1 is underlaid by the PE.sub.2. The description will proceed to the case in which the processor element PE.sub.0 accesses the shared memory of the processor element PE.sub.2 through the processor element PE.sub.1. Each processor element PE includes an MPU 2 (2a, 2b or 2c) and a dynamic address translator DAT 6 (6a, 6b or 6c) for translating a logical address 7 (7a, 7b or 7c) outputted by the MPU 2 into a physical address 8 (8a, 8b or 8c). The physical address 8 (8a, 8b or 8c) is decoded by a decoder 41 (or 41a, 41b or 41c), in which a decode signal is outputted to an arbiter 4 (4a, 4b or 4c) or 43 (43a, 43b or 43c) in dependence upon either the access to an internal shared memory 3 (3a, 3b or 3c) or the access to the lower processor connected to the external bus through the external output buffer. Likewise, the decode signal from the upper processor element is outputted to the arbiter 4 or the arbiter 43 through an area decoder 40 (40a, 40b or 40c) and a decoder 42 (42a, 42b or 42c). The arbiter 4 selects the shared memory 3 through a selector SEL 45 (45a, 45b or 45c). In case an external memory area is to be accessed, the shared memory of the lower processor element is likewise accessed from an output buffer 44 (44a, 44b or 44c) through the arbiter 43 and a selector 46 (46a, 46b or 46c). The selector 45 or 46 inputs a select signal from the arbiter 4 or 43 to select and output data 47 (47a, 47b or 47c) from the data bus 9 (9a, 9b or 9c) of the MPU or the upper processor element.

Next, the description will proceed to the case in which the processor element PE.sub.0 accesses the shared memory of the processor element PE.sub.2 through the processor element PE.sub.1. The segment numbers S of the processor element PE.sub.0, PE.sub.1 and PE.sub.2 are assumed to be 0, 1 and 2, respectively, in view of the PE.sub.0. In order to access the shared memory 3c of the processor element PE.sub.2 from the processor element PE.sub.0, the segment number S=2, the page number P=0 and the physical address RA are first outputted as the logical address 7a from the MPU 2a in accordance with the format of FIG. 5. The logical address 7a is translated into the physical address 8a through the dynamic address translator 6a, but the physical address 8a is assumed for simplicity to have the same values as those of the logical address 7a. Moreover, the shared memory in each processor element is assumed to have one page. The memory spaces of the individual processor element PE.sub.0, PE.sub.1 and PE.sub.2 in the structure of FIG. 13 are shown in FIGS. 14, 15 and 16, respectively. The memory space of the processor element PE.sub.0 looks as shown in its entirety in FIG. 14. Specifically, both the segments S=0 and S=1 look as having the memory areas of two pages P=0 and P=1, and the segment S=2 looks as having the memory area of one page of P=0. The memory space of the processor element PE.sub.1 is that of the processor element PE.sub.0 excepting the space of S=0, as shown in FIG. 15, and looks as having the spaces of the two pages of its own processor segment S=0 and the one page of another processor segment S=2.

The memory space of the processor element PE.sub.2 looks as having the space of one page of its own processor segment S=0, as shown in FIG. 16.

Reverting to FIG. 13, the flow of the address issued by the MPUa (2a) will be described. The logical address 7a outputted by the MPUa (2a) is translated by the dynamic address translator DAT 6a into the physical address 8a, which is inputted to the decoder 4a. This decoder 4a decodes the physical address 8a and that the memory area of S=2 and P=0 is the shared memory built in another processor element in the memory space of the processor element PE.sub.0, thus outputting the access request signal to the arbiter 43a. In response to the access request signal from the decoder 42a, the arbiter 43a outputs: a select signal for selecting the MPU 2a to the selector SEL 46a; the data 9a of the MPUa (2a) to the I/O buffer 44a; and the physical address 8a of the MPUa (2a) to the I/O buffer 44a. The address outputted by the I/O buffer 44a of the processor element PE.sub.0 is inputted to the area decoder 40b of the processor element PE.sub.1. Since the inputted address has S=2 and P =0, the area decoder 40b decodes the accessible memory area of the processor element PE.sub.2, i.e., the areas of S=1 and S=2 in the memory space of the processor element PE.sub.1, as shown in FIG. 14, to output the wait signal to the upper processor element PE.sub.0 and the address to the decoder 42b. In response to the wait signal, the processor element PE.sub.0 extends the bus cycle while it is receiving the wait signal. The decoder 42b judges whether the access area is the shared memory built in its own processor element or the memory of the lower processor element. Since, in this case, the address has S=2 and P=0, it is decided that the memory region is that of the lower processor element, to relay and output the address to the area decoder 40c of the lower processor element PE.sub.2 through the decoder 42b, the arbiter 43b and the I/O buffer 44b like the case of the processor element PE.sub.0. In response to the select signal outputted from the arbiter 43b, on the other hand, the data are inputted or outputted to the I/O buffer 44b through the selector 46b in accordance with the direction of the access and are further inputted or outputted to the lower processor element PE.sub.2 from the I/O buffer 44b in accordance with the access direction. Like the case of the processor element PE.sub.1, the processor element PE.sub.2 decides the area managed by its own processor element with the area decoder 40c to output the wait signal to the upper processor element. At the same time, the processor element PE.sub.2 decodes with the decoder 42c that the address belongs to the area of the internal memory 3c of its own processor element because of S=2 and P=0. Without any access from its own processor MPU 2c, the arbiter 4c outputs the address from its upper processor element to the shared memory 3c and the select signal for selecting the data bus from its upper processor element PE.sub.1 to the selector 45c thereby to read or write the memory 3c in accordance with the direction of the access. When the arbiter 4c outputs the select signal for selecting the bus of the upper processor element, the area decoder 40c releases the output of the wait signal to the upper processor element PE.sub.1. When the wait signal from the lower processor element PE.sub.2 is released, the processor element PE.sub.1 uses the area decoder 40b the wait signal which has been outputted by the processor element PE.sub.1 to its upper processor element PE.sub.0.

In the access reading case, the read data of the lower processor element are naturally relayed and outputted to the output buffer 44a of the upper processor element PE.sub.0 through the output buffer 44b and the selector 46c. When the wait signal from the processor element PE.sub.1 is released, the access source MPUa (2a) reads the data of the data bus 9a to complete the read cycle.

In the case of the data write, the processor element PE.sub.1 writes the data in the I/O buffer 44a to complete the data access. In the case of the write access from the upper processor, therefore, the area decoder of each processor receives, if the MPU 2 and the access of the lower processor element conflict, a bus grant signal BG.sub.EXT from the arbiter 4 or 43 to access the internal memory 3 of the lower processor element or the output buffer 44. The input/output control of the bus control signal (e.g., the address strobe, read strobe, write strobe or wait signal) after reception of the bus grant signal BG.sub.EXT is performed by the area decoder 40. By the control thus far described, the shared memory 3c of the upper processor element PE.sub.0 to the lower processor element PE.sub.2 can be indirectly accessed with the hardware.

As has been described in detail hereinbefore, according to the present invention, the individual processors can be inversely deemed as memory elements having processors by cascading the processor elements implementing individual memories sequentially in the multistage. Thanks to the multistage cascade structure (or the tree structure) of the memory elements, there can be attained an effect that the real memory capacity can be limitlessly expanded without depending upon the address width of the bus. As a result, the system can be expanded and large-scaled merely by adding the processor element of the same structure to the tree structure without being required to remaking the bus wiring or the arbiter.

Since, moreover, the processor is present in the aforementioned memory element, there can be attained an effect that the function distribution such as the connection into the tree structure can be easily realized.

In order to access a remote processor element by sending the addresses and the data, still moreover, there are provided the active address & data communication controller and the passive address & data communication control for sending the addresses and the data when an access is received from the remote processor element. Thus, there can be attained an effect that the logically and physically remote processor can be indirectly accessed through the memory interface to construct a system which is rich in its expandability.