Dynamic I/O Data Address Relocation Facility

This invention is related to computer systems and particularly to a computer having a memory partition facility which may be changed.

The following related applications represent some recent developments for systems with an environment in which the present invention may be used: U.S. Pat. No. 5,345,590, issued Sep. 6, 1994, which is a continuation of U.S. patent application Ser. No. 07/576,344 filed Aug. 31, 1990 by Ault et al. and entitled Method and Apparatus for Cross-Partition Control in a Partitioned Process Environment. The invention there described has been published in Europe, as an example, in European Patent Application publication 472,861 on Jul. 5, 1991; and

U.S. patent application Ser. No. 07/755,246 filed Sep. 5, 1991 by J. E. Bastick et al. and entitled Method and Apparatus for Dynamically Changing the Configuration of a Logically Partitioned Data Processing System, now U.S. Pat. No. 5,253,344, issued Oct. 12, 1993; and U.S. patent application Ser. No. 07/974,393 filed Nov. 10, 1992 by J. George et al. and entitled "Method of Using Small Addresses to Access Any Guest Zone in a Large Memory", now U.S. Pat. No. 5,371,867, issued Dec. 6, 1994; and

U.S. patent application Ser. No. 08/014,977 filed Feb. 8, 1993 by K. J. Duvalsaint et al and entitled "Multi-Zone Relocation Facility Computer Memory System"; and

U.S. patent application Ser. No. 07/816,911 filed Jan. 3, 1992 by J. G. Brenza et al and entitled "Guest Host Large Addressing Method and Means", now U.S. Pat. No. 5,426,748, issued Jun. 20, 1995.

These co-pending applications and the present application are owned by one and the same assignee, International Business Machines Corporation of Armonk, N.Y.

The descriptions set forth in these co-pending applications are hereby incorporated into the present application by this reference.

For convenience we will define some terms as used in this application.

Inventions or advances in the computer system field should be useful in an open environment. The future systems open environment will include systems of the past as well as those of the future. IBM machines, Amdahl, and Hitachi machines employ hypervisors based upon IBM mainframe systems such as the System/390 (e.g. U.S. Pat. No. 4,564,903). IBM's ES/9000 series can operate in basic and hypervisor mode with logical partitioning of resources. Partitions can be created to enable small machines within a hypervisor environment (See e.g. IBM TDB n7B of 12-91, p. 309) to enable shared use of memory in a multiprocessor system for a CEC. It would also be desirable to increase memory utilization and machine operating performance in parallel systems and networks which employ parallel machines and partitioned memory.

Partitions of memory are used for processors of parallel machines, in mainframes and in networked systems, and in massively parallel machines which have hundreds or even thousands of parallel processing nodes with a host or control processor controlling processing nodes. An early example of a massively parallel machine was the IBM RP3, a research machine. Another example, more current, would be the Connection Machines' CM-5. In such systems, a control processor coordinates the functions of parallel processing nodes connected by some son of interprocessor communication network. A computational node which can comprise a group of processing nodes grouped to use a partition of memory. IBM mainframe structures use a group of processors (each of which may have multiple functional units) operating in parallel. In IBM mainframes, as illustrated by U.S. Pat. No. 4,843,541 the real mode can be used, but a hypervisor mode also exists, which permits the data processing system to be partitioned into a plurality of logical partitions. The controlling system is called a host. This host, which could be a control processor in a massively parallel machine, or a data server in a networked system, utilizes host control code, hardware and microcode to enable guests or clients to operate in different partitions. A system can have main storage, expanded storage, and channel and subchannel resources assigned to different (with LPAR logical) partitions in the system so different clients or guests can process applications in the different partitions. In an environment with multiple processors, we recognize that there exists software that exploits channel DAT. It would be desirable to enable such older existing machines with their memory systems as well as those which we are developing to be able to operate in an environment where partition utilization is improved.

In addition it would be desirable if the development could be used with the current trend in computer systems which is to utilize hypervisor environments to exploit more and more the logical partitioning of a CEC into many systems. Each system requires it's own storage and must be able to address that storage as an entity that starts with address 0 and increases to the maximum allocated to it. In a system that doesn't support channel Dynamic Address Translation (DAT), the hypervisor performs CCW address translation prior to issuing START SUBCHANNEL (SSCH). It would also be desirable to allow the advance needed for memory utilization to be used in such systems which utilize channel Dynamic Address Translation as is the case with some older Japanese Computer Manufacturer's software.

One of the problems with partitions, in which a user program or application runs (whether it be a real or a logical partition), is that storage needs to be assigned to the partition. This is done by a control program or an administrator. U.S. Pat. No. 4,843,541 allowed automatic relocation of the absolute addresses of the I/O channel and subchannel resources in the system to their assigned partitions. Absolute and virtual addresses of different guest programming systems, as well as page addresses for any expanded storage, were relocated into their assigned partitions.

The IBM ES/3090 Processor Complex functions as a PR/SM machine having an LPAR hypervisor. The separation and allocation of the physical resources is supported by the interpretive-execution facility of the machine and is performed by the hypervisor internal code called "LPAR". Similarly, the individual partitions can be called "guest" machines. It would be desirable to allow client machines to be treated like guests in a multiple client/data server networked environment.

It would be desirable to increase memory utilization and machine operating performance in parallel systems as well as in network environments and other systems (like highly parallel and massively parallel machines) which employ multiple processors operating concurrently. It would be desirable to improve the functional parallelism (the hardware and controls which permit different functions to be performed concurrently, e.g. I/O and separate addition and multiplication) in both real and hypervisor controlled processing systems. However, one of the barriers for automatically doing this has been the partition control structure. In the past when partitions needed to be changed the machines had to be in a quiescent state. The machine configuration has been controlled from an administration console or data server in order to relocate memory space.

Any advance should be used in advanced parallel processors like the IBM ES/3090 and the more current ES/9000 series. The IBM ES/3090 could perform a sort of zone relocation, as will be described. A publication about this machine was the Processor Resource/Systems Manager Planning Guide for the ES/3090 Processor Complex, IBM Publication No. GA22-7123-3. An advance for an ES/9000 system has been described in a co-pending application in which there has been described a way to relocate memory space and to change its size and location in whole or in pad, see U.S. patent application Ser. No. 08/014,977 filed Feb. 8, 1993 by K. J. Duvalsaint et al and entitled "Multi-Zone Relocation Facility Computer Memory System" which is included by reference.

It would be desirable now to be able to relocate memory space for one or more clients or guests during active processing. It would be especially desirable to do this for existing kinds of systems such as we have described, as well as those which develop in the future. And it would further be especially desirable to provide a way of doing this for existing IBM ES/9000 systems which would be able to operate as processors which can process their own applications as well as function as data servers.(this would be a server/host) A problem with some of the operating systems which operate in these environments is that they use a virtual data address in the CCW. In order to support virtual CCW, the hypervisor (LPAR) is required to perform the CCW address translation prior to issuing a START SUBCHANNEL instruction.

How these disparate desires may be achieved in light of the barriers which must be overcome is addressed by the present invention.

With our inventions we will be able to relocate memory space for partitions for one or more clients or guests during active processing for existing systems and those which can be developed from our technology in the future. Our implementation enhances our zone relocation facility. In accordance with our zone relocation facility the dependency on the host absolute location of the guest space has been removed. Now a channel program need not be stopped and retranslated when a guest is relocated.

As a result, we have provided for a computer system having a main store and functional units located in a server/host, and for a plurality of clients having a copy of the server/host operating system or their own operating system and coacting with the server/host and/or another client of the server/host with a means for dynamically on the fly relocating memory space for one or more applications of a client while the client and one or more of the server/host or another client are actively processing applications.

Our invention supports client server partitions that reside in the same physical system but in different logical partitions. The server starts I/O operations that cause data to be read directly into the clients' storage. This is the result of our CCW dynamic memory partition relocation facility. Our CCW dynamic memory partition relocation facility provides a way for the CCW and the data address to be relocated in the same zone as specified in the subchannel. This CCW relocation also now allows CCWs and IDAWs to be located in a zone other than the one specified in the subchannel and the data address to be either in the zone where the START SUBCHANNEL Instruction is issued or in the zone specified in the subchannel.

The invention defines a bit in the ORB. This bit controls the translation of the CCW address.

The invention also defines a bit in the CCW and a bit in the IDAW. These bits control the translation of the data address.

When the bit (called virtual relocation bit, J) in the ORB is one, the CCW address is based on the zone of the issuer of the START SUBCHANNEL instruction. The bit (called address type bit, A) in the CCW specifies whether the data address is a start subchannel issuers address or an address in the zone specified in the start subchannel instruction.

When the J bit is one and the indirect-data-address (I) bit is one, the A bit in the IDAW specifies whether data address is a start subchannel issuers address or an address in the zone specified in the start subchannel instruction.

In accordance with our invention, the ability of the computer system to translate a CCW without modifying guest storage is extended. Now a program can access data from a guest and a server/host in a single channel program. It can ask the server/host to put data anywhere in its partition. As the partition size can be expanded, this allows a program to dynamically respond to a need for memory by changing the partition and moving data to the expanded partition while other activities are being processed. Now server/host channel command words (CCWs) can be located in the server/host zone and data in the guest zone. The guest zone is specified in the start subchannel instruction. The invention allows a computer system to use server/host data words (via IDAWs) located in the server/host with data to be located in the guest zone. The guest zone is specified in the start subchannel instruction. The guest can be a client of a data server which functions as the server/host, or a guest in a partition of a mainframe which has a host control program administering the system.

FIG. 1 shows schematically a computer system having the dynamic I/O data address relocation facility as pad of a server/host which can perform its functions as a data server for a plurality of clients and also act as a mainframe with PR/SM functions controlled under LPAR.

FIG. 2 represents a prior art host CCW processing which was used in the IBM ES/9000.

FIG. 3 shows a simplified illustration of a host use of the dynamic CCW data relocation facility.

FIG. 4 shows a simplified illustration of a server using the dynamic CCW data relocation facility to provide data to it's clients.

FIG. 5 represents the flow chart of the invention.

FIG. 6 represents the flow chart of the invention as it applies to supporting both CCWs and IDAWs.

FIG. 7 represents use of the invention with an intermix of client/guest storage and server/host storage references within a single IDAW list.

Before turning to the drawings in detail we will describe a way of informing the hardware how to react when a relocation is required. It will be remembered that a problem with some of the operating systems which operate in these environments is that they use a virtual data address in the CCW. In order to support this type of data address, the hypervisor (LPAR) is required to perform the CCW translation prior to issuing a START SUBCHANNEL instruction. The translation is done by copying the CCWs into hypervisor storage (zone 0) and then translating the virtual data addresses to guest absolute addresses. Prior art translated the guest absolute addresses into host absolute addresses and then issued the SSCH instruction to initiate the I/O operation. Dynamic storage reconfiguration enhancements have made it possible to have a partition moved from one host absolute address location to another dynamically. This capability makes it necessary to serialize the I/O operations described with the movement of partitions and retranslate after a move has been completed, or have the hardware use data addresses that are based on the zone where the data is located. This invention provides a means of accomplishing the later of these two methods. This method also provides a way to support I/O operations for partitions that are located above the 2 GB line (>31 bit addressing).

The environment in which we achieve our goals may be illustrated by FIG. 1. In FIG. 1 the server/host is the parallel processing computer system 11 or CEC which can employ operating systems which are managed by a hypervisor, like LPAR. Such a computer system 11 will have a few to many functional units and a main storage which may be accessed by the functional units of the CEC. One such system is the IBM ES/9000 series of mainframes, but this parallel processing computer system could be one of a number of different manufacturers. Another example would be an Amdahl machine which runs MDF. Another computer system which functions as a data server for a plurality of clients could also be the server/host 11 illustrated by the FIG. 1. FIG. 1 illustrates the computer system functioning as a server/host for a plurality of clients. These clients could be, and in the illustrated preferred embodiment are, pad of a LAN network. The LAN servers 12 couple the plurality of clients 13 to the server host 11, which is illustrated as an air cooled machine 11. The server/host 11 could equally be a water cooled server host 14.

FIG. 2 represents a prior art CCW relocation facility which has been used in the IBM ES/9000. In this prior art example the main memory 15 is illustrated by FIG. 2. The operating system can use a virtual address as the CCW data address under the hypervisor LPAR, and LPAR is required to perform CCW address translation prior to issuing START SUBCHANNEL. In this system the relocation mechanism the translated CCW, IDAW and I/O data area addresses are always located in the same zone, Zone 0, which is the host zone. The specific subchannel zone number is set =0, and data addresses are located within zone 0.

FIG. 3 shows a simplified illustration of a host environment with multiple guest in accordance with the invention. In accordance with the invention the CCW data relocation facility allows the translated data address to be in the host zone, indicated by the A bit being set to 1, or in the guest zone, contained in the subchannel and indicated when the A bit is equal to 0. The intermixing capability enables the hypervisor to include both guest and host references within a single channel program. This is necessary to support channel programs which are self modifying.

Our invention will be implemented in a CEC with a zone relocation facility. Preferably the zone relocation facility will be provided in accordance with U.S. patent application Ser. No. 08/014,977 filed Feb. 8, 1993 by K. J. Duvalsaint et al and entitled "Multi-Zone Relocation Facility Computer Memory System". Because the channel program can reference guest space as a relative address within the guest space, the reference is independent of the absolute location of the guest space. While other machines have not had dependency on the absolute location of guest space, the zone relocation facility incorporates this. The removal of the dependence on the absolute location of the guest space allows the guest space to be moved while the channel program is executing. With this facility, the channel program need not be stopped and retranslated to map to a new guest location.

FIG. 4 represents a data server partition with client memory partitions. As illustrated the data server functions in it's own logical partition with some of it's clients residing in other logical partitions. The server partition is given authority to use the CCW data relocation facility via user interfaces on the system console. This allows the server to construct channel programs which move data to/from the client(s) memory.

As described in FIG. 3, the A bit is used so that the channel subsystem is to use the zone of the issuer of the SSCH instruction (A=1) or the zone of the start subchannel instruction (A=0) for data address relocation.

FIGS. 5 and 6 represent the flow chart of the LPAR virtual address relocation process.

The operation-request block (ORB) is the operand of a START SUBCHANNEL instruction. The "J" bit in the ORB (21) specifies the source of the zone number that the CCW address relocation is based on. If this bit is one, then relocation is performed based on the zone origin of the zone specified in the start subchannel instruction. If this bit is zero, then relocation is performed based on the zone origin of the zone of the program issuing the SSCH. In IBMs implementation for LPAR, the CCW address will be a host absolute address when the bit is 0.

If the "J" bit was zero and the "A" bit in the CCW or IDAW is 1 then a program check is presented to the issuing program. Otherwise, all data addresses are relocated based on the zone in the start subchannel instruction.

If the "J" bit is one, the "A" bit ,in either the CCW or IDAW (I bit in the CCW=1), when one causes relocation to be based on the issuers zone number and when zero causes relocation to be based on the zone specified in the start subchannel instruction.

FIG. 7 represents the CCW relocation utilizing the intermix of client/guest storage and server/host storage references within a single IDAW list. The IDAW list can specify either the server/host storage spaces or the client/guest storage spaces by utilizing A bit. Accordingly, the invention allows intermixing of these references within a single IDAW list.

We believe the above embodiment is preferable to other alternative approaches to achieve the result. For instance, alternatively utilizing the hardware illustrated by FIG. 1, two transfer-in-channel commands would be used. The CCW would have a special transfer-in-channel indicating that a next CCW contains a host absolute address and another CCW with a special TIC would indicate that the next CCW address contains a guest absolute address. Still in a third alternative one could use a transfer-in-channel and put a code in the upper 4 bits to indicate that the address is an absolute address in memory. The later embodiment would use the left byte of the length field to contain the op code.

As has been illustrated in detail, in the alternative and preferred embodiments, the invention provides a general facility that will be used by PR/SM hypervisors to build channel programs that reside in the hypervisor space. They can directly use guest/client specified addresses without translation to host Absolute by the Hypervisor. The prior need for the PR/SM Hypervisor to Limit check guest data space addresses is eliminated, thus eliminating code and enhancing performance. Thus our invention can be used to provide enhanced dynamic storage relocation permitting the guest area to be moveable.