US 6,145,066 AGrant
Computer System With Transparent Data Migration Between Storage Volumes
Issue Date:2000-11-07
•7 Claims
•6 Drawing Sheets
Abstract
A computer system includes a transparent data migration facility (TDNff) to accomplish automated movement of data (migration) from one location to another in the system. A data migration program includes a main module to control the start of a migration session when said application programs are using data accessed to and from the source volume, to migrate data from the source volume to the target volume, and to end the migration session whereby the application programs are using data accessed to and from the target volume. The data migration program includes a volume module to control the volumes during the migration session. The data migration program includes a copy module to control the copying of data from the source module to the target module during the migration session. The data migration program includes a monitor module for monitoring I/O transfers to the data volumes during the migration sessions. The computer system may have a plurality of operating systems associated with instances of the data migration program which allows for concurrent data migrations. The plurality of instances of the data migration program may also be controlled in a master slave relationship. A migration session may include a plurality of migration phases such as activation, copy, refresh, quiesce, synchronize, redirect, resume and termination phases.
Metadata
Assignee
- Amdahl Corporation
Inventor
- Carey Phillip Atkin
Application Information
Application Number:US 9714731
Filing Date:1997-11-14
Priority Date:1997-11-14
Art Unit:279
Classifications
IPC:
G06F 1300
Field of Search:
711714709162;161;165;111;112;114;55;6;7208;211;216;217
Patent Drawings (6 sheets)
Description
Background of the Invention
The present invention relates to the management and maintenance of large computer systems and particularly to automated methods and apparatus for the movement of data (migration) from location in the system to another.
In 1983, International Business Machines Corporation of Armonk, N.Y. (IBM) described the requirements and capabilities needed in order to efficiently manage and maintain the storage in the modern large data center and to evolve toward automated storage management IBM's Data Facility/System Management Storage (DF/SMS) capabilities. In discussing device support and exploitation, IBM identified the requirement for the timely support and exploitation of new device technologies and the need for tools to simplify the movement of data (migration) to new subsystems.
A standard set of tools has been provided by IBM and other software developers that has allowed data to be automatically copied, archived, and restored. Evolutionary improvements have been made to these tools in the areas of performance, usability, and in some cases data availability; but a problem still exists in that the availability capabilities of these facilities have not kept pace with the availability requirements that exist in data centers. The storage administrator must be able to support the increasing demands of continuous 24 hour by 7day data availability.
There is an explosive growth in the need to store and have on-demand access to greater and greater pools of data. As capacity requirements skyrocket, data availability demands increase. These factors coupled with the need to control costs dictate that new RAID (Redundant Array of Independent Disks) storage technology be implemented. The dilemma faced by data center management is that the implementation of new storage technology is extremely disruptive and therefore conflicts with the need to maximize availability of the data. Therefore, an additional tool is required that will allow data to be nondisruptively relocated or migrated within the data center.
Essentially, a data migration facility provides the ability to "relocate" data from one device to another device. A logical relationship is defined between a device (the source) and another device (the target). The logical relationship between a source and target volume provides the framework for a migration. The data migration facility controls multiple concurrent migrations in a single group that is called a session. A migration is the process that causes the data on the source to be copied to the target.
The characteristics that are critical components of a transparent data migration facility include the following:
The facility is completely transparent to the end user and the application program. No application disruption is required in order to make program changes and the facility is dynamically and nondisruptively activated.
The facility provides for full data access during the data migration. The data on the source volume is available to the end user for read and write access.
The facility provides for a dynamic and nondisruptive takeover of the target volume, when the source and target volumes are synchronized.
The migration facility must ensure complete data integrity.
The migration facility should NOT be restricted to any control unit model type or device type. All devices in the data center should be able to participate in a migration and the facility should support a multiple vendor environment.
The State of the Industry
Migration facilities that exist today were primarily designed for disaster recovery or the facilities were meant to address single volume failures. However, these facilities can also be used as data migration tools. These facilities differ in implementation and use a combination of host software and/or control unit firmware and hardware in order to provide the foundation for the data migration capability.
Local Mirroring
The IBM 3990 host extended feature IBM Dual Copy and the EMC Symmetrix, from EMC Corporation (EMC), mirroring feature are two examples of local mirrors. A source volume and a target volume are identified as a mirrored paired and at the creation of the mirrored pair, data is transparently copied or migrated to the secondary volume. Continuous read and write access by applications is allowed during the data migration process and all data updates are reflected to the secondary volume.
In the case of the IBM 3990 host, the mirror is under the complete control of the system operator. For example, through the use of system commands a mirrored pair can be created. At create time, the data will be copied to a secondary device. At the completion of this copy, the operator can then disconnect the pair and assign the secondary device to be the primary. This is called Transient Dual Copy and is an example of Dual Copy being used as a migration facility.
The function of the EMC mirroring feature is to maximize data availability. The EMC subsystem will disconnect the mirror in the event of a failure of one of the paired devices. The mirror will be automatically reconstructed when the failing device is repaired by EMC field engineers. Unlike Dual Copy, the EMC subsystem does not provide an external control mechanism to enable the user to selectively initiate and disconnect mirrored pairs. Therefore, the EMC mirroring feature can not be used as a migration facility.
Standard mirroring has a major restriction that prevents its universal utilization as a transparent data migration tool. The source and target volumes must be attached to the same logical control unit and therefore data can only be relocated within a single control unit. Although limited, this capability is an important tool to the storage administrator.
Remote Mirroring
IBM 3990-6 and EMC Symmetrix features support remote mirroring. A remote mirror function exists when paired devices can exist on different control units and subsystems. The primary objective of this function is to provide a disaster recovery capability. However, a remote mirroring function can also be used as a data migrator.
DF/SMS eXtended Remote Copy (XRC) is a host-assisted remote mirror method that uses components of DF/SMS and DFP (Data Facility Product). The major component is the System Data Mover (SDM). This component is also used for Concurrent Copy. An IBM 3990-6 (or compatible) host is required as the primary or sending control unit. The secondary or receiving control unit can be an IBM 3990-3 or -6 or compatible host.
Other characteristics of XRC include:
Host application response time is not impacted because updates are reflected on the secondary volume asynchronously. The application does not have to wait for the new data to be copied to the secondary volume. The SDM reads data from the IBM 3990-6 "sidefile" and records the update on log files and then writes to the remote mirror on the recovery control unit.
A common timer is required to insure updates are processed in the correct order and therefore target data integrity is guaranteed in a multiple system environment.
No dynamic takeover is supported. Intervention is required in order to utilize the secondary copy.
To invoke XRC as a data migration facility, the following steps are required. After identification of the source and target pair of devices, an image copy begins and the session is placed in a "duplex pending" state. All users of the source volume have total read and write access to the data. Source updates are reflected on to the target volume. When the copy is complete, the session enters the "duplex" state. The operator must query the pair in order to determine this state change. At this time, all applications using the source volume must be brought down in order to synchronize the source and target devices. Final synchronization is determined by operator command (XQUERY). This query displays a timestamp that represents the time of last update so that the operator can be assured that all updates have been reflected on the target.
Although XRC does have some of the requirements for transparent migration, XRC does not have all of them.
XRC is transparent to the end user and the application program. No application program changes are required and the facility is dynamically activated.
The data on the source volume is available for read and write access during the XRC migration.
XRC causes a disruption because XRC does NOT support a dynamic and nondisruptive takeover of the target volume when the source and target volumes are synchronized. The impact of this disruption can be expensive. All applications with data resident on the source volume must be disabled during the takeover process.
XRC ensures complete data integrity through the use of the journaling data sets and a common timer.
XRC is a relatively "open" facility and therefore supports a multiple vendor environment. Any vendor that supports the IBM 3990-6 XRC specification can participate as the sending or receiving control unit in an XRC session. Any vendor that supports the IBM 3990-3 or basic mode IBM 3990-6 specification can participate as the receiving control unit in an XRC session.
XRC is complex to use and therefore is operationally expensive and resource intensive.
The IBM 3990-6 host also supports a feature that is called Peer-to-Peer Remote Copy (PPRC). PPRC is host independent and therefore differs from XRC in several ways. First, there is a direct ESCON (Enterprise Systems Connection) fiber link from one IBM 3990-6 host to another IBM 3990-6 host. With this fiber link connection, the primary IBM 3990 host can directly initiate an I/O Input/Output operation to the secondary IBM 3990 host. Secondly, PPRC operates as a synchronous process which means that the MVS (Multiple Virtual Systems) host is not informed of I/O completion of a write operation until both the primary and secondary IBM 3990 host control units have acknowledged that the write has been processed. Although this operation is a cache-to-cache transfer, there is a performance impact which represents a major differentiator of PPRC over XRC. The service time to the user on write operations for PPRC is elongated by the time required to send and acknowledge the I/O to the secondary IBM 3990 host.
The link between the IBM 3990 host controllers utilize standard ESCON fiber but does require an IBM proprietary protocol for this cross controller communication. This proprietary link restricts the use of PPRC to real IBM 3990-6 host controllers only and therefore does not support a multiple vendor environment.
As suggested above, PPRC can also be used as a migration facility. PPRC requires a series of commands to be issued to initiate and control the migration and is therefore resource intensive. IBM has a marketing tool called the PPRC Migration Manager that is used to streamline a migration process with the use of ISPF (Interactive Structured Program Facility) panels and REXX (Restructured Extended Executor) execs.
A migration using PPRC (Release 1) does not support an automatic takeover to the secondary device. In March of 1996, IBM announced an enhancement to PPRC called P/DAS, PPRC Dynamic Address Switch, which apparently when available eliminates the requirement to bring down the applications in order to perform the takeover of the target device. Therefore, P/DAS may allow I/O to be dynamically redirected to the target volume when all source data has been copied to that device.
Use of P/DAS is restricted to IBM 3990-6 controllers and is supported only in an MVS/ESA (Multiple Virtual Systems/Enterprise Systems Architecture) 5.1 and DFSMS/MVS 1.2 environment. Therefore the enhancement offered by P/DAS is achieved at the cost of prerequisite software. Furthermore, the dynamic switch capability is based on the PPRC platform and therefore supports only a IBM 3990-6 environment.
Although PPRC does have some of the requirements for transparent migration, PPRC does not have all of them.
PPRC is transparent to the end user and the application program. No application program changes are required and the facility is dynamically activated.
The data on the source volume is available for read and write access during a PPRC migration.
P/DAS apparently supports a dynamic and nondisruptive takeover of the target volume when the source and target volumes are synchronized.
PPRC ensures complete data integrity because a write operation will not be signaled complete until the primary and the secondary IBM 3990 control units have acknowledged the update request. This methodology will elongate the time required to perform update operations.
PPRC requires a proprietary link between two control units manufactured by the same vendor. For example, only IBM 3990-6 control units can participate in an IBM PPRC migration. Therefore PPRC does NOT support a multiple vendor environment.
PPRC is complex to use and therefore is operationally expensive and resource intensive.
EMC Corporation's remote mirroring facility is called Symmetrix Remote Data Facility (SRDF). The SRDF link is proprietary and therefore can only be used to connect dual Symmetrix 5000 subsystems.
SRDF has two modes of operation. The first is a PPRC-like synchronous mode of operation and the second is a "semi-synchronous" mode of operation. The semi-synchronous mode is meant to address the performance impact of the synchronous process. In the synchronous mode, the host is signaled that the operation is complete only when both the primary and the secondary controllers have acknowledged a successful I/O operation. In the semi-synchronous mode, the host is signaled that the operation is complete when the primary controller has successfully completed the I/O operation. The secondary controller will be sent the update asynchronously by the primary controller. No additional requests will be accepted by the primary controller for this volume until the secondary controller has acknowledged a successful I/O operation. Therefore in the SRDF semi-synchronous mode, there may one outstanding request for each volume pair in the subsystem.
EMC personnel must be involved in all SRDF activities unless the user has installed specialized host software that is supplied by EMC. The proprietary nature of SRDF restricts its use as a data migration facility. The primary function of SRDF is to provide data protection in the event of a disaster.
Late in 1995, EMC announced a migration capability that is based on the SRDF platform. This facility allows a Symmetrix 5000 to directly connect to another vendor's subsystem. The objective of the Symmetrix Migration Data Service (SMDS) is to ease the implementation of an EMC subsystem and is not meant to be a general purpose facility. SMDS has been called the "data sucker" because it directly reads data off another control unit. The data migration must include all of the volumes on a source subsystem and the target is restricted to a Symmetrix 5000.
An EMC Series 5000 subsystem is configured so that it can emulate the address and control unit type and device types of a existing subsystem (the source). This source subsystem is then disconnected from the host and attached directly to the 5000. The 5000 is then attached to the host processor. This setup is disruptive and therefore does cause an application outage.
The migration begins when a background copy of the source data is initiated by the 5000 subsystem. Applications are enabled and users have read and write access to data. When the target subsystem (the 5000) receives a read request from the host, the data is directly returned if it has already been migrated. If the requested data has not been migrated, the 5000 will immediately retrieve the data from the source device. When the target subsystem receives a write request, the update is placed only on the 5000 and is not reflected onto the source subsystem. This operation means that updates will cause the source and target volumes to be out of synchronization. This operation is a potential data integrity exposure because a catastrophic interruption in the migration process will cause a loss of data for any volume in the source subsystem that has been updated.
Although the Symmetrix Migration Data Service does have some of the requirements for transparent migration, SMDS does not have all of them.
a. SMDS is not transparent to the end user and the application program. Although no application program changes are required, the facility cannot be nondisruptively activated. All applications are deactivated so that the 5000 can be installed and attached to the host in place of the source subsystem. Specialized software is loaded into the Symmetrix 5000 to allow it to emulate the source subsystem and initiate the data migration. This disruption can last as long as an hour.
1. The data on the source volume is available for read and write access during a SMDS migration.
2. SMDS may support a dynamic and nondisruptive takeover of the target volume when the source and target volumes are synchronized. At the end of the migration, the source subsystem must be disconnected and the migration software must be disabled and it is unknown whether this is disruptive and an outage is required.
3. SMDS can link to control units manufactured by other vendors. However, the purpose of SMDS is to ease the disruption and simplify the installation of an EMC 5000 subsystem. Data can only be migrated to an EMC subsystem. Therefore SMDS does NOT support a multiple vendor environment.
SMDS does NOT ensure complete data integrity. During the migration, data is updated on the target subsystem and is not reflected on the source subsystem. A catastrophic error during the migration can cause the loss of all application updates.
The State of the Art in a Summary
The products that are available on the market today do not meet all of the data migration requirements.
The maintenance of continuous data availability is a fundamental mission of data centers. In order to support this goal, the migration facility must be initiated transparent to all applications and provide a means for the nondisruptive takeover of the target device.
The value of data and information is critical to the operation and competitiveness of the enterprise and therefore any exposure to possible data loss is unacceptable.
The control of the costs of the ever expanding storage assets is a large component in financing an information technology infrastructure. A competitive multiple vendor environment provides a mechanism to support effective cost management. Therefore, the migration facility should be vendor independent. Accordingly, there is a need for improved data migration methods and apparatus which have all the data migration requirements.
Summary of the Invention
The present invention is a data migration facility for managing and maintaining large computer systems and particularly for automatic movement of large amounts of data (migration of data) from one data storage location to another data storage location in the computer system.
The computer system has a plurality of storage volumes for storing data used in the computer system, one or more storage control units for controlling I/O transfers of data in the computer system from and to the storage volumes, one or more application programs for execution in the computer system using data accessed from and to the storage volumes, one or more operating system programs for execution in the computer system for controlling the storage volumes, the storage control units and the application programs, and a data migration program for migrating data from one of the data volumes designated as a source volume to one of said data volumes designated a target volume while the application programs are executing using data accessed from and to the storage volumes.
The data migration program includes a main module to control the start of a migration session when the application programs are using data accessed to and from the source volume, to migrate data from the source volume to the target volume, and to end the migration session whereby the application programs are using data accessed to and from the target volume The data migration program includes a volume module to control the volumes during the migration session. The data migration program includes a copy module to control the copying of data from the source module to the target module during the migration session. The data migration program includes a monitor module for monitoring I/O transfers to the data volumes during the migration sessions.
In one embodiment, the data migration program includes dynamic activation and termination, includes non-disruptive automatic swap processing that does not require operator intervention, is transparent to applications and end-users while providing complete read and write activity to storage volumes during the data migration, permits multiple migrations per session, permits multiple sessions. Additionally, the installation is non-disruptive (a computer program that can execute as a batch process rather than requiring the addition of a new hardware sub-system to the computer system), requires no IPL (Initial Program Load) of the operating system, is vendor independent with any-to-any device migration independent of DASD (Direct Access Storage Device) control unit model type or device type. The data migration program includes a communication data set (COMMDS) located outside the DASD control unit which helps ensure vendor independence.
The data migration facility has complete data integrity at all times, provides the ability to introduce new storage subsystems with minimal disruption of service (install is disruptive), allows parallel or ESCON channel connections to ensure vendor independence and can be implemented as computer software only, without need for dependency on hardware microcode assist.
The foregoing and other objects, features and advantages of the invention will be apparent from the following detailed description in conjunction with the drawings.
Brief Description of the Drawings
FIG. 1 depicts a block diagram of an enterprise computer system.
FIG. 2 depicts further details of the enterprise system of FIG. 1.
FIG. 3 depicts a single MVS operating system embodiment for migrating data from a source volume to a target volume.
FIG. 4 depicts a multiple MVS operating system embodiment for migrating data from a source volume to a target volume.
FIG. 5 depicts a flow diagram of the different states of the migration process.
FIG. 6 depicts a block diagram of the different software components that control the master slave operation used to perform the migration process.
FIG. 7 depicts a block diagram of the copy sub-task components that control the copying of data in the migration process.
FIG. 8 depicts a block diagram of the customer I/O request components that control the operation of I/O requests during the migration process.
FIG. 9 depicts a block diagram of the interrelationship of the different software components used to perform the migration process.
FIG. 10 depicts the source and target volumes before a TDMF migration.
FIG. 11 depicts the source and target volumes after a TDMF migration.
Detailed Description
Enterprise System--FIG. 1
The Transparent Data Migration Facilities (TDMF) is resident as an application program under the MVS/ESA host operating system of the enterprise system 1 of FIG. 1. The Transparent Data Migration Facilities (TDMF) is an application program among other application programs 2. The TDMF migration provides model independence, operational flexibility, and a completely transparent migration.
In FIG. 1, the enterprise system 1 is a conventional large-scale computer system which includes one or more host computer systems and one or more operating systems such as the MVS operating system. The operating systems control the operation of the host computers and the execution of a number of customer applications 2 and a TDMF application. The TDMS application is used for migrating data from one location in the enterprise system to another location in the enterprise system. The data locations in the enterprise system 1 are located on storage volumes 5. Data is transferred to and from the storage volumes 5 under control of the storage control units 4. The architecture of the FIG. 1 system is well-known as represented, for example, by Amdahl Corporation of Sunnyvale, Calif. (Amdahl) and IBM mainframe systems.
For efficient operation of the enterprise system 1, it is necessary at times to migrate data from one of the storage volumes 5 to another one of the storage volumes 5. The storage volumes 5 between which the migration occurs may be located at the same place under control of a single operating system or may be located in volumes under the control of different operating systems. Also, some of the volumes may be located geographically remote such as in different cities, states or countries. When located remotely, the remote computers are connected by a high-speed data communication line such as a T3 line.
The TDMF migration is intended for flexibility, for example, so that multiple MVS/ESA releases are supported (4.2, 4.3, 5.1, 5.2; OS390 V1.1, V1.2, V1.3, and V2.4), so that shared data system environments are supported, so that CKD/E (Count Key Data/Extended) compatible 388x and 399x control units are supported (Read Track CCW (Channel Control Word) is required in the disclosed embodiment), so that 3380 and 3390 device geometries are supported, so that flexible device pairing options are possible (uses device pairs with equal track sizes and numbers of cylinders and requires the target volume to be equal to or greater than the source volume), so that a single TDMF session can support up to 640 concurrent migrations, so that a single TDMF session can support concurrent migrations with differing control unit and device types, and so that optional point-in-time capability is available.
The TDMF migration is intended to have an ease of use that includes, for example, easy installation, a simple parameter driven process, a centralized control and monitoring capability, and the ability to have integrated online help and documentation.
The TDMF migration is intended to have a minimal impact on performance featuring, for example, a multiple tasking design, efficient user I/O scanning, asynchronous copy and refresh processes, minimization of device logical quiesce time and integrated performance measurement capability.
The TDMF migration is intended to have application transparency including, for example, dynamic installation and termination, no requirement for application changes, continuous read and write access, dynamic enablement of MVS intercepts and dynamic and nondisruptive takeover of target devices.
The TDMF migration is intended to have data integrity including, for example, continuous heartbeat monitoring, dynamic error detection and recovery capability, passive and nondestructive I/O monitoring design, optional data compare capability and maintenance of audit trails.
Multiple Operating System Environment--FIG. 2
In FIG. 2, the multiple operating systems 3 include the MVS operating systems 3-1, 3-2, . . . , 3-M. These operating systems 3 are typically running on a plurality of different host computers with one or more operating systems per host computer. Each of the operating systems 3-1, 3-2, . . . , 3-M is associated with a plurality of application programs including the application programs 2-1, 2-2, . . . , 2-M, respectively. The application programs 2-1, 2-2, . . . , 2-M are conventional application programs as would be found in a customer enterprise system. In addition to the conventional application programs, the TDMF application program 2-T is also present on each of the operating systems 3-1, 3-2, . . . , 3-M that are to be involved in a data migration.
In FIG. 2, each of the operating systems 3-1, 3-2, . . . , 3-M is associated with a storage control unit complex 4 including the storage control unit complexes 4-1, 4-2, . . . , 4-M, respectively.
Each of the storage control units 4-1, 4-2, . . . , 4-M is associated with one or more volumes. In the example shown, the storage control unit 4-1 is associated with the data storage volumes 5-1.sub.1, . . . , 5-1.sub.V1. Similarly, the storage control unit 4-2 is associated with the volumes 5-2.sub.1, 5-2.sub.V2. Finally, the storage control unit 4-M is associated with the volumes 5-M.sub.1, . . . , 5-M.sub.VM.
The operation of the FIG. 2 data storage is conventional in the MVS environment. Specifically, any one of the applications 2-1, 2-2, . . . , 2-M may store or retrieve data from any one of the volumes 5 under control of the operating systems 3-1, 3-2, . . . , 3-M through the storage control units 4-1, 4-2, . . . , 4-M.
At any time, any one of the volumes 5 may be designated for replacement or otherwise become unavailable so that the data on the volume must be moved to one of the other volumes of FIG. 2. For example, the data of a source volume X may be migrated to a target volume Y. In FIG. 2, by way of example, volume 5-1.sub.1 has been designated as the source volume and volume 5-M.sub.1 has been designated as the target volume. The objective is to move all the data from the source volume X to the target volume Y transparently to the operation of all of the applications 2-1, 2-2, . . . , 2-M so that the enterprise system of FIG. 2 continues operating on a continuous 24-hour by seven day data availability without significant disruption.
Single MVS Operating System Environment--FIG. 3
In FIG. 3, the single master MVS operating system 3-1 controls the operation of the applications 2-1 on a source volume 5.sub.X. At some time, the enterprise system requires that the source volume 5.sub.X be taken out of service requiring the data on the source volume 5.sub.X to be migrated to a target volume 5.sub.Y. The operating system 3-1 controls transfers to and from the source volume 5.sub.X through the control unit 4.sub.X. Similarly, the MVS operating system 3-1 controls transfers to and from the target volume 5.sub.Y through the storage control unit 4.sub.Y. The data migration from the source volume 5.sub.X to the target volume 5.sub.Y is under control of the TDMF application 2-T.sub.1. In order to achieve the transparent data migration from the source volume 5.sub.X to the target volume 5.sub.Y, an additional data volume 5-1 which is not involved in the data migration is available to the MVS operating system 3-1. During the entire migration from the source volume 5.sub.X to the target volume 5.sub.Y, the applications 2-1 continue their operation without disruption and while maintaining complete data integrity.
Multiple Operating System Environment Example--FIG. 4
In FIG. 4, the MVS operating systems 3-1, 3-2, . . . , 3-M are all part of the enterprise system 1 of FIG. 1. Each of the operating systems 3-1, 3-2, . . . , 3-M is associated with corresponding application programs 2-1, 2-2, . . . , 2-M. Each of the applications 2-1, 2-2, . . . , 2-M is operative under the control of the MVS operating systems 3-1, 3-2, . . . , 3-M to access the storage volumes 5 through the storage control units 4-1, 4-2, . . . , 4-M.
Specifically, the storage control units 4-1, 4-2, . . . , 4-M control accesses to and from the volumes 5-1, 5-2, . . . , 5-M, respectively. Specifically, the volumes 5-1 include the volumes 5-1.sub.1, . . . , 5-1.sub.V1, the volumes 5-2 include the volumes 5-2.sub.1, . . . , 5-2.sub.V2, . . . , and the volumes 5-M include the volumes 5-M.sub.1, . . . , 5-M.sub.VM, respectively.
At some point in time it becomes desirable in the enterprise system of FIG. 4 to migrate data from a source volume VOL.sub.X to a target volume VOL.sub.Y. In the example of FIG. 4, the source VOL.sub.X is designated as 5-1.sub.1 controlled through the SCU.sub.X 4-1 and the target volume VOL.sub.Y is designated 5-M.sub.1 and is controlled through the SCU.sub.Y designated 4-M. The data migration occurs from the source VOL.sub.X to the target VOL.sub.Y without interruption of the customer applications 2-1, 2-2, . . . , 2-M.
In order to control the data migration from the source VOL.sub.X to the target VOL.sub.Y, the TDMF application 2-T operates in a distributive manner across each of the operating systems 3-1, 3-2, . . . , 3-M. Specifically, the operation of the TDMF migration application is in the form of a master/slave implementation. Each of the operating systems 3-1, 3-2, . . . , 3-M includes a corresponding instance of the TDMF application, namely, the TDMF application 2-T. One of the applications is designated as a master TDMF application and the other of the applications is designated as a slave TDMF application. In the example of FIG. 4, the TDMF application associated with the MVS 3-1 is designated as the master TDMF application 2-T.sub.mas. Each of the other operating systems 3-2, . . . , 3-M is associated with a slave application 2-T.sub.s11, . . . , 2-T.sub.s1M, respectively.
The data migration in the FIG. 3 and FIG. 4 systems is carried out with a TDMF application 2-T (with a separate instance of that application for each MVS operating system) without necessity of any modified hardware connections.
TDMF Migration Stages--FIG. 5
In FIG. 5, the relationship between the migration stages for migrating data between a source volume and a target volume is shown. Specifically, the migration commences with an INITIALIZATION/ACTIVATION phase 10, followed by a COPY phase 11, followed by a REFRESH phase 12, followed by a QUIESCE phase 13, followed by a SYNCHRONIZE phase 14, followed by a REDIRECT phase 15, followed by a RESUME phase 16, and ending in a TERMINATION phase 17. If during the INITIALIZATION/ACTIVATION phase 10, an error occurs, the error is detected by the ERROR module 20 which passes the flow to the TERMINATION stage 17. If an error is detected during the COPY phase 11, the ERROR module 21 passes the flow to the TERMINATION stage 17. If an error is detected during the REFRESH phase 12, the ERROR module 22 passes the flow to the TERMINATION phase 17.
If an error occurs during the QUIESCE phase 13 or the SYNCHRONIZE phase 14, the ERROR modules 23 and 24, respectively, pass the flow to the RESUME phase 16. If an error occurs during the REDIRECT phase 15, the ERROR module 25 passes the flow to the BACKOUT module 26 which then returns the flow to the RESUME phase 16.
The migration phases of FIG. 5 are active in each of the MVS operating systems 3 and the MASTER TDMF application and the MASTER MVS operating system insure that one phase is completed for all operating systems, both master and all slaves, before the next phase is entered.
MASTER/SLAVE System--FIG. 6
The MASTER/SLAVE system operation is under control of the TDMF application 2-T. That application, and each instance thereof in an MVS operating system includes the modules of FIG. 6. Specifically, the TDMFMAIN module is the main task which calls other ones of the modules for controlling the phasewise execution of the migration as set forth in FIG. 5. The TDMFMAIN module starts the TDMF session of FIG. 5 on the master and slave systems, opens all necessary files, reads and validates control cards and validates all parameters and volumes to be involved in the migration session.
The TDMFVOL module controls the migration process of any and all volumes being migrated within the TDMF session of FIG. 5.
The TDMFICOM generates channel programs to implement I/O operations being requested to the COMMUNICATIONS DATA SET (COMMDS) via parameters passed by the caller. The caller may request that the system initialize the COMMDS or to selectively read or write caller specified TDMF control blocks as required by the controller. The TDMFICOM module is called by the TDMFMAIN module only.
The TDMFIVOL module generates channel programs to implement I/O operations being requested to volumes being migrated to the IDMF session of FIG. 5 via parameters passed by the caller.
The TDMFSIO module issues a request to the MVS INPUT/OUTPUT SUPERVISOR (IOS) component to perform the I/O operation represented by the INPUT/OUTPUT SUPERVISOR control BLOCK (IOSB) in conjunction with its SERVICE REQUEST BLOCK (SRB) as requested by the caller. The IDMFSIO module can be called from the TDMFICOM module and the TDMFIVOL module. Upon completion of the I/O operation requested, control is returned to the calling module.
COPY Sub-task--FIG. 7
In FIG. 7, the COPY Sub-task functions with one per volume migration for the master only. The COPY Sub-task includes the TDMFCOPY module which provides the functions required by the TDMF COPY Sub-task. The TDMFCOPY Sub-task is attached as a Sub-task by the module TDMFVOL by means of an ATTACHX macro during its processing. The TDMFCOPY module implements three Sub-task phases, namely, the COPY sub-phase, the REFRESH sub-phase, and the SYNCHRONIZATION sub-phase.
The TDMFIVOL module may also be called by the TDMFCOPY module to generate channel programs to implement I/O operation being requested to volumes being migrated to the TDMF session of FIG. 5 via parameters passed by the caller.
The TDMFSIO module when called by the TDMFIVOL module issues a request to the MVS INPUT/OUTPUT SUPERVISOR (IOS) component to perform the I/O operation represented by the INPUT/OUTPUT SUPERVISOR CONTROL BLOCK (IOSB) in conjunction with its SERVICE REQUEST BLOCK (SRB) as requested by the caller. Upon completion of the I/O operation requested, control is returned to the TDMFIVOL module.
I/O Monitoring--FIG. 8
In FIG. 8, a block diagram of the monitoring of I/O operations during a migration session of FIG. 5 is shown. The FIG. 8 modules are conventional with the addition of the TDMFIMON module. The purpose of the TDMFIMON module is to monitor all customer I/O operations to the source and target volumes during the life of an active migration session of FIG. 5. The TDMFIMON module insures that the primary design objective of insuring data integrity of the target volume by insuring that any update activity by customer I/O operation that changes the data located on the source volume will be reflected on the target volume. The TDMFIMON module only monitors volumes involved in a migration session of FIG. 5 and any I/O operations to volumes not involved in the TDMF migration session are not impacted in any fashion.
TDMF Module Overall Architecture--FIG. 9
In FIG. 9, the overall architecture of the computer software for implementing the TDMF migration of FIG. 5 is shown. The TDMFMAIN module functions to control systems initialization and system determination and calls the TDMFVOL module, the TDMFICOM module, and the TDMFIVOL module.
The TDMFVOL module is responsible for volume initialization with the TDMFIVOL module, volume activation with the TDMFAVOL module, volume quiesce with the TDMFQVOL module, volume resume with the TDMFRVOL module and volume termination with the TDMFTVOL module. The TDMFVOL module calls or returns to the TDMFMAIN module, the TDMFICOM module, the TDMFIVOL module, and the TDMFCOPY module.
The TDMFCOPY module is called by and returns to the TDMFVOL module and the TDMFIVOL module.
The TDMFICOM module calls and is returned to the TDMFSIO module.
The TDMFIVOL module calls and is returned to the TDMFSIO module.
Details of the various modules of FIG. 6, FIG. 7, FIG. 8 and FIG. 9 for performing the migration in the phases of FIG. 5 are described in detail in the following LISTING 1 near the end of the specification before the claims.
Source and Target Volumes Before TDMF Migration--FIG. 10
In FIG. 10, a data migration is to occur between the volume 5-1.sub.1 as the source through the storage control unit SCU.sub.X designated 4-1 to the target volume 5-M.sub.1 through the storage control unit SCU.sub.Y designated 4-M. The SOURCE volume has device address A00, serial number SRC001 and is in the ON-LINE status before the TDMF migration. The target volume has the device address FC0, serial number TGT001 and is in the ON-LINE status before volume migration. Before the volume migration begins, all I/O operations are being directed to the source volume 5-1.sub.1.
Source and Target Volumes After All TDMF Migration--FIG. 11
In FIG. 11, the source volume 5-1.sub.1 has a device address of A00, a serial number of TGT001 and an OFF-LINE status. The target volume 5-M.sub.1 has a device address of FC0, a serial number of SRC001, and an ON-LINE status.
TDMF Migration Operation
TDMF is initiated as an MVS batch job. TDMF's job control language identifies the system type, session control parameters, and the communications data set (COMMDS). The session control parameters define the paired source (VOL.sub.X) and target (VOL.sub.Y) volumes which make up the migrations under the control of this session. The control parameters also define the master system and all slave systems. The COMMDS set is the mechanism that is used to enable all systems to communicate and monitor the health of the migrations in progress. The COMMDS set is also used as an event log and message repository.
All systems that have access to the source volume must have TDMF activated within them. This requirement is critical in order to guarantee data integrity. One system is designated as the master (PARM=MASTER) and all other systems are designated as slaves (PARM=SLAVE).
The responsibilities of the master system include:
Initialization of the master TDMF environment and the COMMDS.
Validation of the initialization of all slave systems.
The control of all migrations defined in this TDMF session.
The monitoring of source volume user I/O activity in order to detect updates.
The copying of data from the source to the target volume.
Collection of update information from all systems.
The refresh and synchronization of the target volume due to detected update activity.
The monitoring of the internal health of the master environment and the health of all slave systems.
The responsibilities of the slave systems include:
Initialization of the slave TDMF environment.
Establishment of communication to the master through the COMMDS.
The processing and acknowledgment of all migration requests as received from the master.
The monitoring of source volume user I/O activity in order to detect updates.
Transmission of update information to the master through the COMMDS.
The monitoring of the internal health of this slave environment and the health of all other systems.
The master system initiates and controls all migrations. A migration is broken into the major phases as illustrated in FIG. 5. The master initiates each phase and all slave systems must acknowledge in order to proceed. If any system is unable to acknowledge or an error is detected, the systems will cleanup and terminate the migration. To ensure data integrity, TDMF is designed with a passive monitoring capability so that all user I/O operations to the source volume will complete as instructed by the application. This completion means that if any unexpected event occurs, the data on the source volume is whole. TDMF will merely cleanup and terminate the migration.
Standard cleanup and recovery is active for all phases with the exception of the REDIRECT phase. During this REDIRECT phase, it may be necessary to take additional recovery actions. This specialized recovery is called the BACKOUT phase. Backout is required due to error detection in a multiple system environment. In a multiple system migration, the systems may be at differing stages within the REDIRECT phase at the point the error is detected. Some of the systems may have completed redirect and others may have not completed redirect processing. Backout recovery dynamically determines the state of each system in order to take the appropriate recovery action. However, even in the REDIRECT and BACKOUT phases, the data on the source volume is whole and user data will not be lost.
Referring again to FIG. 5, a TDMF session begins with an INITIALIZATION phase, immediately followed by an ACTIVATION phase. During this phase, all systems confirm the validity of the source and target volumes and enable user I/O activity monitoring. When all systems acknowledge successful completion of this process, the master will begin the COPY phase and initiate a sub-task (COPY Sub-task) to copy data from the source to the target volume. This COPY Sub-task is a background activity and will copy data asynchronously from the source volume to the target volume. There is an independent COPY sub-task for each active migration.
I/O monitoring provided by the TDMFIMON module of FIG. 8 provides the ability to detect updates to the source volume. A system that detects a source volume update will note this update in a refresh notification record and send this information to the master system.
When a COPY sub-task has completed one pass of the source volume, the REFRESH phase is initiated by the master. During this phase, the updates that have been made to the source volume are reflected onto the target volume. The REFRESH phase is divided into multiple cycles. A refresh cycle is a pass through the entire source volume processing all updates. TDMF measures the time required to process the refresh operations in a single refresh cycle. The refresh cycles will continue until the time for a refresh cycle is reduced that allows the Synchronization Goal to be achieved to a threshold that is based on the collected performance data. When the threshold is reached, this event signals the master to be ready to enter the Quiesce and Synchronization phases. If there are multiple REFRESH phases during a migration, it is due to the inability of TDMF to meet the SYNCHRONIZATION goal. This is usually because of high write activity on the Source volume. If there are no issues with this, then there is no reason to change the SYNCHRONIZATION goal parameter.
Prior to the Synchronization phase, a QUIESCE is issued to the Source volume. In the instance of a multi-system migration, the Master issues a request to the Slave(s) to Quiesce all I/O to the Source volume (from the slave side). At this time the final group of detected updates are collected and applied to the Target volume (SYNCHRONIZATION). At the end of Synchronization, the Master starts the volume REDIRECT (swap) phase. The Target volume has now become the new Source volume. When all systems have verified the Redirect, the Master initiates the RESUME phase so that user I/O can continue to the new source volume. The elapsed time between the last QUIESCE phase and the RESUME phase is approximately four (4) seconds plus the ACTUAL SYNCHRONIZATION time (which should always be less than the specified synchronization goal).
The Synchronization Goal default is five (5) seconds. Synchronization will not occur unless the calculated synchronization time is less than the goal. If the synchronization goal is increased, then the time the user I/O is queued (quiesced) is greater. If the value 999 is used, this equates to synchronize as soon as possible; it does not matter how long it takes. This can be a significant amount of time depending on the write activity of the source volume. Therefore, use discretion when changing this value.
Application I/O operations during the Copy and Refresh phases are impacted no more than if a backup utility is backing up the source volume at the same time as the application is executing. The Synchronization goal parameter may be specified for each volume migration, allowing the customer to specify the amount of time (in seconds) that he will allow the Synchronization phase to execute. This is the maximum amount of time.
Performance Impact by Phase
The Master and Slave(s) wake up for processing based upon a variable which is the minimum number of seconds based upon any migration volume's current phase or stage. The phases of a volume migration and their associated time intervals are:
This allows TDMF to be responsive with a minimum of CPU overhead.
The CPU overhead associated with running IDMF is less than 3 percent on average for the Master system. This is dependent upon the number of storage volumes within a session and write activity against those source volumes. A Slave system's CPU overhead will be almost non-measurable.
For example, if the Master job takes 44 minutes, 22 seconds (2662 seconds total) to migrate 16 volumes, and the TCB time is 63.5 seconds, and the SRB time is 2.92 seconds, then the CPU overhead is equal to 2.49 percent ((63.5+2.92)/2662) for that session.
When a multi-volume migration is running in a TDMF session, not all volumes are going to be in the same phase at the same time. This is because different volumes may have different activity against them. The number of channels available to each Control Unit (CU) will also be a factor in this. Therefore, it is entirely possible to have a 4 volume migration running with volume 1 in the copy phase, volume 2 in a 4th refresh phase, volume 3 completed, and volume 4 is in the Synchronization phase.
Placement of the Master
The Master should be placed on the system that has the most updates or on the system where the owning application is executing. If multiple TDMF Master sessions are being run on multiple operating systems, then the MVS system(s) must have a global facility like GRS or MIM. This is to prevent inadvertent usage of the same volumes in a multi-session environment. If a GRS type facility is not available in the complex, then all Master sessions must run on the same operating system.
Placement of the Communications Dataset
The Communications Dataset (COMMDS) should be placed on a volume with low activity and the volume must not be involved in any migration pairing.
Sample Performance Statistics
In the following tables, time is designated in hours (H), minutes (MM), and seconds (SS.ss) and in the form H:MM:SS.ss; for example, 1:07:20.36 is 1 hour, 7 minutes and 20.36 seconds.
Master System Responsibilities
Initializing the master TDMF environment and the COMMDS.
Starting and controlling the migrations for all TDMF systems.
Monitoring source volume user I/O activity to detect updates.
Copying data from the source to the target volume.
Processing detected updates from all systems.
Performing refresh operations to the target volume to reflect update activity.
Checking the internal health of the master environment and the health of all slave systems.
Slave System Responsibilities
Initializing the slave TDMF environment and establishing communications to the master, using the COMMDS.
Acknowledging and processing migration requests from the master.
Monitoring source volume user I/O activity and detecting updates.
Notifying the master of update activity through the COMMDS.
Checking the internal health of the slave environment and the health of all systems.
At this time, the master issues the quiesce request to all systems so that activity to the source volume will be inhibited. When the quiesce request is received by a slave system, this is the indication for the slave system to enter the QUIESCE phase and to quiesce and send to the master the final group of detected updates so that the master may perform the synchronization process or optionally S restart an additional REFRESH phase.
When all systems have successfully acknowledged the quiesce request, synchronization will begin in the SYNCHRONIZATION phase. The device quiesce is transparent to all applications. An application can issue I/O requests; however, they will not be executed and will be queued until the RESUME phase. The purpose of the collection of performance information during the REFRESH phase is to minimize the device quiesce time. At the conclusion of the SYNCHRONIZATION phase, the master will initiate the REDIRECT phase.
The purpose of the REDIRECT phase is to cause I/O activity to be redirected to the target volume so that it can become the primary device. The master will request that all systems perform and confirm a successful redirect before the redirect operation is performed by the Master system. When all systems have acknowledged this activity, the master will then initiate the RESUME phase request so that any queued user I/O will begin execution. Subsequent I/O operations will then be directed to the target device.
After all systems have processed the resume request, I/O monitoring will be disabled and the migration will terminate in the TERMINATE phase.
The TDMF facility simplifies the relocation of user-specified data from existing hardware to new storage subsystems, for example, without loss of access to data, without the imposition of unfamiliar operator procedures, and without dependence upon vendor firmware or hardware release levels.
TDMF is transparent to the end user and the application program. No application program changes are required and the facility is dynamically activated.
TDMF provides for full data access during the data migration. The data on the source volume is available to the end user for read and write access.
TDMF supports a dynamic and nondisruptive takeover of the target volume when the source and target volumes are synchronized. All applications with data resident on the source volume may remain enabled during the takeover process.
TDMF ensures complete data in integrity through the use of passive I/O monitoring and background copy operations.
TDMF is an "open" facility and therefore supports a multiple vendor environment. Any vendor that supports the IBM 3880 or IBM 3990 ECKD specification can participate as the sending or receiving control unit in a TDMF migration. (Read Track CCW support is required).
TDMF operates in both a single CPU and/or multiple CPU environment with shared DASD.
With TDMF, the implementation of new storage subsystem technologies can be accomplished in a nondisruptive manner.
With TDMF, the value of existing storage equipment is protected because all devices can participate in a migration session.
With TDMF, a competitive multiple vendor environment is enhanced because TDMF is designed to be independent of any specific storage subsystem. ##SPC1##
Claims
What is claimed is:
1. A computer system having a plurality of storage volumes for storing data used in the computer system, one or more storage control units for controlling I/O transfers of data in the computer system from and to the storage volumes, one or more application programs for execution in the computer system using data accessed from and to the storage volumes, one or more operating system programs for execution in the computer system for controlling the storage volumes, the one or more storage control units and the one or more application programs, and a data migration program for migrating data from one of said volumes designated as a source volume to one of said volumes designated a target volume while said one or more application programs are executing using data accessed from and to the storage volumes, said data migration program including, a main module to control the start of a migration session when said one or more application programs are using data accessed to and from the source volume, to migrate data from the source volume to the target volume, and to end the migration session whereby said one or more application programs are using data accessed to and from the target volume, a volume module to control said storage volumes during the migration session, a copy module to control the copying of data from the source volume to the target volume during the migration session, a monitor module for monitoring I/O transfers to the storage volumes during the migration session.
2. The computer system of claim 1 wherein said main module includes a communication data set for communications used for controlling data migration that is not stored on said source volume or said target volume.
3. The computer system of claim 1 wherein said main module establishes said migration session as a plurality of migration phases for controlling data migration.
4. The computer system of claim 1 wherein each of said one or more operating system programs includes an instance of said data migration program and said main module establishes one instance of said data migration program as a master instance and another instance of said data migration program as a slave whereby said migration session is controlled in a master slave relationship.
5. The computer system of claim 1 wherein said main module establishes said migration session as a plurality of migration phases including an activation phase, a copy phase, a refresh phase, a quiesce phase, a synchronize phase, a redirect phase, a resume phase and a termination phase.
6. The computer system of claim 1 wherein each of said one or more operating system programs includes an instance of said data migration program and said main module establishes a plurality of migration sessions for concurrent data migrations.
7. The computer system of claim 6 wherein for each of said data migration sessions each of said operating systems includes an instance of said data migration program and said main module establishes one instance of said data migration program as a master instance and another instance of said data migration program as a slave whereby each of said migration sessions is controlled in a master slave relationship.