Using data copies for redundancy

[0001] The present patent application is a continuation of U.S. patent application Ser. No. 10/741,858, filed on Dec. 19, 2003, now U.S. Pat. No. 7,290,101, entitled “Using Data Copies for Redundancy” and is incorporated by reference herein in its entirety and for all purposes.

[0002] Many businesses use redundant copies to enable fast recovery from loss of data. Still other businesses create a point-in-time (PIT) copy of critical data to guard against corruption or inaccessibility of the critical data. Methods for creating PIT copies are well known in the art. One method involves copying data to a magnetic data storage tape via a tape drive using well known backup data storage systems. Other methods of creating a PIT copy involve copying data to a hard disk. Creating a PIT copy on hard disk is preferable to creating a PIT copy on a magnetic data storage tape since data is more quickly read from a hard disk than from to a magnetic data storage tape.

[0003] FIG. 1 illustrates relevant components of an exemplary data processing system 10 that creates a PIT copy. More particularly, FIG. 1 shows data processing system 10 having a primary node (e.g., a server computer system) 12 coupled to data storage systems 14 and 16 via data link 18. Data storage systems 14 and 16 include memories 20 and 22, respectively. Each of memories 20 and 22 includes a plurality of hard disks for storing data, it being understood that the term “memory” should not be limited thereto. The hard disks of memory 20 store a data volume V, while the hard disks of memory 22 store a PIT copy of volume V. Primary node 12 can access data in either data volume V or its PIT copy using input/output (IO) transactions transmitted via data link 18.

[0004] Primary node 12 may include a data storage management system (not shown) that takes form in software instructions executing on one or more processors (not shown). The data management system includes, in one embodiment, a file system, a system for creating data volumes, and a system for managing the distribution of data of a volume across one or more memory devices. FIG. 2 shows (in block diagram form) data volume V and its PIT copy. Data volume V and its PIT copy are shown having n_max blocks of data. Data of each block is stored in one or more hard disks allocated thereto. Corresponding data blocks in volume V and its PIT copy are equal in size. Thus, data block 1 of volume V can be equal in size to that of data block 1 of the PIT copy. Each of the data blocks in volume V may be equal in size to each other. Alternatively, the data blocks in volume V may vary in size.

[0005] As noted above, primary node 12 creates the PIT backup copy of volume V to guard against data corruption or data inaccessibility. When the PIT copy is first created, the hard disks allocated to store the PIT copy may contain no data. Creating a PIT backup copy of volume V is a procedure well known in the art. In essence, the procedure includes primary node 12 copying data from memory allocated to each data block of volume V to memory allocated to store data of a corresponding block of the PIT copy in a block by block process until the entire data content of volume V is copied to the PIT copy. After the PIT copy is created, primary node 12 logs each IO transaction that modifies data of volume V.

[0006] Subsequent to the creation of the PIT copy, data within the volume V may become inadvertently corrupted as a result of erroneous software and/or hardware behavior. Moreover, data within volume V may become inaccessible as a result of software and/or hardware failure. Primary node 12 can use the PIT copy to correct corrupted or inaccessible data in volume V. The process for correcting corrupted or inaccessible data, however, can be lengthy. To illustrate, suppose the hard disk memory space allocated to store data of block 2 of volume V becomes inaccessible as a result of hardware failure sometime after creation of the PIT copy. If primary node 12 attempts to access data of block 2, data storage system 14 will generate an IO error. When the error is generated, primary node 12 might create a third volume (designated R in FIG. 2) using unused hard disk memory spaces, copy the data contents of the PIT copy to volume R in a block by block fashion, and modify the contents of volume R according to the transactions logged by the primary node. When this process is complete, volume R should be identical in data content to the volume V, and this means that the memory allocated to block 2 in volume V should contain data identical to data contained in memory allocated to block 2 of volume R. Memory allocated to block 2 in volume R, however, is data accessible. Primary node may then allocate new (and accessible) memory space to block 2 of volume V. Once the new memory is allocated, primary node 12 copies data from memory allocated to block 2 of volume R to the newly allocated memory for block 2 of volume V. After copying data to the newly allocated memory, primary node 12 can access block 2 data using an IO transaction.

[0007] An apparatus and method implemented by a computer system of using data copies of a volume for redundancy when data of the volume is rendered corrupted or inaccessible. In one embodiment of the method a data volume is created. The data volume comprises a plurality of data blocks including a first data block. After creation of the data volume, a point-in-time (PIT) copy of the data volume is created, and a redirection map is created. The redirection map comprises a plurality of entries, wherein each entry of the map indicates whether memories allocated to store data of respective data blocks of the data volume and the PIT copy, contain identical data. Data of the data volume may become corrupt or inaccessible after creation of the PIT copy. For example, suppose an I/O transaction is generated to read data from or write data to memory allocated to store data of the first data block of the data volume, and it is discovered that the memory allocated to store data of the first data block is inaccessible as a result of physical damage to this memory. The redirection map and the PIT copy can be used to overcome inaccessibility of memory allocated to the first block. If an entry in the redirection map indicates that identical data is stored in memories allocated to the first data block of the data volume and the first data block of the PIT copy, respectively, data is read from memory allocated to store data of a first block of the PIT copy, and possibly stored in new memory allocated to store data of the first data block of the data volume

[0019] The use of the same reference symbols in different drawings indicates similar or identical items.

[0020] The present invention relates to the method or apparatus for using a data copy (e.g., a PIT copy of a volume V or a replica R of a volume V) for quickly and efficiently responding to an IO error caused by, for example, data corruption or data inaccessibility. In one embodiment, the method may be implemented by a computer system executing instructions stored on a computer readable medium.

[0021] FIG. 3 is a block diagram of a data processing system 30 employing one embodiment of the present invention, it being understood that the present invention should not be limited to use in a data processing system such as that shown in FIG. 3. System 30 includes a primary node 32 coupled to data storage systems 34 and 36 via data link 38. The term “coupled” should not limited to a direct connection between two components such as primary node 32 and data storage system 34 shown in FIG. 3. The term “coupled” is also meant to include an indirect connection between two components. For example, primary node 32 could be coupled to data storage system 34 via a SAN switch (not shown). System 30 also includes a secondary node 42 coupled to data storage system 44 via data link 46. Primary node 32 and secondary node 42 are coupled together via a communication link (e.g., the Internet) 48.

[0022] Data storage systems 34, 36, and 44 include memory systems (memories) 52, 54, and 56, respectively. Each of these memories may take form in one or more dynamic or static random access memories, one or more arrays of magnetic or optical hard disks, or combinations thereof. The term “memory” should not be limited to the foregoing hardware components; rather, memories 52-56 may take form in any hardware, software, or combination of hardware and software in which data may be persistently stored and accessed. The data memories may take form in complex construction of several hardware components operating under the direction of software. For purposes of explanation only, each of memories 52-56 will be described as containing multiple hard disks for storing data.

[0023] For ease of illustration, data memory 52 stores a data volume V, memory 54 stores a PIT copy of volume V, and memory 56 stores a replica R of volume V. It is noted that volume V may be stored in distributed fashion across memory 52 and 54, and that the PIT copy may be stored in distributed fashion across memory 52 and 54. Replica R is a real or near real time copy of volume V maintained using a synchronous, asynchronous, or periodic replication technique. Primary node 32 can access data volume V or the PIT copy by generating and sending 10 transactions to data storage systems 34 and 36 in response to receiving requests from client computer system (not shown) coupled to 32. Secondary node 42 and data storage system 44 are provided to protect against catastrophic failure of primary node 32 and/or data storage systems 34 and 36. To illustrate, if primary node 32 fails as a result of hardware or software error, requests to read or write data from client computer systems can be redirected from primary node 32 to secondary node 42. Secondary node 42 can respond to the requests to read or write data using the replica R contained within memory 56.

[0024] Primary node 32 may include a memory for storing instructions that, when executed by primary node 32, implements one embodiment of the present invention. Primary node 12 may also include a data management system (not shown) that takes form in software instructions executing on one or more processes (not shown). The data management system includes, in one embodiment, a file system, a system for creating a data volume, and a system for managing the distribution of data of volume V and its PIT copy across one or more hard disks. Volume Manager™ provided by VERITAS Software Corporation of Mountain View, Calif., is an exemplary system for managing the distribution of volume data across multiple hard disks. FIG. 4 represents in block diagram form, volume V and its PIT copy. Volume V and its PIT copy each includes n_max data blocks. Data in each block is stored in physical memory of one or more hard disks allocated thereto. These physical memory allocated to a data block need not be contiguous to each other on a single hard disk. FIG. 4 also shows replica R consisting of n_max data blocks. Physical memory of hard disks in memory 56 are allocated to store data for each data block n of replica R.

[0025] Corresponding data blocks in volume V, the PIT copy and replica R can be equal in size. Thus, data block 1 volume V is equal in size to data block 1 of the PIT copy and replica R. Each of the data blocks in volume V may be equal in size to each other. Alternatively, data blocks volume V may vary in size.

[0026] Volume V is the working volume of data processing system 30. In other words, primary node 32 reads or writes data to volume V using 10 transactions in response to receiving requests from client computer systems coupled thereto. The PIT copy of volume V may be real or virtual. In one embodiment, primary node 32 creates a virtual PIT copy using the methods described in copending U.S. patent application Ser. No. 10/143,059, entitled “Method and Apparatus for Creating a Virtual Data Copy,”, which is incorporated herein by reference in its entirety. A virtual PIT copy can be transformed into a real PIT copy using a background copying process described in the aforementioned patent application. The PIT copy may be initially created as real. For example, before primary node 32 modifies data in volume V, primary node 32 copies data from volume V to the PIT copy in a block by block fashion until respective blocks within volume V and the PIT copy contain identical data. A real PIT copy may also be created by “breaking off” a mirrored copy of volume V. A mirrored copy is a real or near real time copy of volume V. A mirrored copy closely tracks changes to the volume V. When primary node 32 writes new data to volume V, the new data is also written to the mirrored copy. A write two volume V is not considered complete until the data is also written to the mirrored copy. When a mirror is broken off, to create a real PIT copy, the mirror no longer tracks changes to volume V. Thus, after the mirrored volume is broken off, primary node 32 writes new data only to volume V.

[0027] Data within volume V is subject to corruption or inaccessibility. For example, primary node 32 may generate and send to data storage system 34 an IO transaction to read data from or write data to block 2 of volume V. As a result of the physical failure of the hard disk allocated to store data of block 2, data storage system 34 will generate an error message indicating that the data sought cannot be accessed. Rather than performing the lengthy and process intensive correction process described within the background section above, primary node 32 may recover from the error message by using a process which involves only data contained in block 2 of the PIT copy or block 2 of replica R. In one embodiment, this process may use a redirection map such as redirection map 72 or 76 shown in FIG. 5. The redirection maps shown in FIG. 5 include n_max entries corresponding to the n_max data blocks of volume V. Each of the entries within redirection map 72 consists of a single bit designated M_n while each entry of redirection map 76 consists of a counter designated C_n. Each entry M_n in redirection map 72 is set to logical 0 or logical 1 depending upon whether memories that store data of respective data blocks of volume V and the PIT copy, contain identical data. Thus, if memory allocated to block 3 of volume V contains data identical to that stored in memory allocated to block 3 of the PIT copy, M_n is set to logical 1, and if memory allocated to block 3 of volume V does not contain data identical to that stored in memory allocated to block 3 of the PIT copy, M_n is set to logical 0. FIG. 5 shows redirection map 72 with each entry M_n set to logical 0. The redirection map 72 may be created when the PIT copy is first created. If the PIT copy is initially created as a virtual volume, then M_n in each entry of redirection map 72 is set to logical 0 since no memory allocated to store data of a virtual PIT copy initially stores data. If, however, the PIT copy within logical memory 64 is created initially as real (e.g., a mirror break-off), each entry within redirection map 72 is initially set to logical 1, thus indicating that at the time of creation, memory allocated to store data of the PIT copy data blocks contain identical data to that stored in memory allocated to blocks of the volume V. Subsequent to creation of the PIT copy and redirection map 72, primary node may modify contents of block n of the volume V or its PIT copy. If primary node 32 modifies data of block n of volume V or its PIT copy such that block n volume V or its PIT copy no longer contain identical data, primary node 32 sets M_n within redirection map 72 to logical 0. In this fashion, redirection map 72 identifies respective blocks in volume V and its PIT copy, which contain identical data.

[0028] Redirection map 76 is also used to determine whether respective blocks within two volumes contain identical data. Redirection map 76 is used to determine whether respective blocks n of volume V and replica R contain identical data where replica R is maintained using asynchronous replication. Because replica R is maintained using asynchronous replication, a data block n of replica R may be out of sync with corresponding block n of volume V by more than one write data transaction. As such, a single bit redirection map is insufficient to determine whether corresponding blocks n within volume V and replica R are synchronized or contain identical data. Redirection map 76 uses counters C_n to determine synchronization between corresponding blocks n of volume V and replica R. When replica R is first initialized, memory allocated to blocks 1-n_max of replica R store no data, and counters C_n are set to −1 before the process to synchronize volume V and replica R begins. When corresponding memory blocks n are synchronized, counter C_n is set to 0 thus indicating that memory allocated to blocks n of volume V and replica R contain identical data. When full synchronization is complete between data volume V and replica R, all counters C_n are set to 0. Whenever primary node 32 writes new data to block n in volume V, primary node 32 increments C_n by 1. Primary node 32 subsequently decrements C_n by 1 when block n within replica R is updated with the new data and primary node 32 receives an acknowledgment from secondary node 42 that block n of replica R has been modified accordingly.

[0029] FIGS. 7-10 illustrate relevant operational aspects of using the PIT copy or the replica R as redundant copies to correct data corruption or inaccessibility in volume V. The processes described in FIGS. 7-10 make use of the redirection maps 72 or 76 in FIG. 5. The process described within FIG. 7-10 may be similar, but not identical to processes using a redundant mirror to overcome an instance in which data of volume V is corrupted or inaccessible. In one sense, the processes shown in FIGS. 7-10 presume the existence of a pseudo-mirror.

[0030] The process shown in FIG. 7 illustrates relevant aspects of using the PIT to recover from a data corruption or data inaccessibility error that results from an attempt to read data from volume V. More particularly, the process shown in FIG. 7 begins when primary node 32 generates an IO transaction to read data of block n of volume V. For purposes of explanation, it will be presumed that the hard drive memory space allocated to store data of block n, is inaccessible as a result of some hardware failure. As a result of the inaccessibility, data storage system 34 may generate an error message in response to receiving the IO transaction generated in 82. In response to receiving the error message, primary node 32 accesses redirection map 72 and determines the state of M_n therein as shown in step 84. If M_n is set to logical 0 in step 84, the data contents of block n of volume V and its PIT copy are presumed different, and the process ends in FIG. 7. If M_n is set to logical 0, primary node 32 may have use the traditional process described within the background section above to overcome the inability to access data in block n of volume V. However, if M_n is set to logical 1 in step 84, the process proceeds to step 86 where new memory is allocated to block n of volume V. It is presumed that the new memory is unused and functioning (i.e., capable of being accessed). Thereafter, primary node 32 copies the data stored in memory allocated to block n of the PIT copy to the newly allocated memory as shown in step 90. Once step 90 is performed, IO transactions to read data from block n of volume V can complete without having to go through the lengthy process described in the background section above. It is noted that if block n of data volume V initially contains corrupted data, block n can be corrected using the process shown in FIG. 7 without the allocation step 86, and with step 90 reduced to overwriting data stored in memory allocated to block n of volume V with data stored in memory allocated to block n of the PIT copy.

[0031] FIG. 8 illustrates relevant operational aspects employed when attempting to write to block n of volume V when, for example, the memory allocated to block n is inaccessible as a result of hardware failure. More particularly, FIG. 8 begins with step 92 when primary node 32 generates an IO transaction to write data D to block n of volume V. It is presumed that the physical memory space allocated to block n is inaccessible as a result of hardware and/or software failure. As a result, data storage system 52 will generate an error message. In response, primary node 32 will access redirection map 32 to determine the state of M_n. If M_n is set logical 0, the process ends and primary node 32 may be required to employ the process described in the background section above to overcome the failure of the physical memory allocated to store block n data. If, however, M_n set to logical 1 in step 94, primary node 32 allocates new memory to store block n data of volume V as shown in step 96. In step 100, primary node 32 copies data stored in memory allocated to block n of the PIT copy to the newly allocated memory to store block n of the volume V. Thereafter, data D is written to the newly allocated memory in accordance with the 10 transaction of step 92, and in step 104, primary node 32 sets M_n to logical 0 thus indicating that the memories currently allocated to respective blocks n of data volume V and its PIT copy, no longer store identical data.

[0032] FIG. 6 illustrates redirection map 72 shown in FIG. 5 after modification by primary node 32 to account for the modification of data block 3 of volume V in accordance to the process shown in FIG. 8. It is presumed within FIG. 5 that initially, all bits within redirection map 72 were set to logical 1. It is noted that primary node 32 may alter entries in the redirection map in response to modifying data in the PIT copy. For example, after creation of redirection map 72, primary node 32 modifies data in block 4 of the PIT copy. In response, primary node clears entry 4 of the redirection map 72 to indicate that the contents of blocks 4 in the volume V and its PIT copy are no longer identical. FIG. 6 Illustrates the result of clearing entry 4 in redirection map 72 to account for differences in blocks 4 of volume V and its PIT copy.

[0033] As noted above, replica R may be employed to correct an instance of data corruption or data inaccessibility of data within volume V. FIG. 9 illustrates relevant aspects of using replica R to recover from a data corruption or data inaccessibility error that results from an attempt to read data from volume V. The process shown in FIG. 9 begins with step 112 when primary node 32 generates an IO transaction for reading data from memory allocated to store block n data of volume V. Again, it is presumed that the memory allocated to block n of volume V is inaccessible as a result of hardware failure even though other memories allocated to store volume V data are accessible. In response to data inaccessibility, data system 14 generates an error message (not shown), and the process proceeds to step 114 where primary node 32 determines the counter value of C_n in redirection map 76. If the counter C_n is set to −1 or to a value greater than 0, then data in the memories are allocated to blocks n of volume V and replica R, do not contain identical data. Eventually, the memories allocated to blocks n will contain identical data when replica R when, for example, synchronous writes generated by primary node 32 complete at the secondary node 42. When block n data of replica R is identical to block n data of volume V, the process proceeds to step 116 where primary node 32 allocates new memory to store block n data of volume V. Thereafter, in step 120, primary node implements a process for copying the data contents from memory allocated to store block n of replica R to new memory allocated to block n of volume V. Once step 120 is performed, IO transactions to read data from block n of volume V can complete without having to go through the lengthy process described in the background section above. It is noted that if memory allocated to block n contains corrupted data rather than inaccessible data, this can be corrected using the process shown in FIG. 9 without step 116 and replacing step 120 with the step of copying data from memory allocated to store block n of replica R to memory allocated to store block n of volume V.

[0034] FIG. 10 illustrates relevant operational aspects of correcting an instance in which data cannot be written to volume V as a result of hardware error that precludes access to allocated memory. The process shown in FIG. 10 begins with step 122 where primary node 32 generates an IO transaction to write data D to memory allocated to store block n data of volume V. Like the process shown in FIG. 9, if memory allocated to store block n data is inaccessible as a result of hardware failure, primary node 32 accesses redirection map 76 to determine the value of counter C_n therein. If C_n is set to a value other than 0, memories allocated to store data of respective blocks n of volume V and replica R, lack identical data. Primary node 32 can simply delay further processing until blocks n of volume V and replica R are synchronized. Once synchronization occurs, or if C_n is initially set to 0, the process proceeds to step 126 where primary node allocates new memory to store block n data of volume V. In step 130, primary node copies the data in memory allocated to block n of replica R to the new memory allocated to block n of volume V. Thereafter, primary node 32 writes data D to the new memory allocated to block n of volume V according to the IO transaction of step 122. Lastly, primary node 32 increments C_n by 1 thereby indicating that blocks n of volume V and replica R are no longer synchronized. FIG. 6 illustrates the redirection map 76 shown in FIG. 5 after implementing the process shown in FIG. 10 with respect to block 2 of volume V.

[0035] The process described with reference to FIGS. 9 and 10 presume that replica R is maintained using asynchronous replication. The same process works for synchronous replication. In synchronous replication C_n will always be 1 and hence step 114 and 124 always result in execution of 116 and 126 respectively. Replica R may be maintained as a real time or near real time copy of volume V using periodic replication. U.S. patent application Ser. No. 10/436,354 entitled “Method and System of Providing Periodic Replication,” filed on May 12, 2003, incorporated herein by reference in its entirety, describes relevant aspects of periodic replication. This application describes use of current data volume maps and next data volume maps to track differences between blocks of data volume V and its replica R. The current and next data volume maps can be used to create a redirection map for tracking differences between respective blocks in volume V and replica R. More particularly, a replication map can be formed as a multi-entry map wherein each entry corresponds to a logical OR of respective bits within the current and next data volume map. Each bit in this redirection map determines whether respective blocks within volume V and replica R are synchronized. This redirection map for periodic replication can be used instead of replication map 76 in processes described in FIGS. 7 and 8 to correct or overcome data corruption or data inaccessibility in volume V.

[0036] Although the present invention has been described in connection with several embodiments, the invention is not intended to be limited to the specific forms set forth herein. On the contrary, it is intended to cover such alternatives, modifications, and equivalents as can be reasonably included within the scope of the invention as defined by the appended claims.