Method and apparatus for secure data mirroring a storage system

[0001] The present invention is generally related to storage systems and in particular to secured data mirroring capability in storage systems.

[0002] Generally speaking, it is very common to provide data mirroring functionality for a storage system that is implemented using microcode program in the storage system. FIG. 3 shows a conceptual view of the data flow for a full data mirroring method, which is very typical in current storage products.

[0003] In the storage system shown in FIG. 3, suppose there are two volumes 81a and 82a. The volume 81a is called a Primary Volume or P-Vol, where original or production data is stored. The volume 82a is called a Secondary Volume, S-Vol or mirrored volume, where mirrored data from the P-Vol 81a is stored. These two volumes are collectively referred to as a pair, and are in an operational state commonly referred to as the PAIR state. In this state, data that is written to the P-Vol will be mirrored to the S-Vol.

[0004] Hosts 1, 2 and 3 are shown connected to the storage system. Host 1 writes data to the storage system. Of course, it is understood that Host 1 also reads data from the storage system, although that is not shown in the figure. A data block is the typical unit of data for a read or a write operation. Any host, including Hosts 1, 2, and 3 can access the S-Vol 82a. The host can read data from the S-Vol. Data write operations can be performed when the pair is in an operational state commonly referred to as SPLIT. In the SPLIT state, the data mirroring operation is not performed (i.e., suspended).

[0005] In FIG. 3, a data block 101a is the object of a Write operation to the P-Vol 81a. In the P-Vol 81a, the block is stored in a location 103a. In the PAIR state, a copy 104a of the data block is stored on the S-Vol 82a. The copy is stored at the same address 113a in the S-Vol 82a as the address 103a in the P-Vol 81a. Host 2 or 3 can then access the data block 101a respectively as blocks 11a, 112a by accessing location 113a in the S-Vol 82a.

[0006] As explained, the mirrored volumes are read/write compatible by other hosts and can be used for any purpose. Typical examples in which data mirroring can be used include application testing and development, data mining/data warehousing, rapid data restores, and non-disruptive backup or maintenance procedures.

[0007] Another mirroring method called Snap Shot or Copy on Write stores only the changed data blocks in a storage pool. These methods result in a volume savings as compared to the full mirroring method explained in FIG. 3 since only differences are maintained.

[0008] The proliferation of electronic information has created demands for data security and data privacy. Users want secure data management in their storage systems. A motivating factor is the trend toward a shared storage model. It is increasingly common for users to share the storage system among different organizations because the shared storage model is very efficient in terms of storage purchasing costs and storage management costs. Storage service providers especially benefit by using the shared storage model, and thus the shared storage model is a common implementation.

[0009] However, the shared storage model also increases the risk of data exposure to other organizations, and more importantly to potentially unauthorized users. The risk includes intentionally stealing data as well as accidental data exposure. Users understandably want to protect their data from such risks.

[0010] Generally speaking, the risk of data exposure is greater for the mirrored data that is stored in the secondary volume than for original data stored in the primary volume. First, other hosts can access the secondary volume after it is created, as explained the above. Access to the primary volume is not a easily achieved because, for example, the host is already attached to the primary volume. For example, if the Host 1 and the Host 3 belong to different organizations, then the Host 3 should not be able to access the mirrored volume 82a. However, enforcing such restrictions is not always convenient to do.

[0011] Second, there is a tendency that users care more about the original data that is stored in the primary volume than for mirrored data that is stored in the secondary volume. As a result, people tend to give less attention to the mirrored data. This tendency increases the risk of data exposure more in case of the mirrored data than for the original data.

[0012] There is a need to provide secure data mirroring capability in a storage system in order to protect the mirrored data from the risk of data exposure.

[0013] The present invention includes a storage system capable of secure data mirroring. The storage system obtains a copy key used for encrypting blocks of data on a primary volume. The encrypted blocks of data are then mirrored to a secondary volume. The copy keys are associated with host systems. Data access from the secondary volume requires the copy key that was used to encrypt the data being accessed.

[0042] FIG. 1 shows a simplified block diagram of a storage system 20 according to an embodiment of the present invention.

[0043] Several host systems 1, 2, 3 are shown in data communication with the storage system 20 over suitable storage networks 11, 12, 13. Though not shown in FIG. 1, the hosts 1, 2, 3 include suitable interface hardware (e.g., host bus adapters) for connection to the storage networks 11, 12, 13. Examples of storage networks include SANs (storage area networks) based on Fibre Channel and SCSI (small computer system interface) standards, NAS (network attached storage) based on IP (internet protocol), NFS (network file system), and CIFS (common internet file system) protocols, virtualization systems, and the like. Generally, the storage system 20 provides the physical storage which constitute the storage networks 11, 12, 13.

[0044] The storage system 20 generally includes channel controllers 21, 22, 23, a memory component 30 (one memory is shown in the figure, but the number is not limited to one), one or more disk controllers 40 (although one is shown to simplify the diagram), and storage devices or volumes 81, 82, 83, 84 (four volumes are shown in the figure, but the number is not limited to four). These components are connected to each other through internal networks 91 and 92. In accordance with a specific implementation, the storage system provides RAID (Redundant Array of Independent Devices) capability in order to improve reliability of the stored data and to provide other benefits offered by RAID architectures.

[0045] The channel controllers 21, 22 and 23 serve as interfaces for connection with the hosts 1, 2, 3. The physical storage devices are organized by the disk controller into logical volumes. In this embodiment, the volumes 81-84 are configured to be logical volumes, which are protected by RAID. For example, FIG. 1 shows that host 1 is associated with the logical volume 81 identified as a primary volume (P-Vol). Another logical volume 82 is identified as a secure secondary volume (SS-Vol). These volumes will be discussed in more detail below.

[0046] The memory component 30 may include a cache memory component and a random access memory (RAM) component. The memory 30 stores, among other things, information for supporting data mirroring operations according to the present invention. As will explained in more detail below, a bitmap region 31 contains bitmap information for data mirroring. The memory 30 includes a key region 32 for storing keys used for data mirroring.

[0047] The disk controller 40 provides data control functionality including mirroring 41, access 42, stripping, parity processing and so on. These functions are usually implemented in microcode programs which execute on the disk controller 40. The disk controller 40 provides functionality in accordance with the present invention, including key management 50, encryption 60, and decryption 70. These functions can be provided in microcode programs or with custom logic or by a combination of hardware and software. The functionality will be discussed in further detail below.

[0048] The internal networks 91, 92 provide power, control signals, and data routing capability to support the storage system 20. The internal network 91 provides data and control communications among the channel controllers 21, 22, 23, the memory component 30, and the disk controller(s) 40. The internal network 92 provides data and control communications among disk controllers and volumes. Examples of internal networks include PCI, Fibre Channel, and so on. FIG. 1 shows a bus architecture, but it can be appreciated that other architectures such switch architectures, matrix architectures, and so on can be used to implement the internal networks 91, 92.

[0049] The hosts 1, 2, 3 include mirror management functions 4, 5 and 6, respectively, that control data mirroring operations in the storage system 20. If a host does not require data mirroring, then that host will not be configured to include mirror management capability. An implementation of the mirror management function is by way of a scripting language such as PERL, or UNIX shell scripts. In an alternate embodiment (not shown), a dedicated host referred to as a service processor can be provided to control data mirroring operations.

[0050] A typical control flow for performing conventional data mirroring includes: a host (e.g., host 1) issuing one or more commands to the storage system 20 to initiate data mirroring in the storage system. The disk controller 40 executes appropriate data mirroring microcode programs in response to the commands. As shown in FIG. 1, the P-Vol 81 is connected to (associated with) the host 1. The volume 82 is designated as a secondary volume and is the target volume for the data mirroring operation. Bitmap tables stored in the bitmap regions 31 in the memory 30 are used to conduct the data mirroring operation, keeping track of which data blocks have been updated and which data blocks need to be mirrored. When data mirroring is in effect, data written to the P-Vol is “mirrored” (93) to the secondary volume. More particularly, the data is stored in the same location (i.e., same block address) on the secondary volume as it is stored in the P-Vol.

[0051] Though FIG. 1 shows a single secondary volume, it can be appreciated that data mirroring can be targeted to more than one secondary volume. It is further understood that cascaded mirroring can be performed, where data is mirrored to a secondary volume SV-1, data in the secondary volume SV-1 is mirrored to another secondary volume SV-2, and so on in cascading fashion. Still other data mirroring configurations are known. It will be clear that the present invention is readily adapted to all such data mirroring configurations.

[0052] As will be explained in more detail below, the foregoing data mirroring steps are enhanced with the following aspects of the present invention. The hosts 1, 2, 3 contain encryption keys (respectively, Key 1, Key 1, Key 2). These encryption keys are communicated to the storage system 20 in connection with a data mirroring operation. The secure data mirroring functionality 41 in the disk controller 40 contains key management capability 50, which manages the keys stored in the key region 32 in the memory 30 in a secure way as long as data mirroring is in effect. When data mirroring is terminated, the keys that are stored in the key region 32 are deleted from the memory 30.

[0053] The secure data mirroring functionality 41 also includes a block encryption capability 60. This function is performed when data mirroring is in effect. The result is that the secondary volume contains encrypted data. This is emphasized in FIG. 1 by referring to the secondary volume as a secure secondary volume, a secured secondary volume, a secure volume, and so on (SS-Vol).

[0054] A further aspect of the present invention is accessing the SS-Vol 82. The secure data access functionality (e.g., microcode program) 42 contains key management capability 50, which manages the keys in the key region 32 in the memory 30 in a secure way for the duration that the SS-Vol 82 is being accessed. After the volume access is terminated, the keys in the key region 32 are deleted from the memory 30. The secure data access function 42 also includes block decryption capability 70 using the keys. The block decryption capability 70 is executed for the duration that a host is accessing the SS-Vol 82.

[0055] As an example, suppose the mirror management function in host 1 is invoked to initiate data mirroring using Key 1. In response, the data on the P-Vol 81 is encrypted and mirrored to the SS-Vol 82. Suppose the host 2 belongs to the same organization as the host 1 and contains the same key, namely, Key 1. Host 2 will be able to decrypt the encrypted data on SS-Vol 82 to produce clear data and thus gain access to the data. Suppose that host 3 belongs to an organization different from that of host 1, and thus contains a key Key 2 that is different from Key 1. The host 3 will receive unintelligible data when it reads from the SS-Vol. 82 because an incorrect key was applied to the encrypted data during the decryption process.

[0056] It will be seen from the following discussion that keys can be shared among hosts in any of a number of ways. For example, each key can be stored in a smart card format. The smart card can be inserted into a suitable reader installed in a computer such as a desktop or a laptop and the stored key can be read. As another example, keys can be exchanged through over a communication network in a secured manner. The access can be controlled by using a password or by an ordinal key exchange protocol like using digital signature.

[0057] Public key encryption (asymmetric encryption) techniques can be used. The public—key/private-key pair generated in accordance with a public key encryption method can be used for encryption and decryption. One of the keys (e.g., the private key) from the key pair can be stored in the host, while the other key (e.g., the public key) from the key pair is stored in the storage system. This aspect of the invention will be discussed in further detail below.

[0058] FIG. 1 shows an embodiment of the present invention in which the physical storage which constitutes the secure secondary volume (SS-Vol) is co-located with the physical storage which constitutes the primary volume (P-Vol). It can be appreciated, however, that the location of the SS-Vol is not relevant to present invention, and that the physical storage which constitutes the SS-Vol can be any suitable physical location in a storage system separate from the storage system of the P-Vol.

[0059] FIG. 1A shows an embodiment of the present invention in which the storage system 20 includes a storage virtualization component 80. The storage virtualization component 80 is a set of hardware and software (firmware) configured to present a virtual volume 82′ that is comprised of one or more external volumes 82″ in a remote storage system (also referred to as external storage system). The virtualization component 80 maps external volumes and logical volume addresses. Write operations made to a virtual volume are translated by the virtualization component 80 to one or more subsequent write operations to the constituent external volumes. An example is shown in U.S. Pat. No. 6,529,976, entitled “Heterogeneous Computer System, Heterogeneous Input Output System and Data Backup Method for the Systems” and is fully incorporated herein by reference for all purposes. The external volumes 82″, in turn, can be provided within the storage system 20, or at a location separate (or is otherwise remote) from the storage system. Data mirroring in accordance with the present invention can proceed as will be described below, with the virtual volume 82′ as the target of the data mirroring operation. The virtualization system 80 transparently converts data access with the virtual volume 82′ to corresponding operations on the external volumes 82″.

[0060] FIG. 1B shows yet another embodiment of the present invention in which the primary volume 81 and the secure secondary volume 82 are located in different storage systems, 20-A and 20-B respectively. Each host 1, 2, 3 can be configured with a communication channel to each storage system; e.g., the figure shows host 1 having a communication channel 11a for communication with storage system 20-A and a communication channel 11b for communication with storage system 20-B. Typically each host is configured with high availability (HA) software which provides safe failover software for each storage system. The storage systems 20-A and 20-B can be in different locations within a computer room, or they can be in different buildings, or in different parts of the world. The disk controller 40-A in the storage system 20-A is configured with the Secure Data Mirror function 41. The disk controller 40-B in the storage system 20-B is configured with the Secure Data Access function 42. A suitable communication channel 95 (e.g., remote copy technology) is provided between the storage systems 20-A and 20-B to allow data mirroring in accordance with the present invention (93) to occur between the P-Vol and the SS-Vol. How and where the storage system 20-B will be deployed will depend on a consideration of factors that are independent of the present invention.

[0061] Secure data mirroring and secure data access in the configuration of FIG. 1B are achieved in the same fashion as in the configuration of FIG. 1. For example, suppose in FIG. 1B the mirror management function in host 1 is invoked to initiate data mirroring with Key 1. In response, the data on the P-Vol 81 is encrypted via the Secure Data Mirror function 41 and mirrored over the communication channel 95 to the SS-Vol 82. Suppose that a host 2 belongs to the same organization as the host 1 and contains the same key, namely, Key 1. Host 2 can access the mirrored data stored in the SS-Vol 82 by accessing the remote storage system 20-B using the Key 1 via the Secure Data Access function 42 in the storage system 20-B. Host 2 will be able to decrypt the encrypted data on SS-Vol 82 to produce clear data and thus gain access to the data. Suppose that host 3 belongs to an organization different from that of host 1, and thus contains a key Key 2 that is different from Key 1. Host 3 will receive unintelligible data when it reads from the SS-Vol. 82 because an incorrect key was applied to the encrypted data during the decryption process.

[0062] The discussion which follows is made with respect to the configuration of FIG. 1. A person of ordinary skill, however, will be able to adapt the specific embodiment of FIG. 1 to the configurations shown in FIGS. 1A and 1B, and to combinations of the configurations of FIGS. 1A and 1B. For example, the storage system 20b shown in FIG. 1B can itself be a virtualization system. Another example is shown in FIG. 1C, wherein the host 1 is configured to access only the storage system 20-A, and hosts 2 and 3 have access only to the storage system 20-B.

[0063] FIG. 2 represents a generalized data flow during data mirroring according to the present invention. A host 1 writes a block of data 101 to a primary volume (e.g., P-Vol 81). The block of data is stored at location 103 in the P-Vol 81. When the block of data 101 is to be mirrored, a copy 104 of the block of data is produced. The block encryption function 60 applies a suitable encryption key (the copy key) to the copy 104 to produce an encrypted block 114. The copy key is derived from Key 1 that was provided by the host 1. The encrypted block 114 is communicated to a secure secondary volume (e.g., SS-Vol 82), where it is stored in a location 113 that is the same as location 103 in the primary volume 81. The foregoing sequence is referred to as a “Secure Block Copy” operation.

[0064] FIG. 2 also shows a generalized data flow for accessing encrypted data from the secure secondary volume 82. The host 2 will pass Key 1 to the storage system 20 in connection with access operations to the secure secondary volume 82. If the host 2 makes a read request to access data stored at the location 113 in the secure secondary volume 82, the data stored at that location will be read out by the disk controller (40, FIG. 1) and processed by the block decryption function 70. The block of data read from the location 113 will be decrypted using the Key 1 associated with the host 2. Since this is the same key as the encryption is used for the decryption, the block of data 111 (read-out data) that is produced is clear data. Thus, the read-out data that is returned to the host 2 is clear data. This sequence is identified in FIG. 2 as a “Secure Data Access” operation.

[0065] Likewise, the host 3 will pass Key 2 to the storage system 20 in connection with access operations to the secure secondary volume 82. If the host 3 makes a read request to access data stored at the same location 113 in the secure secondary volume 82, the data stored at that location will be read out by the disk controller (40, FIG. 1) and processed by the block decryption function 70. The block of data read from the location 113 will be decrypted using the Key 2 associated with the host 3. However, since the key (Key 2) is different from the key (Key 1) that was used to perform the encryption, the result that is produced is a non-decrypted and very likely an unintelligible block of data 112. The decryption process using Key 2 produces unintelligible read-out data (data block 112) that is returned to the host 2. The first read-out data 111 that was returned to the host 2 is different from the second read-out data 112. In this way, host 3 is effectively denied access to the data stored in the secure secondary volume 82, even though host 3 was able to read the encrypted block of data stored at the location 113.

[0066] From the foregoing, the following is understood. Encrypted data is data that is produced from an encryption operation performed on “original data.” Decrypted data is data that results from a decryption operation performed on the encrypted data. The decrypted data can be further referred to as “clear data” if the decrypted data is the same as the original data. On the other hand, the decrypted data can be referred to as “unintelligible data” if the decrypted data is not the same as the original data.

[0067] FIG. 4 is a state transition diagram showing the different operational states of data mirroring according to the present invention. There are six states: SIMPLEX 200, COPY (Pending Duplex) 210, PAIR 220, COPY (Pending Split) 230, SPLIT 240, and COPY (Resync) 250. SIMPLEX state 200, PAIR state 220, and SPLIT state 240 are steady states, while COPY (Pending Duplex) state 210, COPY (Pending Split) 230, and COPY (Resync) state 250 are transitional states. The triggering events (shown in FIG. 4 by dashed lines) for state transitions are data mirroring operations and data mirroring commands which, in accordance with the illustrative embodiment of the present invention of FIG. 1, issue from the hosts 1, 2, 3. The commands include: Secure-Pair Create 260, Secure-Pair Split 270, Secure-Pair Resync 280 and Secure-Pair Delete 290. The arrows show the only valid state transitions in accordance with this particular embodiment of the present invention.

[0068] The SIMPLEX state 200 is an operational state where a pair of volumes 201 and 202 are not linked together. In the SIMPLEX state, each volume can be accessed independently by a host. Secure data mirroring in accordance with this embodiment of the present invention must start from the SIMPLEX state.

[0069] Secure-Pair Create 260 is a command that initiates secure data mirroring operation in the storage system 20. The volumes 201 and 202 are now referred to as a primary volume 201 (P-Vol) and a secure secondary volume 202 (SS-Vol). This operation is discussed in further detail below.

[0070] The COPY (Pending Duplex) state 210 is a transitioning state where a Secure Initial Copy operation 213 is being executed. This state persists until P-Vol 211 and SS-Vol 212 become a duplex (paired), at which time data on the P-Vol is mirrored to the SS-Vol. The Secure Initial Copy operation 213 is discussed in further detail below.

[0071] The PAIR state 220 is the state to which the COPY (Pending Duplex) state 210 transitions. The PAIR state 220 is a steady operational state where P-Vol 221 and SS-Vol 222 are duplex. In this state, updates that are made to P-Vol 221 are mirrored to SS-Vol 222 by a Secure Update Copy 223 operation. This operation is discussed in further detail below.

[0072] Secure Pair Split 270 is a command that suspends the data mirroring operation in the storage system. This aspect of the present invention is discussed in further detail below.

[0073] The COPY (Pending Split) state 230 is a transitional operational state where the Secure Update Copy operation 233 is being executed. This state continues until SS-Vol 232 becomes a consistent image of P-Vol 231 at the time when the Secure Pair Split 270 was executed. More specifically, this state persists until the data on the P-Vol 231 is mirrored to the SS-Vol 232. The Secure Update Copy operation 233 is discussed in further detail below.

[0074] The SPLIT state 240 is the state to which the COPY (Pending Split) state 230 transitions. The SPLIT state 240 is the steady state where P-Vol 241 and SS-Vol 242 are split. In the SPLIT state, updates made to the P-Vol 241 are not mirrored to SS-Vol 242. However, such updates are marked in the memory 30 as so-called “differential data” between P-Vol 241 and SS-Vol 242 to be mirrored during a resync operation. This aspect of the present invention is discussed further below.

[0075] Secure-Pair Resync 280 is a command that restarts secure data mirroring from P-Vol 211 to SS-Vol 212. In this case, differential data that was accumulated during the SPLIT state is used to perform a secure data mirror operation on data updates made in the P-Vol 211 that occurred during the SPLIT state (the differential data). In this particular embodiment of the present invention, a threshold is provided. If the amount of differential data exceeds the threshold, then the Secure Initial Copy operation 213 is executed. The reason is that enough differential data may accumulate during the SPLIT state that it would be more efficient to perform a dedicated data mirror operation to mirror the entire P-Vol (including the differential data), than to mirror only the differential data as a background operation in a live system. This aspect of the invention is discussed in further detail below.

[0076] The COPY (Resync) state 250 is a transitional state where the Secure Update Copy operation 253 is performed on any differential data on the P-Vol 251. This state continues until the P-Vol 251 and the SS-Vol 252 attain a duplex (PAIRED) state, at which time a transition occurs to the PAIR state 220.

[0077] Secure-Pair Delete 290 is a command that causes the storage system 20 to delete the secure data mirroring link between the P-Vol 211 and the SS-Vol 212. After the command is executed, the state of the volumes becomes SIMPLEX, where two volumes have no relationship to each other. Unlike the Secure-Pair Split command 270, differential data is not tracked when a pair is split by the Secure-pair Delete command.

[0078] In accordance to this particular embodiment of the present invention, any host can access the SS-Vol in any of the foregoing operational states. However, it will be clear that only those hosts that have an appropriate decryption key can obtain clear data from the SS-Vol. Other hosts will be able to read data from the SS-Vol, but will received garbled and otherwise unintelligible information instead of clear text. It is understood of course, that the primary host can read or write data to the P-Vol in any of the foregoing states.

[0079] FIG. 5 shows a process flow of the secure data mirroring when the Secure-Pair Create command 260 is executed, according to this particular embodiment of the present invention. A host, referred to as the primary host, will issue a Secure-Pair Create command when data mirroring of its associated volume is desired. The Secure-Pair Create command specifies two volumes, the volume associated with the primary host (the primary volume) and a volume that is the target of the data mirroring operation (the secondary volume).

[0080] A check is performed in a step 301 to confirm that the two volumes each is in the SIMPLEX state 200. If either or both of the two volumes are not in the SIMPLEX state, then appropriate error processing is performed because a duplex state cannot be attained for the data mirroring operation. If the two volumes are in the SIMPLEX state, then the volumes are set to the COPY (Pending Duplex) state 210 in a step 302.

[0081] In a step 303, a key is received from primary host and is stored in the key region 32 of the memory 30. This aspect of the present invention is discussed in further detail below. In a step 304, the process establishes a secure data mirroring pair between the two volumes. In FIG. 4, the volume 201 is defined as the P-Vol 211 and the volume 202 is defined as the SS-Vol 212. Typically, a volume is associated with volume attribute information. This information includes an indication that the volume is being used as a primary volume, or a secondary volume. When a secure data mirroring pair is established between the two volumes that are specified in connection with the Secure-Pair Create command, the volume attribute of one of the volumes is set to indicate that it is a primary volume (P-Vol) and the volume attribute of the other volume is set to indicate that it is a secure secondary volume (SS-Vol).

[0082] Since the data mirroring operation occurs in real time, it is important to ensure consistency between the primary and the secondary volumes. A conventional approach is the use of bitmaps for tracking the various states of each data block in the primary volume. Thus, in a step 305, the process allocates two bitmap tables in the bitmap region 31 of the memory 30: an Update Bitmap (FIG. 7) and a Copy Bitmap (FIG. 8).

[0083] The Update Bitmap indicates for each block in the P-Vol whether that block was updated by the primary host since the last data mirroring operation. FIG. 7 is an example of the Update Bitmap. The figures shows in tabular form an array of 8 columns by N rows to accommodate 8×N blocks, each block having a corresponding entry (bit) in the Update Bitmap. A “0” can be used to indicate that the corresponding block has not been updated since it was last mirrored. A “1” would then indicate that the corresponding block was updated since the last time it was mirrored. Basically then, each time a block is written to, its corresponding bit in the Update Bitmap is set to “1”.

[0084] The Copy Bitmap indicates for each block in the P-Vol whether it has been mirrored to the SS-Vol. FIG. 8 shows an example of the Copy Bitmap. Each entry (bit) in the bitmap indicates the mirrored state of its corresponding block; a “0” indicates that the block has been mirrored, and a “1” indicates the block has not yet been mirrored. Thus, when the Copy Bitmap is allocated in response to the Secure-Pair Create command, it is initialized to all “1”s. It can be appreciated that other techniques for data consistency can be implemented.

[0085] As will be explained, in accordance with this particular embodiment of the present invention, the blocks are mirrored in sequential order. Thus, the Copy Bitmap is scanned beginning from row 1, column 1 in increasing sequential order. Referring to FIG. 8, the bit 411 is next in line to be mirrored to the SS-Vol. However, if the process receives an update request on a block that has not yet been mirrored, then the process will jump to the requested block to perform a data mirror operation in accordance with the present invention on the data that is already contained in that block before servicing the update request in order to maintain data consistency. Sequential processing then resumes where it left off, in this case the block identified by bit 411. Additional examples are shown in FIGS. 7 and 8. For example, bit 402 in FIG. 7 and bit 412 in FIG. 8 indicate that the corresponding block was updated out of sequence. See also bits 403 and 413, and 404 and 414.

[0086] Returning to FIG. 5, in a step 306, at this point the process begins tracking updates that are made to the primary volume.

[0087] In a step 307, a loop constituting the Secure Initial Copy operation shown in FIG. 4 is performed for each block on the P-Vol. Processing of the loop establishes a point in time. As an initial operation, every block in the P-Vol prior to that point in time is mirrored to the SS-Vol in accordance with the present invention. The Secure Block Copy operation as shown in FIG. 2 is performed for each such block, in a step 311. When a block has been mirrored, its corresponding entry in the Copy Bitmap is set to “0” (step 312) to indicate the block has been mirrored to the SS-Vol. The process continues until all the blocks have been mirrored. When all blocks have been mirrored, the P-Vol and the SS-Vol are set to the PAIR state 220 in a step 308.

[0088] In this particular embodiment of the present invention, all mirrored blocks are processed by the Secure Block Copy operation shown in FIG. 2. However, it can be appreciated that in another embodiment of the present invention, some hosts may not be configured to use secure data mirroring. Consequently, some of the mirrored blocks may not be encrypted. In such a case, an additional bitmap that tracks un-encrypted blocks and encrypted blocks, or other similar tracking mechanism, would be needed to identify which blocks are encrypted and which blocks are not encrypted.

[0089] Refer now to FIG. 6 for an example of a key exchange process (step 303, FIG. 5) adapted according to a particular illustrative embodiment of the present invention. This particular key exchange process is currently being discussed by the ANSI/T11.3 committee, for using FC-SP (Fibre Channel Security Protocols) including a key exchange protocol within SAN as discussed, for example, on page 52 of the document available at the web site:

• [0090] http://www.snia.org/education/tutorials/spr2004/security/Security_for_Storage_Networks.pdf

[0091] Continuing with FIG. 6, in a step 321 the host accesses a Copy Key that is associated with the host. This key will be used in the Secure Block Copy operation illustrated in FIG. 2. This step can be triggered by the host itself, when data mirroring is to be initiated. Alternatively, the triggering mechanism can be another host, such as a service processor, or the storage system 20.

[0092] In a step 322, the host obtains a Storage Public Key. The Storage Public Key is one of a pair of keys (public-key/private-key) generated according to a public key encryption method. The matching key is stored in the storage system 20, and is referred to as the Storage Private Key. There are several ways to obtain a public key. A common way is to obtain a public key from a CA (certification Authority). Another way is to install the public key to the host statically beforehand.

[0093] In a step 323, the host encrypts its Copy Key by applying the Storage Public Key it obtained in step 322. The result is an Encrypted Copy Key that cannot be practicably decrypted except by applying the Storage Private Key. In a step 324, the Encrypted Copy Key is communicated to the storage system 20. Since the Copy Key is in encrypted form, any suitable form of unsecured data transmission can be used without concern for breach of security.

[0094] In a step 331, the storage system 20 receives the Encrypted Copy Key. In a step 332, the storage system obtains its Storage Private Key and uses it to decrypt the Encrypted Copy Key (step 333) to recover the Copy Key. Then, in a step 334 the storage system 20 stores the recovered Copy Key in the key region 32 of its memory 30 during the time that data mirroring is in effect. In a particular implementation of this aspect of the present invention, the world wide name (WWN) that is associated with the host HBA can be used as an index for storing the recovered Copy Keys. The WWN facilitates the lookup (search) process for finding the Copy Key associated with a host.

[0095] It can be appreciated of course that a common-key encryption technique can be used instead of the public key encryption system discussed above. In a common-key system, the host and the storage system 20 use the same key; the host uses the key for encryption, the storage system uses the key for decryption. The challenge of using a common-key system is to ensure a secured conveyance of the key to both the host and to the storage system.

[0096] The storage system 20 handles secure data mirroring for many hosts and many volumes. Consequently, many keys from many hosts will be stored in the key region 32 of the memory 30. The key management function 50 (FIG. 1) serves to match a host with the proper key from among potentially many keys for a given data mirroring operation.

[0097] Referring now to FIG. 9, the Secure Update Copy operation shown in FIG. 4 is described. The Secure Update Copy operation is periodically executed in a loop 501 as long as the volumes are in a PAIR state 220. The loop period can be any suitable length of time. Each iteration can be initiated automatically. For example, if the disk controller is implemented under UNIX, the “cron” facility in the UNIX operating system can be configured for continuous operation of the Secure Update Copy process. Each iteration of the loop 501 comprises the following steps: an initialization step 502 and an inner loop 503.

[0098] In the step 502, the process resets the Copy Bitmap to all “1”s to indicate that none of the corresponding blocks have been mirrored.

[0099] The loop 503 is executed to process each block in the P-Vol. As noted above, the blocks are processed in sequence (the sequentially processed blocks) beginning with the block corresponding to row 1, column 1 of the Copy Bitmap shown in FIG. 8. However, blocks that are updated during the processing of the loop 501 are preferentially processed. These blocks are mirrored ahead of the sequentially processed blocks.

[0100] Each iteration of the loop 503 includes checking an entry in the Update Bitmap in a step 504 to determine if the corresponding block has been updated. If not, then there is no need to perform a mirroring operation, and processing proceeds to a step 507. At step 507, the corresponding entry in the Copy Bitmap is set to “0” to indicate that data mirroring of the corresponding block has been processed (although in this case, of course, no data mirroring was actually performed).

[0101] If, in step 504, the corresponding block has been updated, then in a step 505 a Secure Block Copy operation (FIG. 2) is performed. Then, in a step 506 the entry in the Update Bitmap is set to “0” to indicate that the block has not been updated since its last mirroring operation (which was just performed). Processing proceeds to step 507 where the corresponding entry in the Copy Bitmap is set to “0” to indicate that data mirroring of the corresponding block has been performed. Execution, then proceeds with the next iteration of the loop 503 to process the next entry in the Update Bitmap.

[0102] FIG. 10 shows the processing for a Secure-Pair Split command 270 according to the particular embodiment of the present invention shown. In a step 521, the process checks the current state of the volumes specified in the Secure-Pair Split command 270. In FIG. 4, the volume 221 and the volume 222 are specified as the target of the Secure-Pair Split command 270. If the volumes are not in the PAIR state 220, then appropriate error processing is performed. If the volumes are in the PAIR state 220, then the process proceeds to a step 522 where the volumes are set to the COPY (Pending Split) state 230. The process waits in a loop 523 until the current processing of the Secure Update Copy process is complete. Upon completion of the Secure Update Copy process, the SS-Vol will consistent with the P-Vol with respect to a given point in time relating to receipt of the Secure-Pair Split command. In a step 524, the process suspends the Secure Update Copy process. The state of the volumes is set to the SPLIT state 240.

[0103] In the SPLIT state, the P-Vol 241 accepts updates from the host. However, the updates will not be mirrored to the SS-Vol 242. Instead, the updates are tracked by setting the corresponding entry (bit) in the Update Bitmap to “1”. The Copy Suspended function 243 just indicates this state. The function performs updates to the Update Bitmap.

[0104] FIG. 11 shows a process flow for the Secure-Pair Resync command 280 according to the particular embodiment of the present invention shown. In a step 541, the process checks the current state of the volumes specified in connection with the Secure-Pair Resync command 280. In FIG. 4, the volume 241 and the volume 242 are specified as the target of the Secure Pair Resync command 280. If the volumes are not in the SPLIT state 240, then appropriate error processing is performed. If the volumes are in the SPLIT state 240, then the process proceeds to a step 542 where the volumes are set to the COPY (Resync) state 250.

[0105] In a step 543, the process executes an iteration of the Secure Update Copy operation discussed in FIG. 9. In a particular implementation of this embodiment of the present invention, a threshold can be applied to the Update Bitmap to make a determination whether to perform an iteration of the Secure Update Copy operation or a Secure Initial Copy operation (loop 307, FIG. 5). If “most” of blocks in the P-Vol have been updated after entering the SPLIT state, it might be more efficient to simply data mirror all of the blocks on the P-Vol (via Secure Initial Copy), rather than perform the incremental data mirroring of the Secure Update Copy process. Thus, a predetermined number of bits that are set to “1” in the Update Bitmap can be used as a threshold, above which the Secure Initial Copy operation is performed.

[0106] In a step 544, the operational state of the volumes is set to the PAIR state 220 after the Secure Update Copy or the Secure Initial Copy has completed. Then, in a step 545, the Secure Update Copy operation is set up for continuous operation. As discussed above in FIG. 9, this can be implemented, for example on UNIX, using the “cron” facility.

[0107] FIG. 12 shows a process flow for the Secure-Pair Delete command 290 according to the particular embodiment of the present invention shown. In a step 561, the process checks the current state of the volumes specified in connection with the Secure-Pair Delete command. In FIG. 4, the volume 241 and the volume 242 are specified as the target of the Secure-Pair Delete command 290. If the volumes are not in the SPLIT state 240, then appropriate error processing is performed. Alternatively, commands can be issued to move the operational state of the specified volumes to the SPLIT state instead of generating an error indication.

[0108] Continuing with FIG. 12, if the volumes are in the SPLIT state 240, then the process proceeds to a step 562. The process can cease updating bit entries in the Update Bitmap since secure data mirroring is going to be terminated by this command. Likewise, in a step 563, the Update Bitmap and the Copy Bitmap are deallocated and returned to the memory 30.

[0109] In a step 564, the process resets the volume pair. In the memory 30, the volume 241 was registered as a P-Vol and the volume 242 was registered as an SS-Vol. In response to the Secure-Pair Delete command, the volume P-Vol is no longer identified as a primary volume. As discussed above in FIG. 5, the volume attribute information associated with the P-Vol can be set to indicate that it is no longer a primary volume. In accordance with the present invention, however, the SS-Vol maintains its identity as a secure secondary volume. By maintaining the status of the volume 242 as a secure secondary volume even after secure data mirroring has ceased, the system can still recognize that the volume contains encrypted data. In this way, as will be explained below, appropriate processing can be applied to access the encrypted data that is stored on the SS-Vol.

[0110] In a step 565, the key that was used to perform the Secure Block Copy operation (FIG. 2) is deleted from the key region 32 in the memory 30. Then, in a step 566 the process sets the state of the volumes to the SIMPLEX state 200.

[0111] Referring to FIG. 13, the discussion will now turn to accessing a secure secondary volume in accordance with the foregoing disclosed embodiment of the present invention. When a logical volume is attached to a logical channel and is ready for data access by a host, the following processing is performed in addition to the processing conventionally performed during an “attach” operation:

[0112] In a step 601, a check of the volume attribute information is made to determined if the volume is a secure secondary volume (SS-Vol). If the volume attribute information indicates that the volume is not an SS-Vol, then the process proceeds to a step 604 to perform conventional data access operations.

[0113] If the volume is an SS-Vol, then in a step 602 a Copy Key is obtained from the host and is stored in the key region 32 in the memory 30. An illustrative embodiment of this aspect of the present invention is discussed above in connection with FIG. 6. The storage system 20 handles secure data access for many hosts and many volumes. Consequently, many keys from many hosts will be stored in the key region 32. The key management function 50 serves to match a host with the proper key for a given secured data access.

[0114] Then, in a step 603, secure data access processing is performed for subsequent data access requests from the host. Data accesses proceed according to the Secure Data Access process shown in FIG. 2. The Copy Key that is obtained from the host is used to decrypt the encrypted data blocks that are read out from the SS-Vol.

[0115] Another aspect of the present invention is a write operation to the secure secondary volume (SS-Vol). The host can write data to the SS-Vol, as long as the SS-Vol is in the of SIMPLEX state or the SPLIT state. When the host issues a write operation to the SS-Vol, the data block associated with the write operation is encrypted using the Copy Key that was obtained from the host. The resulting encrypted data block is then stored in the location on the SS-Vol that is identified in the write operation.

[0116] A Secure Reverse Resync command can be issued to the SS-Vol when it is in the SPLIT state. This will perform a data mirroring operation in the direction from the SS-Vol to the P-Vol. Moreover, since the data blocks in the SS-Vol are stored in encrypted form and the data blocks in the P-Vol are in clear-text form (i.e., not encrypted), the reverse data mirroring operation includes a step of decrypting the encrypted data block to produce a clear data block. The clear data block is then mirrored to the P-Vol.

[0117] It can be appreciated that in a variation of this embodiment of the present invention, write operations to the SS-Vol can be performed without encryption (a separate bitmap would be needed to distinguish encrypted blocks and unencrypted blocks). In such a case, conventional reverse resync processing can be performed to mirror updates made in the SS-Vol to the P-Vol; i.e., there is no step of decrypting. Data reads of the SS-Vol, of course, require decryption as discussed above.

[0118] Referring to FIG. 14, when a logical volume is detached from its logical channel, additional processing is required when the logical volume is a secure secondary volume (SS-Vol). In a step 621, a check of the volume attribute information associated with the logical volume that is being detached is made to determine if the volume is an SS-Vol. If the logical volume is an SS-Vol, then its associated Copy Key in the key region 32 of the memory 30 is deleted.

[0119] Additional mirroring options can be included as further aspects of the present invention:

[0120] 1) Secure Consistency Group

[0121] Users can define consistency groups in which several secure volume pairs move among the different operational states (FIG. 4) at the same time. As a result, several SS-Vols can contain time consistent images. Within the group, the same Copy Key might be used.

[0122] In another embodiment, a consistency group contains both conventional un-secured volume pairs and secure volume pairs per the present invention. The consistency group could include a table in which the attribute of each volume pair is managed. This would allow for data mirror operations to proceed for both kinds of volumes, simply by referring to the attributes.

[0123] 2) Secure Cascade Mirroring and Secure Multi-Target Mirroring

[0124] Cascade mirroring is a multistage mirroring method in which multiple secondary volumes are arranged to perform data mirroring in cascaded fashion where each secondary volume performs data mirroring to a succeeding secondary volume. Secure Cascade Mirroring is a multistage mirroring method in which any of the secondary volumes in the cascade can be a secure secondary volume. If a volume of a stage becomes SS-Vol, all volumes cascaded after the stage keep encrypted data. The steps described above in connection with FIG. 13 can be used to access these volumes.

[0125] Multi-Target Mirroring is a data mirroring technique in which data in a primary volume is targeted to multiple secondary volumes. A Secure Multi-Target Mirroring technique according to the present invention allows for one or more of the target volumes to be a secure secondary volume.

[0126] A variation of the foregoing techniques is a combination of cascade and multi-target data mirroring.

[0127] 3) Quick Operations

[0128] So-called “quick” operations execute mirroring function such as Pair Split, Pair Resync, and Pair Restore on the fly; i.e., the SS-Vol is immediately available because the differential data copy is performed as a background process. These operations are also available as long as the operations follow the Secure Block Copy process described in FIG. 2.

[0129] In accordance with the foregoing illustrative embodiment of the present invention, each host 1, 2, 3 obtained a Copy Key (e.g., by generating it or otherwise creating it) that was associated with that host, and communicated the Copy Key to the storage system 20. In accordance with another illustrative embodiment of the present invention, the Copy Keys can created (or otherwise obtained) by the storage system 20, and stored and maintained in the storage system. This particular embodiment of the present invention will now be described.

[0130] Referring to FIG. 15 then, a storage system 20a according to one such embodiment of the present invention is shown. The keys are created (or otherwise obtained) and stored in the storage system 20a and so they are protected by the RAID architecture. On the other hand, there is a risk that the copy keys are exposed to other hosts or users who may also be able to gain access the storage system, whether with authorization to access to other areas of the storage system or via unauthorized access such as by hackers.

[0131] In accordance with this aspect of the invention, a plurality of Copy Keys 33a are stored in a specific volume 84a in the storage system 20a. The Copy Keys 33a are stored in encrypted form. In particular embodiment shown in FIG. 15, a public key encryption technique is used. A plurality of Public Keys 34a are stored in the volume 84a and are used to encrypt the Copy Keys 33a. The corresponding Private Keys 7a are stored in the hosts 1a, 2a, 3a. Each host has its own Private Key 7a. Thus, though the Copy Keys 33a are stored in a storage system that is potentially accessible by others, the keys are nonetheless secured by the public key encryption system.

[0132] FIG. 16 shows how the encrypted Copy Keys 33a are managed by the key management function 50a. A table 34a is provided with an entry for each secure secondary volume (SS-Vol). An identification field 701 identifies the particular SS-Vol. The Copy Key 33a associated with an SS-Vol is stored in encrypted form in a key field 702. In another embodiment, the Copy Keys 33a can be stored on other media such as IC card, as a backup. Additional redundancy can be provided by maintaining the Copy Keys in IC cards in addition to storing them in a volume 84a.

[0133] When a secure data mirroring operation is performed, the host will communicate its Private Key 7a to the storage system 20a, which will be discussed in further detail below. The storage system 20a then retrieves the encrypted Copy Key 33a that is associated with the SS-Vol that is the target of the secure data mirroring operation. The Private Key received from the host is then applied to the retrieved encrypted Copy Key to produce a clear-form of the Copy Key. The decrypted Copy Key is then used in the manner discussed above in connection with the data mirroring operation of the present invention.

[0134] FIG. 17 shows the flow for producing the clear form of the Copy Key in accordance with this embodiment of the present invention. In a step 711, the host obtains its own Private Key 7a. This step can be triggered by the host itself, when data mirroring is to be initiated. Alternatively, the triggering mechanism can be another host, such as a service processor, or the storage system 20a.

[0135] In a step 712, the host obtains a Storage Public Key. The Storage Public Key is one of a pair of keys generated according a public key encryption method. The matching key is stored in the storage system 20a, and is referred to as the Storage Private Key. There are several ways to obtain a public key. A common way is to obtain a public key from a CA (certification Authority). Another way is to install the public key to the host statically beforehand. Still other ways to obtain a public key can be contemplated, but are not relevant to the present invention.

[0136] In a step 713, the host encrypts its Private Key by applying the Storage Public Key it obtained in step 712. The result is an Encrypted Private Key that cannot be practicably decrypted except by applying the Storage Private Key. In a step 714, the Encrypted Private Key is communicated to the storage system 20a. Since the host Private Key is in encrypted form, any suitable form of unsecured data transmission can be used without concern for breaches of security.

[0137] In a step 721, the storage system 20a receives the Encrypted Private Key. In a step 722, the storage system obtains its Storage Private Key and decrypts the Encrypted Private Key (step 723) to recover the Private Key. Then in a step 724, the storage system 20a decrypts the encrypted Copy Key using the host Private Key and stores the Copy Key in the key region 32 of memory 30. The key region 32 can be protected against unauthorized access to ensure the security of the Copy Key.

[0138] The embodiment shown in FIG. 15 can be adapted to the configurations shown in FIGS. 1A-1C. FIG. 15A, for example, shows a configuration similar to the configuration of FIG. 1A; except that in the embodiment of FIG. 15A the volume 84a contains the copy keys. Likewise, FIG. 15B shows a configuration similar to the configuration of FIG. 1B. Each storage site 20a, 20b in FIG. 15B, however, includes the table shown in FIG. 16.

[0139] FIG. 18 shows yet another embodiment of the present invention, where the Copy Keys 33b are created (or otherwise obtained) and stored in the storage system 20b and protected by access codes 7b that are associated with the hosts 1b, 2b, 3b. As will become clear, the access codes serve to identify the hosts. The communication link 11 is typically an unsecured channel. Consequently, it is susceptible to attack by unauthorized individuals. It is therefore important to be able to provide secure communication between the host and the storage system. It is also important to ensure that the host is properly identified before accessing its Copy Key.

[0140] An example of an access code is the WWN (World Wide Name) of the host bus adapter (HBA) in the host that is used to connect the host to the storage system 20b. Whenever the host is attached to the SS-Vol, the Copy Key associated with the host can be obtained based on the WWN of the host. One technique is called Encapsulating Security Payload (ESP) over Fibre Channel, which has become the de-facto way to secure transmissions in the Fibre Channel network. A protocol called Challenge Handshake Authentication Protocol (CHAP) for use as a fibre channel security protocol is in the process of standardization. These techniques can be used to communicate the host's WWN to the storage system 20b in a secure fashion. As mentioned above, this technique is being discussed by the ANSI/T11.3 committee.

[0141] FIG. 19 shows an example of a table 34b for storing access codes in accordance with this particular embodiment of the present invention. As the figure shows, several hosts can be managed as a domain. A domain field 731 identifies the various domains. Hosts belonging to a domain are identified in an access code field 732. This field stores the access code (e.g., WWN of the host HBA) for each host in a particular domain. A key field 733 contains the Copy Key 33b that is used to perform secure data mirroring and to access an SS-Vol as disclosed above that is associated with each domain.

[0142] FIG. 19A shows a variation of the table shown in FIG. 19 in which each host is associated with its own copy key. Thus, an access code field 732′ contains an access code associated with each host. A key field 733′ contains the copy key 33c associated with that host.

[0143] Using the WWN as the access codes serves to uniquely identify each host. Of course, other suitable forms of identification can be used. The amount of effort taken to authenticate a host using the access code technique shown in FIGS. 18 and 19 depends on the particular environment of the storage system. Thus, different forms of access codes (i.e., host identifiers) may be used depending on the level of security that is desired.

[0144] Returning to FIG. 18, the storage system 20b can receive the access code (e.g., WWN of the host's HBA) via a secure protocol. The access code can then be used to search the table 34b to obtain the copy key 33b that corresponds to the host. Secured data mirroring according to the present invention can then be performed as discussed above.

[0145] FIG. 18A shows a variation of the embodiment of FIG. 18 configured in a manner similar to the configuration shown in FIG. 1A. FIG. 18b shows another variation of the embodiment of FIG. 18, configured in a manner similar to the configuration shown in FIG. 1B.