Computer data storage system with parallelization migration plan generator

[0002] The present application contains subject matter related to a concurrently filed U.S. patent application Ser. No. 09/394,221 by Elizabeth L. Borowsky, Richard A. Golding, and Dushyanth Narayanan entitled “COMPUTER DATA STORAGE SYSTEM WITH MIGRATION PLAN GENERATOR” now U.S. Pat. No. 6,381,619. This related application is being filed contemporaneously herewith, and is hereby incorporated by reference.

[0003] The present invention relates generally to computer data storage systems and more particularly to systems for reconfiguration of data storage.

[0004] Computer data systems today are huge and complex. Examples are: data related banking records, the internet, data mining systems, libraries, and scientific applications like planning for weather forecasting. All of this data is now being stored on-line so that the data can be manipulated.

[0005] One major problem is efficient access to this stored data. An analogy would be data storage as a library, with the data as the books in the library, and the data storage devices as shelves for the books. Different data storage devices hold different blocks of data, just as in a library where different shelves hold different books. It is desired that data be arranged on storage devices, as books are arranged on shelves, for the most efficient access.

[0006] From a library perspective, it is most efficient for storage of all of the most frequently read books to be stored on one shelf. However, this is not the most efficient from a user perspective because all the users would be trying to access that one shelf. The same is true for data storage. If all of the most frequently used data is on one data storage device, it creates a bottleneck on the data storage device because it is possible to retrieve only a certain amount of data from a data storage device per unit of time. Thus, a way to configure data storage to optimize throughput of the data storage system is necessary.

[0007] However, it is difficult to determine optimum data storage system configurations. It is necessary to look at workload patterns, how data is accessed, and what is important to the user. There are thousands, if not millions of blocks of data which must be considered.

[0008] There are individuals employed in many companies who regularly reconfigure the data storage in the data storage systems, and this is currently all being done manually. At the present time, the experts try and partition the problem into easy pieces. The people who do this are highly skilled and rely primarily on past experience, but there is no formalism to the reconfiguration process. Thus, a long sought, but unattained, objective in the field of data storage is to formalize the process and to make it easy, quantifiable, and quick to optimize a data storage system.

[0009] However, even if it were possible to optimize each storage system, even automatically, the problem is that workloads change. Over time, the most frequently used data changes. There are different blocks of data in the storage system. The access patterns to these blocks of data changes over time. Further, devices may fail or be added or subtracted. Thus, the ultimate goal is a data storage system which is not only able to automatically configure itself, but to reconfigure itself ‘on-the-fly’; i.e. move stored data around based on changing access patterns.

[0010] Currently, it is necessary to take the whole data storage system down off-line, reconfigure it, and then bring it back up. This is an increasing problem because customers cannot really afford to take their systems down at all. For example, a bank needs to have its automated teller machine systems available twenty-four hours a day, seven days a week, fifty-two weeks a year.

[0011] In the past, it has been a major set of goals to automatically optimize data storage configurations, change data storage configurations during data system operation, and to minimize the time that it takes to reconfigure the data storage system.

[0012] The present invention provides a computer data storage system with a migration plan generator which includes a “Simple” migration planner which provides for making terminal moves until no further terminal moves are possible based on random, preset, or functional ordering. A terminal move is a move of data to a data storage device where it belongs in a goal configuration. The moves make up a migration plan which is further processed to parallelize moves where possible to speed up the migration plan.

[0013] The present invention further provides a computer data storage system with migration plan generator which includes a “Simple” migration planner which provides for making terminal moves until no further terminal moves are possible, then making “shunt” moves, or non-terminal moves, until further terminal moves can be made. The moves make up a migration plan which is further processed to parallelize moves where possible to speed up the migration plan.

[0014] The present invention further provides a computer data storage system with a migration plan generator which includes a “Greedy” migration planner which uses a “contention” metric. The “contention” of a data storage device is defined as the total size of the data stores that need to move onto such data storage device, divided by the amount of free space on such data storage device. The contention of the entire system is the sum of the contention over all the data storage devices. The migration plan generator develops a plan that leads to the lowest contention for the system.

[0015] The present invention further provides a computer data storage system with a migration plan generator which includes a “Cycle” migration planner which finds shunts by first trying to find cycles with small data stores in a Greedy fashion. Once the smallest data store is moved off a data storage device, the largest data store in the cycle is shunted to any data storage device that can accommodate it.

[0016] The present invention further provides a computer data storage system with a migration plan generator which includes a “Hillclimbing” migration planner which uses a global contention metric which seeks the closest local minimum for the contention metric.

[0017] The present invention further provides a computer data storage system with a migration plan generator which includes a “Simulated Annealing” migration planner which uses a probabilistic Hilldlimbing approach that will sometimes choose moves that do not improve the optimization metric.

[0018] The present invention further provides a computer data storage system with a migration plan generator which includes a “Meta” migration planner which shifts from one planner to another based on the planner's performance.

[0019] The above and additional advantages of the present invention will become apparent to those skilled in the art from a reading of the following detailed description when taken in conjunction with the accompanying drawings.

[0020] FIG. 1 is an overview illustration of the data system of the present invention and its migration from an initial to a goal configuration;

[0021] FIG. 2 is an exemplary migration plan used in the present invention;

[0022] FIG. 3 is a migration plan generator used in the data system of the present invention;

[0023] FIG. 4 is a parallelization migration planner used in the migration plan generator of FIG. 4;

[0024] FIG. 5 is an alternative migration planner which could be used in the migration plan generator of FIG. 3. and

[0025] FIG. 6 is a parallelization mechanism used in the present invention to minimize reconfiguration time.

[0026] Referring now to FIG. 1, therein is shown an initial configuration data system 10 migrating through the migration 12 to become a goal configuration data system 14.

[0027] The initial and goal configuration data systems 10 and 14 have a plurality of hosts 16 through 18 which are devices or systems such as computers, terminals, user devices, etc. which run different types of computer applications and access different blocks of data.

[0028] The plurality of hosts 16 through 18 are connected to a storage network 20, which could be a local area network, a wide area network, the Internet, or any other communication network.

[0029] The storage network 20 in turn is connected before the migration 12 to an initial configuration data storage system, or initial configuration system 22, and after migration 12 to a goal configuration data storage system, or goal configuration system 27.

[0030] Both the initial and goal configuration systems 22 and 27 have a plurality of data storage systems which are exemplified by the data storage devices designated as device 1 through 4. The devices 1 through 4 could be disks, disk drives, disk arrays, tape drives, or other data storage mechanisms. The devices 1 through 4 have various blocks of data, in data stores, assigned to them designated as store A through store E. The assignment of the stores A through E to devices 1 through 3 is called a configuration.

[0031] In the initial configuration system 22, the initial configuration has device 1 with store A and store B assigned to it, device 2 with store C and store D assigned to it, device 3 with store E assigned to it., and device 4 with no store assigned to it.

[0032] In the goal configuration system 27, the goal configuration has device 1 with store A, store D, and store E assigned to it, device 2 with store B assigned to it, and device 3 with no data block assigned to it, and the device 4 with the store C assigned to it.

[0033] The data stores are moved or migrated among the storage devices under the direction of a control 28 such as a computer, processor, controller, etc.

[0034] Referring now to FIG. 2, therein is shown the initial configuration system 22, a first to intermediate configuration system 23, a second intermediate configuration system 24, a third intermediate configuration system 25, a fourth intermediate configuration system 26, and the goal configuration system 27.

[0035] In data storage systems, each data storage device has specific capacities, or limits, on the amount of data that can be stored. For exemplary purposes only, the devices 1 through 4 in FIG. 2 are shown in parenthesis having capacities of 6, 5, 2, and 2, respectively. The capacity units could be megabytes, terabytes, gigabytes, etc., but this is not important for purposes of the example. Similarly, the various data stores, store A through store E have in parenthesis capacities of 3, 2, 2, 2, and 1, respectively. Again, the units are only important in that they match the capacity units of the devices 1 through 4.

[0036] Also shown in FIG. 2 is a plurality of data migrations or “moves” from one configuration into another. Arrows represent the “moves” of data stores among the data storage devices. A move 30 designates the migration of store E from device 3 to device 1. Move 32 designates the migration of store D from device 2 to device 3. Move 34 designates the migration of store B from device 1 to device 2, and move 36 designates the migration of store D from device 3 to device 1.

[0037] Move 38 designates a low priority migration of store C from device 2 to device 4. The priority is set by the set of constraints, and in a sequential migration plan, the move 38 would come after the previously described moves. Dashed arrows represent moves 40 and 42 which designate alternate times when the move 38 could be performed when the migration plan to make parallel moves.

[0038] The move 30 between the initial configuration system 22 and the first intermediate configuration system 23 may also be described as a “terminal” move in that a data store is assigned to a data storage device where it belongs in the goal configuration, such as the goal configuration system 27 for store E. Similarly, the move 38 would be a terminal move.

[0039] The moves 32 from the first intermediate configuration system 23 to the second intermediate configuration system 24 may be described as a “shunt” move. In a shunt move, a data store is moved from a data storage device that it is on to a data storage device that is not its goal as in interim move. Thus, in move 32, store D is first moved to device 3 from device 2. This permits store B through move 34 to make a terminal move to device 2. This in turn permits a terminal move 36 of store D to its goal configuration on device 1.

[0040] In FIG. 2, the moves 30, 32, 34, 36, and 38 taken collectively in sequence are called the total order “migration plan”.

[0041] Referring now to FIG. 3, therein is shown a parallelization migration plan generator 100. The parallelization migration plan generator 100 is connected to receive initial configuration data 110, goal configuration data 112, and a set of constraints 114 from the control 28.

[0042] The initial and goal configuration data 110 and 112 contain information on the initial and goal configuration systems 22 and 27, respectively. Such information would include the set of data storage devices in each, the set of data stores in each, and the assignments or “binding” of the data stores to the data storage devices. The set of constraints 114 contains the capacities of the data storage devices, the capacities of the data stores, the bandwidth, movement rate, and other limitations on the moves.

[0043] The set of constraints 114 prevents certain moves. For example, device 3 has a capacity of 2, so two data stores having a capacity of 2 each cannot be placed on it. If the data stores are to be accessed during the migration, there must be enough bandwidth for a host to access the data store on the data storage device where it resides. If the data storage device is already bottlenecked on bandwidth and the data store is moved to a very heavily used data storage device, it is desirable to move an existing data store off the destination data storage device before moving the new data store onto it so that the data store can be accessed once it gets to the data storage device.

[0044] Another constraint is movement rate. There may be timing restrictions. For example, the plan might have to be executed in one hour. With a very slow data storage device, like a tape drive, and a large data store that needs to be moved, it might not be feasible to move the data store onto and off the tape drive because that may take more time than the hour set for completing the plan.

[0045] In the parallelization migration plan generator 100, the initial configuration data 110, the goal configuration data 112, and the set of constraints 114 are provided to a migration planner 116. The migration planner 116 either fails to develop a total order migration plan and exits to “fail” block 118 or produces a viable total order migration plan 146 for parallelization in a parallelization mechanism 119. The parallelization mechanism 119 causes moves that have dependencies to be put into a migration plan 120 in parallel with moves that have no dependencies for implementation by the control 28.

[0046] Referring now to FIG. 4, therein is shown one embodiment of the migration planner 116 of FIG. 3 which is designated as migration planner 116A. The migration planner 116A is a program which begins at a “start” 130 and goes to “build plan” block 132. The “build plan” block 132 receives the initial and goal configuration data 110 and 112. The output of the “build plan” block 132 is provided to an “out of time?” decision block 134 which outputs to “fail” block 118 if the program takes longer than a predetermined time. If there is remaining time, the “out of time?” decision block 134 is output to the “terminal move?” decision block 136. If a terminal move is possible, the program proceeds to the “constraints met?” decision block 138. The “constraints met?” decision block 138 receives the set of constraints 114 to determine whether or not the constraints have been met. If they have not been met, the program returns to the “build plan” block 132. If the constraints have been met, the program proceeds to the “select terminal move” block 140.

[0047] Once a terminal move is selected, it is added to the plan in an “add move to plan” block 142. The configuration after the added move is compared with the goal configuration data 112 in a “goal configuration met?” decision block 144. If the goal configuration is not met, the program returns to the “build plan” block 132, and if it is, it exits as an initial plan 146.

[0048] Referring back to the “terminal move?” decision block 136, if no terminal move is possible, the program moves to a “select shunt” block 146 where a shunt move is selected and the program proceeds to the “add move to plan” block 142. From the “add move to plan” block 142 the program proceeds as previously described.

[0049] Referring now to FIG. 5, therein is shown a migration planner 116B which is an alternative embodiment of the migration planner 116 in FIG. 3.

[0050] The migration planner 116B begins at a “start” block 150 and proceeds to a “select plan” block 152. The “select plan” block 152 receives the initial configuration data 110, the goal configuration data 112, and the set of constraints 114. The program proceeds from the “select plan” block 152 to an “out of time?” block 154 to determine if the plan has run out of time and if so, to proceed to the “fail” block 118. If there is time remaining, the program proceeds to a selected plan comprising: a “Random/Present/Function Sample” plan 156; a “Greedy Simple” plan 158; a “Greedy” plan 160; a “Cycle” plan 162; a “Hillclimbing” plan 164; a “Simulated Annealing” plan 166; an “Other” plan 168; or a “Meta” plan 170. After a program has processed the possible moves, the then current configuration is compared with the goal configuration in the “goal configuration met?” decision block 172. If the goal configuration has not been met, the program proceeds back to the “select plan” block 152 and, if it has it, exits as the total order migration plan 148.

[0051] Referring now to FIG. 6, therein is shown the parallelization mechanism 119 which has the total order migration plan 146 as its input. The total order migration plan 146 is examined to identify “existence” dependencies in an “identify existence dependencies” block 182. The “existence” dependencies are those where one move is dependent on another such as where a move to remove a data store from a data storage device cannot be performed before a move to place the data store on such data storage device has occurred.

[0052] The total order migration plan 146 is further subject to the identification of “utilization” dependencies in an “identify utilization dependencies” block 184. A “utilization” dependency is where one move is dependent on another move such as where the data storage device for one move is the destination for a move that comes after it.

[0053] After the dependencies have been determined, in the “identify existence dependencies” block 182 or the “identify utilization dependencies” block 184, the “independent” moves are parallelized with the moves having dependencies in a “parallelized independent moves” block 186. The “independent” moves are those which do not have any dependencies and which operate between data storage devices which are not involved in the moves where dependencies exist. These moves can be thus performed in parallel to reduce the overall time for the combination of moves. After parallelization in the “parallelized independent moves” block 186, this partial order migration plan is output as the migration plan 120.

[0054] It should be noted that the order of identifying existence and utilization dependencies can be interchanged, or even be performed in parallel. In operation, the initial configuration data system 10 has the hosts 16 through 18 communicating over the storage network 20 to the initial configuration system 22. Then the initial configuration system 22 has devices 1 through 4 with the store A through store E bound as previously described. Over time, the initial binding of the data stores to the data storage devices may become less optimal due to such factors as:

[0055] 1. the nature of the access patterns to the data stores changing;

[0056] 2. the service requirements of the access changing;

[0057] 3. the number of data stores changing;

[0058] 4. new data storage devices being added or subtracted; and

[0059] 5. data storage devices failing or approaching failure.

[0060] In such cases, the data stores need to be redistributed among the data storage devices to match the new needs. Based on the above, a goal configuration such as the goal configuration system 27 can be determined by the control 28. The problem then becomes one of determining the fastest migration plan for the migration 12 to the goal configuration system 27. The plan must meet the criteria that:

[0061] 1. intermediate states do not violate capacity constraints (i.e., whenever a data store is moved, there has to be enough space on the target data storage device); and

[0062] 2. the load placed by the move on the system should be minimized (in terms of data stores moved, time taken (parallel or sequential), bandwidth used, or similar metric).

[0063] The exemplary migration plan of FIG. 2 starts by determining the terminal moves possible by comparing the initial configuration system 22 with the goal configuration system 27. In this comparison, it is that store B which has a capacity of 2 cannot be moved from the device 1 to the device 2 because the device 2 has a capacity of 5 and the store C with a capacity of 2 plus the store D with a capacity of 2 and the store B with a capacity of 2 will equal a capacity of 6 which is greater than the device 2 capacity of 5. Similarly, the store D with a capacity of 2 cannot be moved immediately to the device 1 because the device 1 has a capacity of 6 and the store A with a capacity of 3 plus the store B with a capacity of 2 plus the store D with a capacity of 2 will result in a capacity of 7 which is greater than the device 1 capacity of 6. However, it is possible for the store E with a capacity of 1 to be moved in the terminal move 30 to device 1 since the store A with a capacity of 3 plus the store B with a capacity of 2 and the store E with a capacity of 1 will be equal to the device 1 capacity of 6.

[0064] In a sequential migration plan, there may be constraints which make it desirable for the move 38 of store C with a capacity of 2 from the device 2 to the device 4 to be placed last.

[0065] After establishing that no other terminal moves are possible after the store E move 30, a shunt move will be considered. This involves the move 32 of the store D with capacity 2 from the device 2 to the device 3 with a capacity of 2. This permits the store B with a capacity of 2 to make the move 34 from the device 1 to the device 2 with a capacity of 5 which only contains the store C with a capacity of 2. Then the store D with a capacity of 2 can make move 36 from the device 3 to the device 1.

[0066] Finally, the store C with a capacity of 2 can make the move 38 from the device 2 to the device 4.

[0067] Once the goal configuration system 27 is arrived at, the exemplary migration plan of FIG. 2 will be complete in five steps or moves. Similarly, an alternative migration plan would have been to move the store B from the device 1 to the device 3 and move the store D from the device 2 to the device 1 and then subsequently move the store D from the device 3 to the device 1. However, this would have the same number of steps and would probably not have any significant advantages.

[0068] In FIG. 3, the migration planner 116 and the parallelization mechanism 119 of the parallelization migration plan generator 100 generates the complex migration plan 120 of the present invention. The initial configuration data 110 would contain the data on the set of data storage devices (e.g. devices 1 through 3), of the set of data stores (e.g. stores A through E) and the binding of the data stores to the data storage devices (e.g. device 1 with stores A and B; device 2 with stores C and D; and device 3 with store E).

[0069] The goal configuration data 112 would similarly contain the data on the set of data storage devices (e.g. devices 1 through 3), of the set of data stores (e.g. stores A through E), and of the binding of the data stores to the data storage devices (e.g. device 1 with stores A, D, and E; device 2 with store C and B; and device 3 with no stores). The set of constraints would be the capacities (e.g. device 1 with a capacity of 6, device 2 with a capacity of 5, and device 3 with a capacity of 3, store A with a capacity of 3, store B with a capacity of 2, store C with a capacity of 2, and store E with a capacity of 1). The migration planner 116A shown in FIG. 4 starts at “start” 130 and goes to the “build” plan block 132. One arrangement of the migration planner 116A uses a “Simple” plan which embodies a family of simple heuristics to build a plan. Essentially, the heuristic is to make terminal moves until no terminal move can be made. Thus, the program would proceed from the “build plan” block 132 through the “out of time?” decision block 134, the “terminal move?” decision block 136, the “constraints met?” decision block 138, “select terminal move” block 140, “add move to plan” block 142 and then to the “goal configuration met?” decision block 144. If the goal configuration is not met, the migration plan goes back to the “build” plan block 132 to determine another terminal move. The migration planner 116 with a “Simple” plan will either fail to “fail” block 118 or generate the total order migration plan 146.

[0070] The choice of the sequence in which the possible terminal moves will be made can be determined randomly, by a preset ordering, or based on some function of the data store (such as the smallest data store first), etc. These plans are the alternatives in the “Random/Preset/Function Simple” plan 156 of FIG. 5. Each plan will result in a plan with the minimum possible number of moves since all data stores that need to move are moved directly to their terminal or goal configurations or will terminate the migration planner 116 without finding a plan when time runs out.

[0071] Since it is possible for the parallelization migration plan generator 100 to enter a state where there are no valid terminal moves, it is desirable that the migration planner 116 be able to perform shunts by migrating data stores to data storage devices that are not the terminal or goal positions. This is accomplished in a number of different ways as will hereinafter be described.

[0072] The migration plan heuristic must try to move towards the goal configuration while at the same time avoiding “bad” states that make further progress difficult or impossible. To that end, a “contention” metric that quantifies a “bad” state has been developed. The “contention” of a data storage device is defined as the total size of the data stores that need to be moved onto a data storage device (but currently reside elsewhere) divided by the amount of unused capacity on such data storage device. When the “terminal move?” decision block 136 cannot find valid terminal moves, a shunt is performed in the “select shunt” block 146. To select the data store to move in the shunt, the most contended and the least contended data storage devices are determined and the largest possible data store from the most contended data storage device is moved to the least contended data storage device. The Greedy shunt move is then added to the “add move to plan” block 142 and the program continues to seek terminal moves and performs shunts only when there are no valid terminal moves. This plan is also the “Greedy Simple” plan 158 of FIG. 5.

[0073] The contention of the entire data storage system, or global contention, is the sum of the contention over all the data storage devices. When selecting a move, the migration planner 116A will always select the move that leads to the lowest contention for the data storage system rather than the terminal moves. This plan is also the “Greedy” plan 160 in FIG. 5.

[0074] In the “select shunt” block 146 of FIG. 4, the “Cycle” plan finds shunts by first trying to find cycles with small data stores. It does this Greedily, namely, by finding the smallest data store to be moved on one data storage device and then finding the smallest data store to be moved off the second data storage device. Once this cycle is determined, the largest data store in this cycle is shunted to any data storage device that can accommodate it. This will definitely allow at least one terminal move in the cycle to proceed. When picking terminal moves, the terminal move of the smallest possible data store (out of all valid terminal moves) is selected based on any of the above described criteria for a “Simple” plan. This plan is also the “Cycle” plan 162 of FIG. 5.

[0075] In the migration planner 116B of FIG. 5, the program begins at “start” block 150 and moves to select a plan in “select plan” block 152. The “select plan” block receives the initial configuration data 110, the goal configuration data 112, and the set of constraints 114. From the “select plan” block 152 the program goes to the “out of time?” decision block 154 to determine if the plan has been running too long, and if so, to exit to the “fail” block 118. The “Random/Preset/Functional Simple” plan 156, the “Greedy Simple” plan 158, the “Greedy” plan 160, and the “Cycle” plan 162 have previously been described.

[0076] Another plan which could be selected is the “Hillclimbing” plan 164 which uses the global contention metric. At each step the move that minimizes the global contention metric is selected. The migration proceeds directly towards the closest local minimum for the contention metric. Where a local minimum is reached in which no move reduces the contention metric and the goal configuration has not been attained, the migration planner 116B will time out and exit to the “fail” block 118.

[0077] The main drawback of the “Hillclimbing” plan 164 is that it gets stuck easily in local minima, but there are a number of approaches to circumvent this problem, such as backtracking from the local minimum or performing a random perturbation.

[0078] Another plan is the “Simulated Annealing” plan 166 which is a probabilistic Hillclimbing approach that will sometimes choose moves that do not improve the optimization metric, thus allowing the opportunity to get out of local minima. The “Simulated Annealing” planner 166 also uses a contention metric (a slightly different one—) as the energy function to minimize. The contention metric is the sum of log(1+InSetSize(D)/Free(D)) over all data storage devices D where InSetSize (D)=sum (capacity (S)) for all stores that are not currently placed on a storage device, D, that reside on the device in the final configuration. It uses a very simple cooling schedule: the temperature is reduced by a constant ratio every M transitions where M is a numerical trade off between the length of the algorithm and the search space examined for any or a limited number of good plans. The migration will terminate when it reaches the user's set cut-off time or when it finds the goal state.

[0079] The “Simulated Annealing” plan is well suited to finding an optimal or near optimal configuration rather than finding a short path towards a given global optimum. Thus, an alternative would be to base the optimums on migration plans (i.e. sequences of moves) rather than configurations. The “energy” of a migration plan would be its cost (in terms of length). Even migration plans that violate the set of constraints and/or did not reach the goal configuration would be permitted with appropriate penalties added to the energy.

[0080] Based on the above, it would be apparent to those skilled in the art that there are a number of other programs, designated as “Other” plans 168, which could be used to generate migration plans. One such “Other” program 168 involves genetic programming on a population of candidate plans. The fitness criterion depends on how long a plan is, whether it violates constraints, and whether it reaches the goal configuration. As an overall approach, the genetic programming is better than the “Simulated Annealing” plan 166 since plans are easily represented as strings or discrete sequences of moves, which is in keeping with genetic algorithm program formulation.

[0081] Another “Other” program 168 uses the algorithms and heuristics from graph theory. A “sufficiency” graph is built. Since the goal configuration is a legal one, each data store is either in its final destination, has a valid terminal move to its final destination, or has a dependency on (a move to) some other data store. If there are no valid terminal moves, the sufficiency graph has no data stores with moves in but not out, which means it is either empty or contains a cycle. If the graph is empty, then the programn fails, however if there are cycles, graph theory is applied to find the best place to break the cycle(s), i.e., the shunt that would do the most good. This approach is different from the “Cycle” plan 162, which finds cycles with no guarantee that breaking the cycle will enable all the other data stores in the cycle to move to their goal configuration.

[0082] An example of the above graph-theoretical approach encodes the data stores as nodes and dependencies (such as “store B needs to move from device 1 before store A moves onto it”) as moves for “edges”. However, this results in a hypergraph, with edges from sets of data stores to sets of data stores such as, “store A and store B and store E could move to device 1 if store C and store D moved off of it”. Some information is lost by building a sufficiency graph which indicates that there is an edge from store A to store B if it is sufficient that store B move off device 1 for store A to be moved to device 2.

[0083] The “Meta” plan 170 in the migration planner 116B of FIG. 6 switches from one plan to another while generating a plan based on some predetermined factor (e.g. the plan's performance so far). This “Meta” plan 170 is used when there are several plans which work well in some cases and badly on others. The various plans could then be run either concurrently, switching back and forth from one to another, or in parallel, using the shortest resulting plan. For example, the “Hillclimbing” plan 164 would be used until it got stuck, then the “Simulated Annealing” plan 166 would be used for awhile, and then the migration planner 116B would return to the “Hillclimbing” plan 164.

[0084] Using one or a plurality of the plans described above, the migration planner 116B will either fail to produce a migration plan at “fail” block 118 or will produce the total order migration plan 146. The migration plan 120 is used by the control 28 to automatically optimize data storage configurations and change data storage configurations during data system operation.

[0085] The total order migration plan 146 is provided to the parallelization mechanism 180 in FIG. 6.

[0086] In the “identify existence dependencies” block 182, the total order migration plan 146 is taken as “T”. If a migration “A” precedes a migration “B” in T, and A and B migrate the same data store, then there is a dependency B→A in the parallelized migration plan “P”. This ensures that all migrations of any given data store are always in the same order as in T so that the plan does not try to migrate a data store from a data storage device where it does not reside. For example, in FIG. 2, move 36 cannot be executed before move 32 since the move of the store D from the device 3 to the device 1 cannot occur until after the store D has been moved from the device 2 to the device 3.

[0087] After the existence dependencies have been determined, the utilization dependencies are determined in the “identify utilization dependencies” block 184. In the “identify utilization dependencies” block 184, if the migration A precedes migration B in T and the source data storage device D of the migration A is the same as the target data storage device of migration B, then there is a dependency B→A. The rationale is that migration A might be “freeing up” some constraint without which migration B cannot take effect. Since there is some degree of uncertainty, the conservative approach is taken and the dependencies always inserted. In this case, in FIG. 2 move 34 is dependent on move 32 since store B with a capacity of 2 cannot be moved from device 1 to device 2, whose capacity is being totally utilized, until move 32 migrates store D with capacity 2 from device 2 to device 3.

[0088] Thus, the above examines in turn each migration move in T and all the migrations before each migration move in the total order T. For each migration A, only the migrations that qualify as either “existence” or “utilization” dependencies are retained to produce a partial order migration plan “P”. Thus, the P is a list of every migration of T, each tagged with a sublist of its utilization and existence dependencies. There are many equivalent data structure representations of this information.

[0089] As proof, assume a total order migration plan T′ based on P computed from T. Assume by induction that a prefix T′/I is a valid plan and that T′[I+1] is a valid migration. Since T′ satisfies all the “existence” dependencies, the only way that T′[I+1] could be invalid is it violates some constraint on its target data storage device “D”.

[0090] Consider the sequence of all the migrations in T′/[I+1] that involve D (either as a source or as a target) to be “M′”. Let M′[J] be the migration in M that occurs last in T. Let T/K be the sequence of all migrations in T′ up to M′[J]. By definition, T/K contains all the migrations contained in M′. Also, M′ contains all the migrations in T/K with source D (since M′[J] is the last migration in T/K, it has “utilization” dependencies on all migrations with source D that preceded in T).

[0091] Thus, M′ contains a superset of the migrations in T/K with source D and a subset of the migrations in T/K with target D. This means that the set of data stores on D after executing T′/[I+1] is a superset of the data stores on D after executing T/K. Since T/K is a valid plan (being a prefix of T), it follows that the migration T′[I+1] does not violate any constraint on D. The above relies on constrains being non-decreasing, i.e., if assigning a set “S” of data stores to a data storage device satisfies a constraint, so does assigning any subset of S to said data storage device.

[0092] Finally, it is evident that T′ contains the same set of migrations as T, though possibly in a different order. The above indicates that T′ is valid, and that the sequence of migrations on each individual data storage device is the same as in T. This implies that after applying T′, each data store is on the same data storage device as after applying T, i.e., T leads to the global configuration.

[0093] Initial indications are that the above will produce parallelized migration plans that take approximately half as long to execute as the non-parallelized migration plans.

[0094] In the “parallelized independent moves” block 186, any migrations in the migration plan that are not dependent on another migration is set to be executed in parallel to produce the migration plan 120.

[0095] While the invention has been described in conjunction with a specific best mode, it is to be understood that many alternatives, modifications, and variations will be apparent to those skilled in the art in light of the aforegoing description. Accordingly, it is intended to embrace all such alternatives, modifications, and variations which fall within the spirit and scope of the included claims. All matters set forth herein or shown in the accompanying drawings are to be interpreted in an illustrative and non-limiting sense.