Weighted System and Method for Spatial Allocation of a Parallel Load

The present invention relates generally to parallel processors, and more specifically the invention pertains to a dynamically weighted method for spatial allocation of parallel loads to parallel processors that are controlled by a master processor.

Conventional methods of processor allocation generally assume that all processors have the same computational power, even though computing loads on parallel computers typically vary by 50% or more. Further, most techniques repeatedly divide the problem in two, meaning that the number of processors must be a power of the number two to achieve even load distribution, even if fewer processors will suffice. Finally, most conventional methods require that the processor allocation be done serially on a single processor. This can be prohibitively expensive, particularly for load rebalancing, since the entire computing load must be moved to one processor before allocation can proceed.

The task of load balancing the problems solved by parallel processors is alleviated, to some extent by the systems disclosed in the following U.S. Patents, the disclosures of which are incorporated herein by reference:

U.S. Pat. No. 4,400,768 issued to Tomlinson; and

U.S. Pat. No. 4,920,487 issued to Baffles.

Most parallel load balancing is static in nature, with very few methods extending to dynamic allocation techniques. What is needed is a system that is able to dynamically load balance while taking actual processor (node) speed into account. The present invention is intended to satisfy this need.

The present invention includes a system and a method for allocating an arbitrary spatial load in two or three dimensions to an arbitrary number of parallel processing elements. The method produces as output spatial boundaries for each processor, which can then be used to redistribute the parallel computing load. This is best described as a six-step process which is summarized below by a single master processor to perform some centralized allocation tasks, but the load characterization is done in parallel on all processors:

1. Create an allocation structure consisting of a set of spatial boundaries, and set of processors assigned this area of space. Initially create a single structure whose bounds are the problem boundaries containing all N processors on our master processor. Next for each major axis (dimension) repeat the following steps:

2. For each active allocation structure (on the master processor), create D.sqroot.N columns of processors (D=number of dimensions in problem) along the current axis. For each column of processors, total the weight W›i! from individual columns of processors to form a column weight C›j!. Also total all of the column weights to get an overall total weight Tweight.

3. Pass the boundaries of each active allocation structure to each processor. Have each processor generate loading counts for each allocation structure along the active axis as follows:

An array of counts is created and initialed to zero. The size of the counts array depends on the resolution of allocation desired 1000-20000 counts are sufficient for most applications, since counts are applied in each dimension.

The computing load is characterized along the current axis using the counts array. Most often this is done by serially walking the computed elements (particles, points in a computing grid, equations to be computed spatially, or number of data elements in a database) and incrementing the count that spatially corresponds to this element along the current axis. Note that only the counts for the allocation structure that currently contains the element being considered must be incremented. This takes at most (K) time for K elements on a node.

4. Next the counts that were locally gathered in step three are consolidated using message passing on a single processor. Counts for each allocation structure are joined by simply adding corresponding elements of the counts array. When this step is complete, our master processor contains a counts array for each allocation structure that effectively characterizes the current load for that space across all processors.

5. Now the master processor uses a simple iterative technique to create new allocation structure that divide each current allocation structure into columns so that each column contains a load proportional to the weight of the processors assigned to the column. Specifically for each current allocation structure, perform the following steps:

Total the counts for this allocation structure to get a total count called Tcount.

For each column of processors (determined in step 2), calculate the desired count as the column weight C›j! times the total count Tcount divided by the total overall weight Tweight.

Using an index I, and a sum S initialized to zero, iterate through the counts array, summing the counts. When the sum S is equal to or exceeds the desired count for the current column J, create a new allocation structure representing the column with spatial coordinates bounded by the last and current space corresponding to the current counts. This will divide the current allocation structure into weighted columns where the load in the column is approximately proportional to the relative weights for processors in the column.

6. Finally, using the newly created allocation structures (columns), repeat the above process beginning with step 2 above for all remaining dimensions. For a three dimensional problem, this will mean a total of three iterations of the above process. The only change to the above process is that in the final iteration, step 2, each allocation structure is divided into a number of columns equal to the number of remaining processors assigned to the allocation structure rather than columns. This is required to handle an arbitrary number of processors.

It is an object of the present invention to achieve the optimal weighted allocation of N processors, corresponding to the spatial bounds of each processor.

It is another object of the present invention to allocate an arbitrary spatial computational load to an arbitrary number of weighted parallel processing elements. The invention performs this allocation in such a way that each processing element receives a spatial portion of the computing load that is proportional to its weight. As a result, an arbitrary spatial load may be efficiently allocated to an arbitrary number of weighted processing elements for efficient parallel computing.

These objects together with other objects, features and advantages of the invention will become more readily apparent from the following detailed description when taken in conjunction with the accompanying drawings wherein like elements are given like reference numerals throughout.

The sole figure of the drawings is FIG. 1, which is a block diagram of a parallel processor system of the present invention.

The present invention includes a system and a method for allocating an arbitrary spatial load in two or three dimensions to an arbitrary number of parallel processing elements. The method produces as output spatial boundaries for each processor, which can then be used to redistribute the parallel computing load. Unlike existing techniques, this method allows the load to be characterized in communication time proportional to the number of dimensions and memory proportional to the number of processors in parallel. Further, this method takes as input an arbitrary number of processors, as well as arbitrary weight for each processor, and allocates a load proportional to the processor weight to each. As a result, computing loads can be efficiently allocated to the processing elements to maximize speedup due to parallelization.

The reader's attention is now directed towards FIG. 1, which is a block diagram of a parallel processor computer system that uses the present invention. The system of FIG. 1 includes a master processor 100, to allocate tasks to a parallel processor array 103 of N parallel processors P0-P3. When the master processor 100 is programmed with the source code of Table 1, it uses a dynamic weighted method for spatial allocation of parallel loads to parallel processors.

The purpose of this invention is to allocate an arbitrary spatial computational load to an arbitrary number of weighted parallel processing elements. The invention performs this allocation in such a way that each processing element receives a spatial portion of the computing load that is proportional to its weight. As a result, an arbitrary spatial load may be efficiently allocated to an arbitrary number of weighted processing elements for efficient parallel computing.

This description assumes a Multiple Instruction Multiple Data (MIMD), distributed memory parallel processing model with a generic message passing architecture. This invention could easily be converted to run on a Single Instruction Multiple Data (SIMD) shared memory model by replacing message passing references with shared memory accesses. We assume MIMD distributed memory and message passing, as this is generally the more challenging of the two primary parallel computing models to efficiently implement.

Conventional methods of processor allocation generally assume that all processors have the same computational power, even though computing loads on parallel computers typically vary by 50% or more. Further, most techniques repeated divide the problem in two, meaning that the number of processors must be a power of the number two to achieve even load distribution, even if fewer processors will suffice. Finally, most conventional methods require that the processor allocation be done serially on a single processor. This can be prohibitively expensive, particularly for load rebalancing, since the entire computing load must be moved to one processor before allocation can proceed.

As an example, consider the Orthogonal Recursive Bisection Algorithm (ORB), commonly used for parallel load allocation. The ORB method starts by dividing the problem space in two along a line calculated to split the load in half in one dimension. It then recursively repeats this process on the two halves, forming four rectangles with approximately the same load. By the repeatedly dividing these quarters, the ORB algorithm can create areas for any number of processors that is a power of two. Note that the ORB method does not allow for arbitrary weighting or an arbitrary number of processors.

This invention requires an arbitrary set of N processing elements connected by a network capable of performing basic message passing. It also requires a positive weight W›i! for each processing element that characterizes the relative speed of the processor. Portions of the problem will be allocated in proportion to the weight of each processor, with larger weights getting a larger portion of the problem. The problem to be allocated must be able to be allocated spatially. Note that spatial allocation need not indicate a real physical dimension, but could represent a nonphysical dimension, such as the keys in a database. Fortunately the class of spatially divisible problems is large. Examples of problems that can be divided and allocated spatially include most Lagrangian and Eularian numerical codes, database tables, which may be divided in one dimension by row or key, virtually any graphic or rendering problem, and many data gathering and analysis problems. Finally, the problem load must be characterizable in each of its spatial dimensions. For example, a simple numerical code might use the number interpolation points or cells in space to characterize the computing lad. A database might use the number of records. For a graphical problem, the number of items to be rendered in a given space, or number of non-background pixels in an image might be used as the load measure.

Given a problem with the above characteristics, one can apply the following method to achieve the optical weighted allocation of N processors. Output will be a set of rectangular or cubic boundaries corresponding to the spatial bounds of each processor. We use a single master processor to perform some centralized allocation tasks, but the load characterization is done in parallel on all processors:

1. Create an allocation structure consisting of a set of spatial boundaries, and set of processors assigned this area of space. Initially create a single structure whose bounds are the problem boundaries containing all N processors on our master processor. Next for each major axis (dimension) repeat the following steps:

2. For each active allocation structure (on the master processor), create D.sqroot.N columns of processors (D=number of dimensions in problem) along the current axis. For each column of processors, total the weight W›i! from individual columns of processors to form a column weight C›j!. Also total all of the column weights to get an overall total weight Tweight.

3. Pass the boundaries of each active allocation structure to each processor. Have each processor generate loading counts for each allocation structure along the active axis as follows:

An array of counts is created and initialized to zero. The size of the counts array depends on the resolution of allocation desired. 1000-2000 counts are sufficient for most applications, since counts are applied in each dimension.

The computing load is characterized along the current axis using the counts array. Most often this is done by serially walking the computed elements (particles, points in a computing grid, equations to be computed spatially, or number of data elements in a database) and incrementing the count that spatially corresponds to this element along the current axis. Note that only the counts for the allocation structure that currently contains the element being considered must be incremented. This takes at most O(K) time for K elements on a node.

4. Next the counts that were locally gathered in step three are consolidated using message passing on a single processor. Counts for each allocation structure are joined by simply adding corresponding elements of the counts array. When this step is complete, our master processor contains a counts array for each allocation structure that effectively characterizes the current load for that space across all processors.

5. Now the master processor uses a simple iterative technique to create new allocation structures that divide each current allocation structure into columns so that each column contains a load proportional to the weight of the processors assigned to the column. Specifically for each current allocation structure, perform the following steps:

Total the counts for this allocation structure to get a total count called Tcount.

For each column of processors (determined in step 2), calculate the desired count for the current column J, create a new allocation structure representing the column with spatial coordinates bounded by the last and current space corresponding to the current counts. This will divide the current allocation structure into weighted columns where the load in the column is approximately proportional to the relative weights for processors in the column.

6. Finally, using the newly created allocation structures columns), repeat the above process beginning with step 2 above for all remaining dimensions. For a three dimensional problem, this will mean a total of three iterations of the above process. The only change to the above process is that in the final iteration, step 2, each allocation structure is divided into a number of columns equal to the number of remaining processors assigned to the allocation structure rather than D.sqroot.N columns. This is required to handle an arbitrary number of processors.

As mentioned previously, most existing load balancing methods attempt to evenly distribute the load to all processors by assuming that all processors are equal. As a result, these existing methods typically run as fast as the slowest processor on the parallel machine. Because most modem parallel computers and all workstation networks are multitasking systems, processor speeds due to varying load on homogeneous parallel computers can vary by 100% or more. By properly weighting the load on each processor according to the relative processor speed, one can take advantage of the extra capacity available on some timeshared nodes to enhance performance. What is unique about this particular method for dynamic processor allocation is that it allows the problem to be efficiently characterized in parallel, and then dynamically allocated in proportion to processor weight. Further, the allocation is accomplished in a constant number of communications steps, the largest of which is less than the total number of processors. This allows for very efficient load balancing, since the load can be optimally allocated to faster and slower processors without excess communication or memory overhead. We have demonstrated identical problems running 30% faster using this load balancing method on the homogeneous IBM SP-1 parallel computer and up to 50% faster on heterogeneous groups of workstations. This method was demonstrated on a free-Lagrangian Smooth Particle Hydrocode, using nearly half a million irregularly distributed particles.

Variations are possible. Our initial, non-optimal implementation of this method performed a single characterization of the problem in all three dimensions, allowing the problem to be characterized and allocated in a single communications step. Unfortunately, this non-optimal implementation could not properly handle some large irregular load distributions. The current implementation included here very closely approximates an optimal distribution, within the limits of the ability to discriminate counts in each dimension. Note, however, that since some problems are initialized with computed points arranged in a plane along one or more dimensions, slight irregularities in final distribution may exist. Also, the resolution of the counts array used to characterize the load may not be sufficient to allow for resolution of some densely populated spaces. We believe this could be mitigated by allowing for variations in the rectangular or cubic boundaries output from this model. For example, allowing each processor to control a volume described by two or more adjacent boxes or rectangles would let us split planes of contiguous data. Since the existing method has been sufficient for our relatively fine-grained application, we have not yet experimented with these extensions.

This method is particularly effective when tied to an efficient processor weighting method. If the method for weighting the processors efficiently evaluates processor performance and assigns a proper weight as input to this method, large efficiency gains can be made on networks of timeshared computers.

The method described above is implemented when the source code of Table 1 is loaded into the controlling processor of a network of parallel processors. Suitable parallel processing networks are described in the above-cited Baffles and Tomlinson patents.

As mentioned above, the present invention describes a method for allocating an arbitrary spatial load in two or three dimensions to an arbitrary number of parallel processing elements. The method produces as output spatial boundaries for each processor, which can then be used to redistribute the parallel computing load. While prior art systems generally use static load balancing, the present invention uses dynamic allocation techniques.

While the invention has been described in its presently preferred embodiment it is understood that the words which have been used are words of description rather than words of limitation and that changes within the purview of the appended claims may be made without departing from the scope and spirit of the invention in its broader aspects. ##SPC1##