Estimation of column cardinality in a partitioned relational database

[0002] The present invention is directed to an improvement in computing systems and in particular to an improved system for the estimation of column cardinality in a partitioned relational database.

[0003] In relational database management systems (RDBMS) it is common to utilize query optimizers to improve the efficiency of processing of queries to be carried out on a relational database. One of the most commonly required statistics for such query optimisation is the column cardinality for a table in a relational database. Column cardinality is the number of distinct values contained in a column in the table of the database. In a serial database, column cardinality may be calculated relatively easily if an index is defined on the column being considered. Where there is no index on a column for which the cardinality is to be calculated, a sort of the values in the column is required to provide an exact measurement. Such sorting is an expensive operation and is not desirable in a database system. Because query optimisers do not require exact statistics to function effectively, a good approximation of column cardinality is sufficient in most cases to obtain a good query plan.

[0004] There are a number of techniques known in the prior art to obtain approximations for column cardinality without requiring the sorting of column values in a database table. Examples of such prior art techniques include sample counting, linear counting, and logarithmic counting. These techniques are described in Morton M. Astrahan, Mario Schkolnick, and Kyu-Young Whang, “Counting Unique Values of an Attribute Without Sorting,” Information Systems 12, 1(1987).

[0005] In a partitioned RDBMS, such as a share-nothing parallel database management system, tables may be partitioned across several nodes which do not share data. In such an environment it is potentially difficult to calculate column cardinality. The same value may occur in multiple nodes and therefore it is not possible to simply sum the column cardinality values for each node to obtain a table's overall column cardinality value for the different nodes in the parallel database. One approach is used in the DB2 universal database (UDB) (trade-mark) in the parallel database environment. This approach relies on statistics for column cardinality being calculated on a single node. The node used will be treated as being representative of the data in the column across the different nodes in the partitioned database. In fact, the node may or may not be representative of the data as a whole. As a query is optimised, the overall column cardinality (across all nodes) is estimated using a known probabilistic formula. The column cardinality for the representative node, the number of table rows in that node, and the number of nodes across which the table is partitioned are used to estimate the overall column cardinality. There is overhead involved in such an approach, and the approach is also limited where the node used to represent the data as a whole is in some way atypical of the data value distribution. As a result the estimated overall column cardinality using this approach may vary considerably from the actual value.

[0006] It is therefore desirable to have a technique for estimating the cardinality of a column in a partitioned relational database table which is efficient and which provides a reliable estimate of the column cardinality across all nodes in which the table data is stored.

[0007] A system, method and computer readable medium for estimating a column cardinality value for a column in a partitioned table stored in a plurality of nodes in a relational database is disclosed. According to one embodiment of the present invention, a plurality of column values for the partitioned table stored in each node are hashed, and a hash data set for each node is generated. Each of the hash data sets from each node is transferred to a coordinator node designated from the plurality of nodes. The hash data sets are merged into a merged data set, and an estimated column cardinality value for the table is calculated from the merged data set.

[0008] Advantages of the invention include an efficient technique for providing a reliable estimate of column cardinality in a partitioned relational database.

[0009] FIG. 1 is a block diagram illustrating example nodes in a database in accordance with the preferred embodiment of the invention.

[0010] In the drawing, the preferred embodiment of the invention is illustrated by way of example. It is to be expressly understood that the description and drawings are only for the purpose of illustration and as an aid to understanding, and are not intended as a definition of the limits of the invention.

[0011] FIG. 1 illustrates, in a block diagram, two example nodes of a partitioned database used to demonstrate the estimation of column cardinality according to the preferred embodiment. Node 10 and node 12 are shown, containing table partitions 14, 16, respectively. The table in the partitioned database is partitioned between the nodes 10, 12. In the block diagram of FIG. 1, node 10 is shown containing estimate function 18 and node 12 containing estimate function 20. Node 10 further contains hash data set 22 and similarly, node 12 contains has data set 24.

[0012] According to the preferred embodiment, an initial step of a column cardinality estimation technique is carried out in each of nodes 10, 12 by estimate functions 18, 20, all shown in FIG. 1. Each of estimate functions 18, 20 generate hash data sets 22, 24, respectively. In the preferred embodiment illustrated in FIG. 1, node 10 is shown as a coordinating node which uses combined estimate function 26 to create combined hash data set 28, and the estimated column cardinality value for the table made up of table partitions 14, 16.

[0013] In the preferred embodiment, estimate functions 18, 20 and combined estimate function 26 collectively generate the estimated column cardinality value using a selected one of the known techniques of sample counting, linear counting or logarithmic counting. The most appropriate of these known techniques may be selected for use in a given query optimizer in an RDBMS based on requirements and system resources available. The manner in which each of the three techniques is implemented in the preferred embodiment is set out below.

[0014] Where a sample counting technique is used, a hash function is applied to each data value of the column and a list of hash values is maintained (H list). Subsequent hash values are compared to those contained in the H list and are added to the H list if not present. In practice, the H list is maintained such that there is filtering of hash values before entry into the H list. A sampling parameter K and a reference pattern are used. The values to be entered into the H list must have K bits that exactly match the corresponding K bits of the reference pattern. Initially K is set to 0 (all hash values are entered in the H list). When the H list becomes full, the value of K is increased and the H list is purged of all hash values that do not match the reference pattern to K bits. The reference pattern is typically a string of one values of a length K.

[0015] As will be apparent, the size of the H list is effectively reduced by half on each increase of the value of K. The column cardinality is found by multiplying the final length of the H list by two to the power K(2^K). The column cardinality is then corrected to allow for hash function collisions with a known adjustment formula derived using probabilistic arguments known to those skilled in the art.

[0016] In the examples shown in FIG. 1, the hash data set 22 generated by estimate function 18 includes the H list and the related K value. A similar H list with associated K value is calculated by estimate function 20 in node 12 and is shown as hash data set 24 structure in FIG. 1. In the preferred embodiment, after hash data sets 22, 24 are generated, hash data set 24 is sent to node 10 (the coordinator node). It will be understood by those skilled in the art that the parallel nature of the RDBMS described enables node 10 to be selected as the coordinator node. In other embodiments it may be desirable to process the hash data sets by using a process distinct from the nodes in the parallel database.

[0017] In the preferred embodiment illustrated in FIG. 1, the coordinator node (node 10) computes the overall column cardinality for the two nodes shown in FIG. 1. Node 10 receives hash data set 24 and merges that data set with hash data set 22 to generate combined hash data set 28. It will be appreciated by those skilled in the art that although FIG. 1 shows combined hash data set 28 as a distinct data structure, the merger may occur by one of the existing hash data sets being extended to include the other. As is described in more detail below, the merger of the hash data sets will depend on the column cardinality estimation technique being used in estimate functions 18, 20 and in combined estimate function 26. Once the merger of the hash data value set is carried out, combined estimate function 26 carries out a counting and correction step, as is described in more detail below. The result is the estimated column cardinality value for the table.

[0018] A comparison is carried out in node 10 to determine which of the hash data sets (22, 24) has the highest K value. This hash data set is used as a starting point (primary data structure) and the values in the H list of the hash data set (secondary data structure) are treated as candidates to be entered into the H list of the primary data structure. The resultant merged data is shown in FIG. 1 as combined hash data set 28. The overall column cardinality is calculated once all values of the secondary H list have been added to the primary H list. This is done by multiplying the final length of the primary H list by two, raised to the power of the final sampling parameter in the primary data structure. The column cardinality estimate for the entire partitioned table is then corrected to allow for hash function collisions, as referred to above.

[0019] It is also possible to carry out the above technique for multiple nodes. It is possible to incrementally update the combined hash data set 28 by serially receiving estimate data structures from such multiple nodes. The values in each data structure are transferred to the coordinating node to create the combined estimate data structure in the manner described above. This process is continued until all data structure estimates are received from all nodes in the database.

[0020] In the sample counting technique implementation of the preferred embodiment, the hash data sets from other nodes are received in the coordinator node and are maintained in a queue for processing. A combined hash data set sampling parameter K′ initially set to zero and, until the first hash data set arrives from another node, the value of K′ is set by the coordinator node in the manner described above as it generates a hash data set using the sample counting technique locally. When other hash data set elements are in the queue for processing, the combined hash data set sampling parameter K′ is increased and the global H-List is purged whenever:

[0021] 1. The combined hash data set reaches a maximum defined size, in which case K′ is increased by 1, one purge is executed and the number of masking bits in the reference pattern is increased by 1; or

[0022] 2. The sampling parameter K of an incoming hash data set from another node is larger than the value of K′, in which case (K-K′) purges are executed on the combined hash data set, the number of masking bits in the reference pattern is increased by (K-K′), and K′ is set to the value of K.

[0023] The column cardinality for the table is obtained by multiplying the final length of the combined hash data set by 2 raised to the power K′ and this product is then corrected for hash-function collisions, as described above and known in the art.

[0024] In an alternative implementation, a different estimate technique is used in the nodes and in the calculation of a column cardinality estimate for the entire table (across all nodes containing partitioned portions of the table). Instead of using the sample counting technique described above, a linear counting technique may be used. This linear counting technique is known in the art and is referred to in Astrahan, et al., above.

[0025] In this implementation of the preferred embodiment, the linear counting technique is carried out on each node. According to the linear counting technique, a bit vector is produced. The linear counting technique hashes each value of the column in the node to a bit position in a bit vector. The vector is initially set to be all 0's. Where the data in the node in question has a given value, the corresponding bit in the bit vector is set to 1. When all values in the column in the node have been hashed into the bit vector, the number of distinct values may be derived by counting the number of bits that are set to 1, and adjusting the count with an adjustment formula, derived using probabilistic arguments, to allow for collisions (distinct values in the column that hash to the same location).

[0026] The linear counting technique is used in the alternative embodiment as follows. The linear counting technique is carried out at each of the nodes in the database (in the example of FIG. 1, nodes 10, 12). Each node is constrained to use the same hashing algorithm and the same-sized bit vector in carrying out the linear counting technique. When all column values have been hashed on a node, and without further processing of the bit vector at that node, the bit vector is sent to a single coordinator node for that column (node 10 in the example of FIG. 1). The coordinator node then performs a logical inclusive OR on each bit vector together into a single bit vector. The OR-ing of the N bit vectors is carried out bit-wise: for each bit position L in the resultant bit vector R, the value is a 1 if and only if the value is 1 in one or more of the input bit vectors. Once a combined bit vector is calculated (combined estimate 22 in FIG. 1), the known steps are carried out to determine the estimate for column cardinality for the database as a whole.

[0027] As with the sample counting variant, the bit vector for the database may be incrementally constructed by logically OR-ing the combined estimate bit vector with bit vectors from different nodes in the database as they are received by the coordinating node.

[0028] A further implementation of the preferred embodiment involves the use of the logarithmic counting technique known to those skilled in the art (see Astrahan et al., above). The logarithmic counting technique is used on each node, producing a bit map for each column on each node. The details of the logarithmic counting technique are as follows. For each data value in a column in a node, a binary hash value is calculated. Each binary hash value is then transformed by a function which leaves only the leftmost ‘1’ bit in the binary value unchanged, all other bits being set to zero. Each transformed value is then OR-ed into the bit map for the node. When all data values of the columns have been processed, the estimate for number of distinct values in the column is calculated by dividing 2 raised to the power n by q, where n is the position of the leftmost ‘0’ bit in the bit map, starting with zero on the left, and q is a parameter specified by the size of the bit map used. For example, q 0.7735 when a 32-bit map is used. The values of q for different sized bit-maps are known to those skilled in the art.

[0029] The logarithmic counting technique is used, in this implementation of the preferred embodiment, at each of the nodes in the database. The technique is constrained to use the same hashing algorithm, the same transformation function, and the same-sized bit-map at each node. When all data values have been processed on a node, the resultant bit map is sent to a single coordinator node for that column. The coordinator node carries out a logical (inclusive) OR of the bit-maps (in a bit-wise fashion). The result is a bit-map for the database as a whole. The technique of logarithmic counting is then carried out on this combined database to calculate an estimate for the column cardinality of the entire database.

[0030] As with the sample counting and linear counting approaches described above, the logical OR-ing of the bit-maps in the coordinator node does not need to wait until all bit maps have been received. The bit-map for the table may be incrementally defined.

[0031] As may be seen from the above description, in the preferred embodiments of the invention, a merge step is carried out in the coordinating node (node 10 in the example of FIG. 1). Where column cardinality for different columns is estimated to assist in query optimization, it is possible to improve the techniques set out above using parallelization. In the merge step described above, the coordinator node computes the overall column cardinality for the C columns of a given table in a sequential manner, and that the other nodes are idle during this time. For a table T with C columns C1, C2, . . . , Cc, the computation of the column cardinality for column Ci, is independent from the computation of the column cardinality for another column Cj,. It is possible to assign the computation of the merging step for each column to a different node, thereby computing that step for different columns, in parallel.

[0032] To achieve this parallelization, a plurality of nodes will be identified as coordinator nodes, and each column is assigned to one of these nodes. The estimate data structures will be sent to the appropriate node as defined by the column assignments. The process of identifying the plurality of nodes which will participate in the final steps of the technique is carried out based on the total number of nodes across which the table is partitioned and the total number of columns in the table. If the number of columns is greater than the number of nodes, then all the nodes are eligible to participate in the merge phase. Each node will compute the overall column cardinality for an average of C/N columns, where C is the number of columns in the table and N is the total number of nodes across which the table is partitioned.

[0033] However, if the number of columns is less than or equal to the number of nodes across which the table is partitioned, then only C<=N nodes are eligible to participate in the final step of the algorithm. Each of the C nodes computes the overall column cardinality for one column of the table. Where maximum parallelization is not required, other approaches to assigning coordinator nodes may be used.

[0034] Although a preferred embodiment of the present invention has been described here in detail, it will be appreciated by those skilled in the art, that variations may be made thereto, without departing from the spirit of the invention or the scope of the appended claims.