Histogram-based approximation of set-valued query-answers

[0002] 1. Field of the Invention

[0003] The present invention relates to the field of databases. More particularly, the present invention relates to a method for generating histogram-based approximations in response to complex queries to a database.

[0004] 2. Description of the Related Art

[0005] Extremely complex queries are posed to Database Management Systems (DBMSs) through software applications, such as decision support applications and experiment management applications. Given the exploratory nature of such queries, many of the queries require a significant amount of time to execute and then produce results that may be of no particular interest, even though the results are accurate. Answering such queries approximately has been proposed as a technique for reducing query response times when a precise answer is not necessary or when early feedback is helpful. Time can be saved when an approximate answer can be rapidly generated so that a decision can be made based on the approximate answer whether the complete execution of a query should be completed.

[0006] An approximate answer to a query is easily conceptualized when the answer to the query is, for example, an image. Instead of returning the actual image as the answer to the query, a compressed version of the image can be returned as an approximate answer. Alternatively, a series of compressed images can be retrieved, with each successive image being compressed less (more accurate) than the previously retrieved image, and with the last image retrieved being the actual image. This particular approach is provided by many existing systems, including web browsers, such as Netscape Navigator.

[0007] There has been extensive work in connection with approximate query answering ranging from theoretical foundations, to actual systems, to using query generalization for obtaining non-empty query answers. For background regarding established theoretical foundations, see, for example, P. Buneman et al., A semantics for complex objects and approximate queries, Proc. 7th ACM SIGMOD-SIGACT Symposium on Principles of Database Systems, pp. 305-314, April 1988, which is incorporated by reference herein. For background regarding actual systems, see, for example, Ozsoyoglu et al., Processing real-time non-aggregate queries with time-constraints in CASE-DB, Proc. 8th International Conference on Data Engineering, pp. 140-147, Tempe, Ariz., February 1992; and S. Vrbsky et al., APPROXIMATE: A query processor that produces monotonically improving approximate answers, IEEE Trans. on Knowledge and Data Engineering, 5(6), December 1993, each of which is incorporated by reference herein. For background regarding query generalization for obtaining non-empty query answers, see, for example, A. Motro, Query generalization: A method for interpreting null answers, L. Kerschberg, editor, Expert Database Systems, Proceedings from the First Inter-national Workshop, Benjamin/Cummings, Inc., Menlo Park, Calif., pp. 597-616, 1986, which is incorporated by reference herein. All of this work has been based on a subset/superset definition for approximations that has been obtained mostly through a partial query process.

[0008] Online aggregation is another technique used for approximate query answering, but is applicable only for queries that return aggregates, that is, individual values, not sets. The focus of online aggregation techniques has been to efficiently compute aggregates in an online fashion using an interactive interface providing continuous feedback relating to the expected distance of the current aggregate approximation from its actual value. Online aggregation has also been based on defining approximations as subset and/or supersets of an actual answer. For background regarding aggregate queries, see, for example, J. M. Hellerstein et al., Online aggregation, Proc. ACM SIGMOD Conference on the Management of Data, pp. 171-182, Tucson, Ariz., June 1997, which is incorporated by reference herein. For background regarding approximations as subsets and/or supersets of an actual answer, see, for example, P. Buneman et al., supra.

[0009] There has also been a considerable amount of work using statistical techniques for approximating data in databases, particularly in the context of selectivity estimation in query optimizers. Three widely studied classes of statistical techniques are sampling techniques, parametric techniques (approximating the data using a mathematical distribution), and histogram (or non-parametric) techniques.

[0010] For additional background regarding sampling techniques, see, for example, P. J. Haas et al., Sampling-based estimation of the number of distinct values of an attribute, Proc. of the 21st Int. Conf on Very Large Databases, pp. 311-322, 1995; R. J. Lipton et al., Practical selectivity estimation through adaptive sampling, Proc. of ACM SIGMOD Conf., pp. 1-11, May 1990; and S. Seshadri et al., Sampling issues in parallel database systems, Extending Database Technology (EDBT), pp. 328-343, March 1992, each of which is incorporated by reference herein. For background regarding parametric techniques, see, for example, C. M. Chen et al., Adaptive selectivity estimation using query feedback, Proc. of ACM SIGMOD Conf., pp. 161-172, May 1994, which is incorporated by reference herein. For background regarding histogram techniques, see, for example, Y. Ioannidis, Universality of serial histograms, Proc. of the 9th Int. Conf on Very Large Databases, pp. 256-267, December 1993; Y. Ioannidis et al., Optimal histograms for limiting worst-case error propagation in the size of join results, ACM TODS, 1993; Y. Ioannidis et al., Balancing histogram optimality and practicality for query result size estimation, Proc. of ACM SIGMOD Conf., pp. 233-244, May 1995; RP. Kooi,. The optimization of queries in relational databases, Ph.D. thesis, Case Western Reserve University, September 1980; M. V. Mannino et al., Statistical profile estimation in database systems, ACM Computing Surveys, 20(3):192-221, Sept 1988; M. Muralikrishna et al., Equi-depth histograms for estimating selectivity factors for multi-dimensional queries, Proc. of ACM SIGMOD Conf., pp. 28-36, 1988; G. Piatetsky-Shapiro et al., Accurate estimation of the number of tuples satisfying a condition, Proc. of ACM SIGMOD Conf , pp. 256-276, 1984; and V. Poosala et al., Improved histograms for selectivity estimation of range predicates, Proc. of ACM SIGMOD Conf, pp. 294-305, June 1996, each of which is incorporated by reference herein.

[0011] Of these particular techniques, histograms are probably the most widely used statistical technique in commercial database systems. For example, histograms are used in the DBMSs of Oracle, Sybase, Microsoft and IBM, because histograms occupy small amounts of space, do not incur significant overhead at estimation time and histograms are particularly suited for accurately approximating the skewed distributions arising in real-life.

[0012] Previously, several classes of histograms for building on one or more attributes have been identified. Additionally, techniques have been proposed for incrementally maintaining many of the classes of histograms up-to-date as the database is updated. For background regarding classes of histograms, see, for example, V. Poosala et al., Improved histograms for selectivity estimation of range predicates, Proc. of ACM SIGMOD Conf., pp. 294-305, June 1996; and V. Poosala et al., Selectivity estimation without the attribute value independence assumption, Proc. of the 23rd Int. Conf on Very Large Databases, August 1997, each of which is incorporated by reference herein. For background regarding techniques for maintaining many of the novel classes of histograms incrementally, see, for example, Phillip B. Gibbons et al., Fast incremental maintenance of approximate histograrms, Proc. of the 23rd Int. Conf on Very Large Databases, August 1997, which is incorporated by reference herein. Except for sampling none of the other techniques, however, have been previously studied in the context of approximate query answering.

[0013] Nevertheless, such conventional approaches are not particularly useful for many database applications because an answer to an SQL query is a relation, i.e., a (multi)set of tuples. (The term “set”, as used herein, means a multiset whenever no confusion can arise.) Improved insight into whether an accurate answer should be pursued is provided by a set having roughly the same number of tuples as the actual answer to a query and containing values that are approximations of the actual values (e.g., a numeric field having the approximate value of 10 instead of the actual value of 9). Additionally, the improved insight becomes more apparent when the query result is presented visually. In that regard, an analogy of a visually presented approximate answer can be drawn to an approximate (compressed) image because a large number of somewhat misplaced (approximate) points provide an improved approximation to an actual answer than a small number of actual points.

[0014] To illustrate this concept, consider an exemplary employee relational database EMP and a hypothetical query requesting values of the ‘salary’, ‘age’, and ‘department’ attributes from the EMP database. Further assume that the query result is to be visually displayed as a set of points or elements (a starfield) in the salary-age space, such that each point represents an employee having the corresponding salary and age. For this example, the shade (or color) of each respective point represents the corresponding departmental attribute of the employee represented by the point. FIG. 1 shows an exemplary starfield display representing an actual answer to the hypothetical query. The abscissa of FIG. 1 shows the attribute salary, while the ordinate shows the attribute age. The departmental attribute for each respective element is represented by a hollow or solid point.

[0015] Consider two approximations to the query answer of FIG. 1 that is shown in FIGS. 2 and 3, and which each use the same point display scheme as FIG. 1. The first approximation, shown in FIG. 2, is a small subset of the actual answer obtained by a sample-based query processing and is based on a 20% sample. The second approximation, shown in FIG. 3, is a set having elements that are close to elements of the actual answer to the query (value proximity). In most cases, the approximation represented by FIG. 3 is preferable to the approximation represented by FIG. 2 because the approximation of FIG. 3 provides improved insight for the actual answer of FIG. 1. The natural question that arises from the example of FIGS. 1-3 is how can approximate query results of the form shown in FIG. 3 be generated.

[0016] The present invention provides a technique that can be used for generating histogram-based approximate query results. The advantages of the present invention are provided by a method for generating an approximate answer in response to a query to a database in which an SQL query Q for operating on a relation R in a database is received. According to the invention, relation R has an associated histogram H that is preferably a MaxDiff histogram. The SQL query Q is translated to be a query Q′ for operating on histogram H for obtaining a result histogram. The result histogram is then expanded into a relation having tuples containing approximate attribute values based on a frequency of each value. Translating the SQL query Q includes forming a relational algebra operator tree T that is equivalent to query Q. The relational algebra operator tree T includes at least one of a select operation, a project operation and a join operation. A tree T′ is formed by replacing each base relation in leaves of tree T by a corresponding histogram. Another histogram is generated by translating each operator into an SQL query that takes at least one histogram as an input based on children of the operator. When the SQL query Q includes a join operation, the join operation is translated into a sequence of two queries.

[0017] The present invention is illustrated by way of example and not limitation in the accompanying figures in which like reference numerals indicate similar elements and in which:

[0018] FIG. 1 is an exemplary starfield display representing an actual answer to a hypothetical query;

[0019] FIG. 2 is a starfield display that is a first exemplary approximation of the display of FIG. 1 that is based on a sample-based query processing;

[0020] FIG. 3 is a starfield display that is a second exemplary approximation of the display of FIG. 1 that having elements that are close to elements of the actual answer to the query;

[0021] FIG. 4 shows a set of exemplary multisets illustrating the distance metric of the present invention;

[0022] FIG. 5 shows a flow diagram of a valid query translation according to the present invention;

[0023] FIGS. 6aand 6brespectively show an equivalent operator tree and the corresponding translation result according to the present invention;

[0024] FIG. 7 shows the architecture for the AQUA system that can be used with the present invention;

[0025] FIG. 8 shows the results of frequencies as a function of attribute values for the Cusp-Max distribution and the Uniform distribution for various techniques considered for providing approximate answers;

[0026] FIG. 9 shows set difference errors due as a function of space for range queries for range queries;

[0027] FIG. 10 shows the set different error (in log scale) as a function of frequency skew (z value) for range queries;

[0028] FIG. 11 shows the percentage error on estimating counts as a function of space (in bytes) for range queries;

[0029] FIG. 12 shows the percentage error on estimating averages as a function of space (in bytes) for range queries;

[0030] FIG. 13 shows the set difference errors in estimating the distinct values in attributes with a Uniform distribution as a function of space for projections;

[0031] FIG. 14 shows the set difference errors in estimating the distinct values in attributes with a Cusp-Max distribution as a function of space for projections;

[0032] FIG. 15 shows the percentage relative errors in estimating count for the Cusp-Max distribution as a function of space.

[0033] FIG. 16 shows the percentage relative errors in estimating average for the Cusp-Max distribution as a function of space for projections;

[0034] FIG. 17 shows the set difference errors in estimating the result set as a function of space for set join queries;

[0035] FIG. 18 shows the set difference errors in estimating the result set as a function of frequency skew for set join queries; and

[0036] FIG. 19 shows a program storage device having a storage area in which information is stored that tangibly embodies a program of instructions executable by the machine for performing the method of the present invention.

[0037] The present invention generates approximate responses to queries using histograms in a relational database environment. The present invention stores histograms as regular relations in a relational DBMS and translates complex SQL database queries into equivalent algebraic operations on the corresponding histograms so that approximate query answers can be obtained using the same mechanism as exact query answers. To this end, the present invention uses a histogram algebra that can be used for expressing all required queries on histograms. The present invention also provides a dissimilarity metric between relations (in general, multisets), so that errors between accurate and approximate query answers can be satisfactorily quantified.

[0038] As an illustration of the advantages of the present invention, consider the attribute ‘age’ in the exemplary relational database Emp described above and the following simple SQL query attempting to analyze the distribution of age values with respect to a particular threshold:

[0039] select unique age from Emp where age >40

[0040] Further assume that the 3-bucket histogram shown in Table 1 is maintained for the age attribute:

[0041] The 3-bucket histogram of Table 1 can be used for generating an approximate answer to the simple SQL query. The approximate answer is shown horizontally in Table 2 rather than in a traditional vertical manner.

[0042] The approximate answer can be obtained based on the uniform spread assumption within each bucket, that is, assuming that the values that fall within the range of a bucket (the number recorded in the third column of Table 1) are positioned at equal distances from each other. For background regarding the uniform spread assumption, see, for example, V. Poosala et al., Improved histograms for selectivity estimation of range predicates, Proc. of ACM SIGMOD Conf., pp. 294-305, June 1996, which is incorporated by reference herein. The uniform spread assumption may not be as accurate because several actual relations can generate the histogram of Table 1. Consequently, the resulting set is almost never accurate. Nevertheless, the resulting set is a good approximation of the actual answer. Moreover, given the fact that histograms are stored in catalogs that typically fit in memory, obtaining such an approximation is almost instantaneous, thereby providing a valuable and quick first glance at what the nature of the actual answer.

[0043] In principle, the concepts of using histograms for obtaining approximate answers can be generalized for much more complex queries involving multiple attributes from the same relation (either through the attribute value independence assumption or by maintaining multidimensional histograms), joins, aggregates, etc. For details regarding maintenance of multidimensional histograms, see, for example, M. Muralikrishna et al., Equi-depth histograms for estimating selectivity factors for multi-dimensional queries, Proc. of ACM SIGMOD Conf., pp. 28-36, 1988, which is incorporated by reference herein. After all, the histograms maintained within a database are an approximation of the database, so executing a query on the histograms provides an approximation of the actual answer. Moreover, the great advantage of taking the histogram approach of the present invention for approximate queries is that almost all conventional database systems already maintain histograms, so obtaining an approximate answer based on histograms does not require any fundamental changes to conventional database systems.

[0044] Approximation, according to the present invention, is based on accepting any arbitrary set as an approximate answer and determining a distance from the actual answer based on a dissimilarity metric. In the following description, a relational model is used as the underlying environment for the present invention, it should be understood that the methodology used by the present invention can be applied in more general settings.

[0045] Identifying an appropriate metric for measuring the distance between two multisets is essential for any systematic study of approximation of set-valued query answers. In the case of the present invention, the distance that is to be measured is between the multiset representing the actual answer and the multiset representing an approximate answer Unfortunately, sets (let alone multisets) are not typically embedded in metric spaces, so there is no metric for measuring the distance between two multisets, except for the Hausdorff distance, which is used in connection with data clustering. The Hausdorff distance addresses aspects of set differences that are different from approximate query answering. For background relating to the Hausdorff distance, see, for example, Huttenlocher, Comparing images using the Hausdorff distance, IEEE Transactions on pattern analysis and machine intelligence, 15(9), September 1993, which is incorporated by reference herein.

[0046] Two well-known approaches for comparing sets are based on general sets and data distributions. For general sets, a comparison of sets is based on the lattice of subsets generated by a universe of set elements. A typical distance that can be defined is based on the cardinality of the symmetric difference between the two sets S₁and S2, i.e.,

[0047] dist_{{ }}(S₁, S₂)=|(S₁−S₂)∪(S₂−S₁)|.

[0048] This particular distance can be generalized for multisets by making each copy count in the multiset difference.

[0049] For data distributions, a distance measurement is based on various distribution moments. Specifically, if F₁={f₁₁, . . . , f_1n} and F₂={f₂₁, . . . , f_2n} represent the frequency sets of two data distributions on a universe of n elements, then the family (for i≧1) of moment-related distances is: ${\mathrm{dist}}_{\mathrm{dd}}^{\left(m\right)}\left(F_1,F_2\right)=\frac{1}{n}{\left(\sum _{k=1}^n{\left|f_{1k}-f_{2k}\right|}^m\right)}^{1/m}.$

[0050] Note that dist coincides with dist_dd⁽¹⁾for multisets. Clearly, both of these approaches are only applicable for special cases and do not capture proximity the way that is required for approximate query answering. A key limitation is that neither approach takes into account the actual values of the set elements or the data distribution. Instead, the focus is on the presence or absence of the elements and their corresponding frequencies.

[0051] According to the present invention, there is a straightforward total order that can be defined on elements of the data for most common data types (i.e., integers, reals, and even strings) that can be used as a basis for a definition of a distance between two elements. For numeric data types (both integer and real numbers), there is essentially one logical distance metric, that is, the distance dist_numbetween two numbers b₁and b₂is

[0052] dist_dd^(m)(b₁, b₂)=|b₁−b₂|

[0053] Most data models are defined by recursive applications of two major constructors on the basic data types, i.e., a tuple constructor and a set constructor. For tuple objects, several distance metrics have been defined based on the needs of different applications. Among the distance measures for objects, the most common distance measure is a family of distances expressed as power-sums of the individual distances of the tuple elements. In particular, consider two tuples t₁=<(b₁₁, . . . , b_1n) and t₂={b₂₁, . . . , b_2n}. Consider a generic distance metric dist_basicfor the object types of the individual elements of the tuples. The dis tan ce_tup^(m)for any m≧1 between t₁and t₂is ${\mathrm{dist}}_{\mathrm{tup}}^{\left(m\right)}\left(t_1,t_2\right)=\frac{1}{n}{\left(\sum _{k=1}^n{{\mathrm{dist}}_{\mathrm{basic}}\left(b_{1k}-b_{2k}\right)}^m\right)}^{1/m}.$

[0054] The so-called Manhattan distance is obtained when m=1. When m=2, the Euclidean distance is obtained, and so on. Assuming that dist_basicbehaves in the same manner (as is typically the case), the actual values of the tuple elements play an important role in computing the tuple distance.

[0055] Contrary to tuple objects, there are no distance metrics for set or multiset objects that are in common use. Conventional distance metrics are for special cases only and do not incorporate the actual values of the set elements. A key criterion for the distance metric between multisets is to capture as much of the nature of approximation as possible implied in approximate query answering in which both the actual values in an answer and their frequencies are significant.

[0056] Consider two multisets S₁={u₁, . . . , u_n} and S₂={v₁, . . . , v_m}, and a distance metric dist for the object type of the multiset elements. Also consider a complete bipartite graph G_s.swhere the two classes of nodes correspond to the elements of the two multisets. Each element is represented by as many nodes as there are copies of the element in the multiset. Each edge (u_i, v_i) is associated with a cost equal to dist (u_i, v_i). The distance DIST^(l,m)(for any power l≧0m≧0) between S₁and S₂is $\begin{array}{cc}{\mathrm{DIST}}^{\left(l,m\right)}\left(S_1,S_2\right)=\sum _{\left(\mathrm{ui},\mathrm{vj}\right)\in C}{\mathrm{mult}}^1\left(u_i,v_j\right)\times {\mathrm{dist}\left[q\right]}^m\left(u_i,v_j\right),& \left(1\right)\end{array}$

[0057] where C is a minimum edge-cost node cover of G_s.s, i.e., a subset of the edges of G_s.ssuch that (a) for each node, there is at least one edge in C adjacent to the node, and (b) the following expression is minimized: $\begin{array}{cc}\mathrm{MINCOVER}\left(S_1,S_2\right)=\sum _{\left(\mathrm{ui},\mathrm{vj}\right)\in C}\mathrm{dist}\left(u_i,v_j\right).& \left(2\right)\end{array}$

[0058] Essentially, minimum edge-cost node cover C indicates which element of one multiset approximates which element of the other multiset. Distance DIST then determines the distance that the overall approximation implies. For each edge (u_i, v_j) in C such that d₁, edges are incident to u_iand d₂edges are incident to v_iin Eq. (1), the following hold: $\mathrm{mult}\left(u_i,v_j\right)=\{\begin{array}{cc}1& \text{if}\max \left(d_1,d_2\right)=1\\ \max \left(d_1,d_2\right)=1& \text{otherwise}\end{array}\mathrm{dist}\left[q\right]\left(u_i,v_j\right)=\{\begin{array}{cc}\mathrm{dist}\left(u_i,v_j\right)& \text{if}\max \left(d_1,d_2\right)=1\\ \mathrm{dist}\left(u_i,v_j\right)+q& \text{otherwise}\end{array}$

[0059] It can be easily shown that min(d₁, d₂) always equals 1. Essentially, mult(u_i, v_j) captures the extent to which one of u_ior v_jis paired with multiple elements of the other multiset. The multiplication by mult(u_i, v_j) in Eq. (1) penalizes that case accordingly. Likewise, q (for quantum, any small integer, typically 1) increases the distance between paired elements whenever the pairing is not exclusive on both sides.

[0060] FIG. 4 shows a set of exemplary multisets illustrating the distance metric of the present invention. The top and third rows of FIG. 4 respectively show different exemplary multisets for multisets S₁and S₂. The second row of FIG. 4 shows different minimum edge-cost node covers C for the different exemplary multisets S₁and S₂. The fourth row of FIG. 4 shows the DIST metric for the corresponding multisets S₁and S₂. Many well-known metrics fail in correctly computing the distance between multisets, whereas DIST works quite satisfactorily. The specific form of Eq. (1) combines the differences between two multisets on both the actual value level and their frequency of occurrence.

[0061] Some indication of the appropriateness of Eq. (1) as a distance measure between multisets is that the formula is symmetric, i.e., DIST(S₁, S₂)=DIST(S₂, S₁). For certain special cases, Eq. (1) reduces to well-accepted specialized distances. For example, DIST for singleton sets is equal to the distance of their elements. For frequency distributions having the same set of elements appearing in sets and for q=1, DIST is equal to the corresponding moment-related distance raised to the lth power. For identical multisets, DIST is always equal to 0. Additionally, as more space is allocated to various approximation techniques, the DIST value for the actual and approximate results becomes smaller, as should happen. Efficient algorithms are available for computing Eq. (1)

[0062] Before describing additional details of the present invention, the following definitions are presented, which are useful for understanding the histogram-based approximate query answering technique of the present invention. The “value set V_i” of attribute X_iis defined to be the set of values of X_ithat are present in R. Let V_i={v_i(k): 1≦k≦D_i}, where v_i(k)<v_i(j) when k<j. The “spread s_i(k) ” of v_i(k) is defined to be s_i(k)=v_i(k+1)−v_i(k), for i≦i≦D_i. (We see s_i(D_i) =1.) The “frequency f_i(k)” of v_i(k) is defined to be the number of tuples in R with X_i=v_i(k). The “area a_i(k)” of v_i(k) is defined to be a_i(k)=f_i(k)×s_i(k). The “data distribution” of X_iis defined to be the set of pairs T_i={(v_i(1), f_i(1)), (v_i(2), f_i(2)), . . . , (v_i(D_i),D_i))}. Typically, several real-life attributes tend to have “skewed” or highly non-uniform data distributions. The main characteristics of such distributions are unequal frequencies and unequal spreads.

[0063] A “histogram” on an attribute X is constructed using a “partitioning rule” for partitioning the data distribution of attribute X into β (≧1) mutually disjoint subsets called buckets and approximating the frequencies and values in each bucket in some well-known manner. In particular, the most effective approach for values is the uniform spread assumption, such as disclosed by V. Poosala et al., Improved histograms for selectivity estimation of range predicates, Proc. of ACM SIGMOD Conf., pp. 294-305, June 1996, which is incorporated by reference herein. Under the uniform spread assumption, u attribute values are assumed to be placed at equal intervals between the lowest and highest values in a bucket, where u is the actual number of distinct values grouped into the bucket. The most effective approach for frequencies is the uniform frequency assumption, under which the frequencies in a bucket are approximated by their average. The following example, shown in Table 3, illustrates these concepts for a hypothetical attribute X.

[0064] Consider a 4-bucket histogram for hypothetical attribute X with the following bucketization of attribute values: {10} {50, 60}, {70}, {80, 85, 100}. The approximate distribution captured by the 4-bucket histogram is shown in Table 4.

[0065] It is often useful to “expand” a histogram into a relation having tuples containing the approximate attribute values, with each tuple appearing as many times as the approximate frequency of the value. An expanded relation is referred to as the approximate relation (ApproxRel) of the histogram.

[0066] A histogram can also be built on multiple attributes together by partitioning the value combinations of the joint distribution of the attributes into multi-dimensional buckets. The value domain is approximated by an extension of the uniform spread assumption. Details are omitted here, but suffice it to say that the values are spaced at discrete and equal intervals throughout the multi-dimensional region occupied by the bucket. The frequencies are once again approximated by a single average value. It is also possible to combine two histograms on different sets of attributes to obtain a single histogram on the union of the two different sets by making the attribute value independence assumption.

[0067] Given the mechanisms of approximation within a histogram, the accuracy of the approximation is determined by which attribute values are grouped together into each bucket. Several partitioning rules can be used for this purpose. For example, consider the well-known Equi-Width and Equi-Depth histograms, which group contiguous ranges of attribute values into buckets, but differ in the partitioning rule that is used for grouping. In an Equi-Width histogram, all buckets are assigned value ranges of equal length. In an Equi-Depth histogram, all buckets are assigned the same total number of tuples.

[0068] A particular class of histograms called MaxDiff(V,A) performs the best for estimating the selectives of most kinds of queries. In a MaxDiff(V,A) histogram, there is a bucket boundary between two successive attribute values if the difference between the areas of these values is one of the β+1 largest such differences. The area captures both the frequency and value domains. By avoiding grouping dissimilar frequencies or spreads, a MaxDiff(V,A) histogram ensures that the uniform frequency and spread assumptions do not cause many errors in the approximation. As an illustration, consider the exemplary histogram shown in Table 3, which is a MaxDiff(V,A) histogram. Note that the two skewed values 10 and 70 are separated from other values.

[0069] Each bucket in a histogram contains information relating to the total number of tuples that fall in the bucket (tot), and for each dimension, the low and high values (lo, hi) and the number of distinct values (count) in that dimension. Preferably, histograms are stored as regular relations in a database with each bucket forming a tuple. The attributes in a d-dimensional histogram are named as follows: lo₁, . . . , lo_d, hi₁, . . . , hi_d, u₁, . . . , u_d, and tot. The subscripts are dropped for a one-dimensional histogram. For ease of explanation, an additional set of columns sp₁. . . sp_dthat contain the average spreads along each dimension are stored in the histogram, as well as the average frequency of each unique value combination in the bucket (avg). Of course, the average spreads and frequencies can be computed from other columns $\left({\mathrm{sp}}_i=\frac{h_{i_1}-l_{o_1}}{u_i-1}\text{and}\mathrm{avg}=\frac{\mathrm{tot}}{u_1{u}_2\dots u_d}\right)$

[0070] and do not need to be stored. In the following description, the term “histogram” refers to a histogram relation, as described herein. For illustration, the histogram relation for the exemplary histogram of Table 3 is shown in Table 5.

[0071] The present invention provides techniques for automatically translating arbitrary SQL queries into SQL queries on histograms (stored as relations), which can be executed efficiently, for providing approximate answers as if the original SQL queries were executed on the approximate relations represented by the histograms. A main benefit of this approach is that a regular SQL-compliant query processor can be used to provide approximate answers with no modification. An alternative is to implement special-purpose query processing techniques on histograms outside of any standard relational DBMS. Both schemes have been implemented, but only the approach involving query translation is described herein.

[0072] For a histogram H built on a set of attributes S, there is an equivalent approximate relation having the same attributes as in S. Such a relation is denoted as ApproxRel(H). For H, a 1-dimensional histogram can be computed using an exemplary SQL query, set forth below, called Expand.sql. It is straightforward to generalize Expand.sql for a multi-dimensional histogram, but a generalized Expand.sql becomes complex without offering any additional insight, so such a generalization will not be presented.

[0073] SELECT (H.lo+I_c.idx* H .sp)

[0074] FROM H, I_c, I_a

[0075] WHERE I_c.idx≦H .ct & I_a.idx≦H .avg;

[0076] In the exemplary Expand.sql query, H is the histogram stored as a relation, and I_Aand I_Care auxiliary relations, each having a single attribute idx. Relation I_A(I_C) contains the integers 1, 2, . . . , A (1, 2, . . . , C), where A (C) is the largest count (average frequency) in the buckets of H. Essentially, query Expand.sql uses I_Cfor generating the positions of values within each bucket and then uses the low and spread values of the bucket for computing each of the approximate values, under the uniform spread assumption. Likewise, Expand.sql uses I_Ato replicate each value based on its respective frequency. For example, for the histogram of Table 3, A and C are 600 and 3, respectively. Performing Expand.sql on the relation yields a relation having a single attribute containing 100 10's, 125 50's, 125 70's, 600 70's, and 160 tuples each of 80, 90, 100.

[0077] In general, there may be several one- and/or multi-dimensional histograms involving attributes of a given relation R, a combination of which provides an approximation for the entire relation R (e.g., through the attribute value independence assumption). For simplicity of presentation in Definition I below, it is assumed that there is a single multi-dimensional histogram on R.

[0078] Definition 1: Consider a query Q operating on relations R₁. . . R_n, and let H₁, . . . H_nbe the histograms on relations R₁. . . R_n. The correct approximate answer for Q and {H_i} is the result of the following two-step process:

[0079] 1. Compute ApproxRel(H_i) using Expand.sql; and

[0080] 2. Execute a version of Q where, R_ihas been replaced by ApproxRel(H_i) for 1≦i≦n.

[0081] Using Definition 1 to obtain approximate answers, as is, however, is impractical because ApproxRel(H_i) may have as many tuples as R_i, thereby defeating the whole purpose of providing an approximate answer to a query. A different, but equivalent approach to approximate query processing that postpones the application of query Expand.sql to the end is given by Definition 2.

[0082] Definition 2: Consider a query Q operating on relations R₁. . . R_n, and let H₁. . . H_nbe the histograms on relations R₁. . . R _n. A query Q′ on histograms H₁. . . H_nis a valid translation of Q if the result of Q′ is a histogram whose corresponding approximate relation is identical to the correct approximate answer for query Q and {H_i}.

[0083] Definition 2 is illustrated in the transition diagram 50 of FIG. 5. Essentially, Q′ is a valid translation of Q when the diagram commutes. That is, both paths 51 and 52 from the histogram node 53 to the approximate answer node 54 generate the same answer.

[0084] The dashed path 51 in FIG. 5 through approximate relation node 55 corresponds to a naive application (Definition 1) to obtain a correct query answer. The solid path 52 in FIG. 5 through result histogram node 56, however, is the following, much more efficient approach for obtaining the same result:

[0085] 1. obtain a valid translation Q′ of Q.

[0086] 2. execute Q′ on {H_i} to obtain a result histogram H.

[0087] 3. compute ApproxRel(H) using Expand.sql.

[0088] Because most of the query processing takes place on small histogram relations, the approach of path 52, that is, the approach of the present invention, is extremely efficient.

[0089] Aggregate and non-aggregate SQL queries containing Select, From and Where clauses are described below, but nesting, Group-by or Having clauses are not described herein for simplicity and because such features can be added in a straightforward manner. According to the present invention, non-aggregate queries are equivalent to relational algebra expressions involving only selection, projection, and join operations. A query Q in this category of queries is translated as follows:

[0090] 1. Construct a relational algebra operator tree T of select, project, and join operations that is equivalent to Q.

[0091] 2. Replace all the base relations in the leaves of T by their corresponding histograms to obtain another tree T′.

[0092] 3. Starting from the bottom of T′, translate each operator into an SQL query that takes one or two histograms as an input based on the children of the operator and generates another histogram as an output.

[0093] In general, there are many algebraic expressions that may be chosen in step 1 of the non-aggregate query translation process that each provide a different valid translation. Although each respective algebraic expression may differ in cost, the cost is so low that there is no real need for optimization among the different algebraic expressions.

[0094] A key aspect of the non-aggregate query translation process is step 3 because it involves translations of individual operators, which, although not particularly complex, are not completely straightforward either. Consequently, they are described below. For simplicity, only one-dimensional histograms are dealt with. Multi-dimensional histograms are similar, but more cumbersome.

[0095] An equality selection (σ_A=c) is translated into the following query Q=:

[0096] SELECT c, c, 1, ave

[0097] FROM H

[0098] WHERE (c≧1) & (c≦h) & (mod(c−lo.sp)+0);

[0099] A Range selection (σ_A≦c) query is translated into the following query Q_σ: $\begin{array}{ccc}{\mathrm{SELECT}}^{*}& & \mathrm{SELECT}\mathrm{lo},\mathrm{lo}+{\mathrm{sp}}^{*}{\lfloor \frac{c-\mathrm{lo}}{\mathrm{sp}}\rfloor }^{*}\mathrm{ct},{\lfloor \frac{c-\mathrm{lo}}{\mathrm{sp}}\rfloor }^{*}\mathrm{ct},\mathrm{avg}\\ \mathrm{FROM}H& \bigcup & \mathrm{FROM}H\\ \text{WHERE}{h}_i\le c;& & \text{WHERE}\left(\mathrm{lo}\le c\right)\&\left(\mathrm{hi}>c\right);\end{array}$

[0100] Other range predicates can also be written in a similar manner, with suitable modifications.

[0101] Assuming duplicate elimination, a projection (π_A) is translated into the following query Q_x:

[0102] SELECT lo, hi, count,1

[0103] FROM H;

[0104] Assuming no duplicate elimination, projection is merely the identity query for one-dimensional histograms, i.e., selecting all tuples from the histogram. relation with no changes.

[0105] For an Equi-Join(R_1R·A=R.BR₂), let H_ibe the histogram on the joining attribut of R_i, and N_ibe the largest count in the buckets of H_i. A join is translated into a sequence of two queries Q1 and Q2. According to the present invention, an equi-join is more efficient by initially running another simple query for identifying overlapping buckets in the histograms and then executing Q1, and Q2, for each pair of overlapping buckets. For simplicity, this step is omitted. The first query Q1 computes the frequency distribution of the approximate join result by joining the approximate frequency distributions of H₁and H₂. The existence of two auxiliary relations of integers I_Nand I_Nthat are defined in the same fashion as I_Aand I_C, described above, are assumed.

[0106] SELECT (H₁.lo+I_N.idx* H₁.sp) as v, H₁.lo as lo₁, S.so as lo₂, H₁.avg *H₂.avg

[0107] FROM H₁, H₂, I_N, I_N

[0108] WHERE (H₁.lo+I_N.idx* H₁.sp=H₂.lo+I_N.idx* H₂.sp) & (I_N.idx≦H₁.count) & (I_N.idx≦H₂.count);

[0109] The second query Q2. converts the result Q1R of query Q (a frequency distribution) into a histogram by appropriate grouping.

[0110] SELECT min(v), max(v), count(*), avg

[0111] FROM Q1R

[0112] Group By lo₁, lo₂, avg;

[0113] The overall concept of a translation of a non-aggregate query is illustrated in the following example. Consider the following SQL query:

[0114] SELECT R2B

[0115] FROM R1, R₂

[0116] WHERE R1.A≦10 & R1.B=R2.B;

[0117] An equivalent operator tree and the corresponding translation result (a histogram query sequence depicted as nodes in a tree) are shown in FIGS. 6aand 6b, respectively. In

[0118] FIG. 6a, an equality selection σ query is made of a relation R1 at 61. An equi-join query is then performed at 62 using the equality selection of R1 and a relation R2. A projection query is performed at 63. In FIG. 6b, a translation of the equality selection query (61) is formed at 64 and then the query is performed on a histogram H1 that is associated with relation R1. A translation of the equi-join query (62) is formed and the translated query is performed on H1 at 65. Similarly, a translation of equi-join query (62) is formed and the translated query if performed on a histogram H2 that is associated with relation R2. The first query computes the frequency distribution of the approximate join result by joining the approximate frequency distributions of H₁and H₂. A translation of the projection query (63) is formed and executed at 66.

[0119] In general, an aggregate query Q_aggcan be viewed as computing an aggregate over selected attributes in the result of a non-aggregate query Q. Hence, a valid translation for Q_aggconsists of a valid translation for Q producing a histogram H followed by an aggregate-specific SQL query on H computing a 1-element, 1-bucket histogram containing the aggregate value. Aggregate-specific queries for computing aggregate histograms are given in Table 6, for the most common aggregate operators. In Table 6, bsum is equal to $\left(\mathrm{avg}*\mathrm{count}*\left(\mathrm{lo}+\frac{\mathrm{sp}*\left(\mathrm{count}-1\right)}{2}\right)\right).$

[0120] It can be easily seen that the computational complexity for translated queries for selections and projections access only β tuples, where β is the number of buckets in the histogram. This number of buckets in the histograms is usually insignificant compared to the number of tuples in the relations. For joins, the translated queries access u₁+u₂+β₁+β₂tuples, where u, is the number of distinct attribute values in H_iand β_iis the number of buckets in H_i. The total number of operations is also proportional to u₁+u₂+β₁+β₂because a translated join query can be best run using SortMerge joins by always storing the histograms and the auxiliary relations in sorted order. Nevertheless, the complexity of joins is still significantly less than the cost of running the original query because the number of distinct attribute values in H_iis u_i, which is typically much less than the number of tuples in the relations.

[0121] Query rewriting according to the present invention has been incorporated into a general-purpose query processor called AQUA (Approximate QUery Answering System). The architecture 70 of the AQUA system is shown in FIG. 7. User queries 71 are received by a query rewriter 72. Query rewriter 72 generates translated queries 73 that are input to a query processor 74. Query processor 74 accesses a statistics portion 75 of a data warehouse 76, and outputs approximate answers 77 based on histograms. Query processor 74 is also capable of providing accurate answers. The translated queries are rewritten in terms of histogram relations so that query processor 74 can approximately answer most SQL queries. Architecture 70 of the AQUA system also provides incremental maintenance of histograms and sophisticated sampling techniques.

[0122] An extensive set of test runs using the AQUA system were conducted using synthetic and real-life data (TPC-D) for analyzing the effectiveness of various statistical techniques for providing approximate query answers. The test runs involved different data sets and queries having set-valued as well as aggregate results. The results showed an overall superiority of generating approximate answers based on histograms, often producing excellent approximations using the distance metric of the present invention. Furthermore, the same histograms that are optimal for selectivity estimation are optimal for approximate answers based on histograms as well.

[0123] MaxDiff(V,A), Equi-Width, Equi-Depth classes of histograms were used for approximating the data. The corresponding approximate answering techniques are referred to herein as MaxDiff, Equi-Width, and Equi-Depth, respectively. Only MaxDiff captured skews in both value and frequency domains of the data by way of the area parameter because MaxDiff has been shown to be one of the most accurate and efficient histograms for selectivity estimation.

[0124] The traditional uniformity assumption over the entire data was also studied, which is equivalent to a histogram having a single bucket. This technique is referred to herein as Trivial. The final technique, which is referred to herein as Sampling, uses sampling for providing approximate answers. For Sampling, a set of samples was collected on each relation in the database and the submitted query was executed on the sample relations, with appropriate scaling of the final result.

[0125] To ensure a fair comparison among the different techniques, the same amount of space was allocated to each technique. The number of buckets of sample items corresponding to a space of s bytes were computed as follows. Consider a relation with d integer attributes. Because each bucket in a d-dimensional histogram stores (3*d+1) numbers, the number of buckets is $\frac{s}{12*d+4}$

[0126] (assuming 4 bytes per number). Similarly, because each sampled tuple contains d numbers, the corresponding sample set contains $\frac{s}{4*d}$

[0127] tuples. In other words, for a histogram and a sample using the same amount of space, the sample contained 3 to 4 times the number of tuples as the number of buckets in the histogram.

[0128] The test runs used an extensive synthetic database consisting of relations having 50K to 500K tuples. Parameter values of the synthetic database are shown in Table 7. Test runs were also performed on the TPC-D benchmark database, (Transaction processing performance council (TPC). TPC-D Benchmark Manual, 1996, incorporated by reference herein) generated at scale factor of 0.6, which occupied nearly 0.6 GB. The synthetic data offers more insights into the performance of various techniques and is focussed on herein rather than the TPC-D data, which has mostly uniform and independent attributes.

[0129] FIG. 8 shows the frequencies as a function of attribute values for the Cusp-Max distribution and the Uniform distribution. Frequency values for the Cusp-Max distribution are shown at 81, while frequency values for the Uniform distribution are shown at 82. The set of possible values of various data set parameters are given in Table 7.

[0130] The attribute values for the test runs were generated from a combination of Zipf distributions. For additional information regarding Zipf distributions, see, for example, G. K. Zipf, Humatt behaviour and the principle of least effort, Addison-Wesley, Reading, Mass., 1949, which is incorporated by reference herein. The details of all such combinations are given by V. Poosala et al., Improved histograms for selectivity estimation of range predicates. Proc. of ACM SIGMOD Conf, pages 294-305, June 1996. The Cusp-Max distribution consisted of increasing spreads (distances between successive values) followed by decreasing spreads, with the spreads taken from a Zipf distribution. The skew in the spreads is controlled via the z parameter (higher z implies higher skew). The Uniform distribution consisted of equally spaced values.

[0131] The frequencies of the different attribute value combinations were also generated based on Zipf distributions with different levels of skew. Because of non-uniform frequencies, a few combinations of the attributes occurred more frequently together than others, thus modeling limited forms of dependencies between the attributes.

[0132] Non-aggregate and aggregate classes of queries were used for the test runs. Non-aggregate classes included range queries, projections (with duplicate elimination), equi-joins, and select-join (SJ) queries. Range queries had selection predicates of the form ((X_i≦a) and (X_i≧a), where X_iwas an attribute in one of the relations. Two-dimensional range queries, i.e., predicates of the form X_i≦a & X_j≦b, were also studied. In each test run, 100 queries each were randomly generated for the “≦” and “≧” conditions. Select-join queries contained both join and selection operations. The canonical form of the select-join queries was as follows:

[0133] SELECT R₁.A

[0134] FROM R₁, R₂

[0135] WHERE (R₁.A=R₂.A) & (R1.B≦c);

[0136] Projection and equi-join queries are self-explanatory.

[0137] Three classes of queries, 2-dimensional range predicates, SJ queries (which include 1-dimensional range predicates and joins), and projections, that capture all levels of complexity were focussed on.

[0138] In addition to the set-valued results of the above non-aggregate queries, the following aggregate values were also computed on the results: average, variance, sum, and count. The reasons for this are twofold. Aggregate operations are common in decision support systems, and by including count, the effectiveness of various techniques in providing selectivity estimates were studies. In all test runs, the error patterns of any given technique in estimating average and variance were extremely similar. On the other hand, the patterns for sum and count were also similar to each other, but often different from average and variance. The results for two of the aggregate operators are presented, namely, average and count.

[0139] For queries returning sets of tuples as answers, the set dissimilarity metric of the present invention (Eq. (1)) was used as an error metric. For aggregate queries, the percentage relative error $\left(i.e.,\frac{\mathrm{actual}-\mathrm{estimate}}{\mathrm{actual}}*100\right)$

[0140] was used as an error metric.

[0141] The results indicate that the MaxDiff(V,A) histograms provide quality approximations for both set-valued and aggregate queries, outperforming sampling, which is competitive only for aggregate queries having no join operators.

[0142] For each class of queries, the effects of the following parameters as the approximation error were studied: space available to store the statistics (800-80000 bytes), frequency skew (z_f=0-3), and value domain skew (Uniform and Cusp-Max distribution with z_v=0;2-3). In addition to the error metrics, the times taken by various techniques in answering the SJ queries were also measured.

[0143] FIG. 9 shows errors for the various techniques (in log scale) as a function of space (in log scale). The set difference errors for Sampling, Equi-Width, Equi-Depth and MaxDiff are shown at 91-94, respectively. The frequency skew was fixed at z_f=0.86, which roughly corresponds to the “80-20” rule, and the value domain was Uniform. The number of tuples was 200,000. The error for Trivial is not shown in FIG. 9 because it was extremely high (above 4×10⁶) and is out of the range depicted. As expected, the performance of the remaining techniques improved with increasing space. Among these techniques, MaxDiff performed the best and Sampling was noticeably the poorest. The reason for the superior performance of MaxDiff (and other histograms) can be traced back directly to a fundamental property of histograms, that is, they approximate the entire data distribution and capture the more skewed attribute values with high accuracy using a constant amount of space (at most 1 bucket per such value). Hence, histograms are able to perform well on selection queries on random parts of the data.

[0144] Among the histograms, MaxDiff performed best because it captured the skew in the data more accurately than other histograms. In contrast, Sampling captured a fraction of the given set precisely and missed the remaining parts completely. Furthermore, Sampling allocated disproportionate amounts of space to the high frequency values. Because the queries covered the entire value domain uniformly, many of the queries contained the low-skewed regions that may not be captured (or even approximated) at all by Sampling, resulting in a high set difference error.

[0145] For analyzing the effect of frequency skew, the space was fixed at 400 bytes. FIG. 10 shows set different errors (in log scale) as a function of frequency skew (z value) for the various techniques for range queries. In FIG. 10, set difference errors for Sampling, Equi-Width, Equi-Depth and MaxDiff are shown at 101-104, respectively. MaxDiff performed excellently for all levels of skew, whereas the other histograms and Sampling suffered with increasing skew. For Sampling, errors increased with increasing frequency skew because Sampling allocated more samples for the high frequency values, degrading performance for queries on the remaining regions. On the other hand, Equi-Width and Equi-Depth performed poorly because frequency skews are not significantly considered when buckets are formed, unlike MaxDiff.

[0146] Interestingly, MaxDiff performed best for extreme values of skew (0 and 3) because at high skew values, there are very few “important” values in the relation that must be captured and histograms are able to capture the important values using a small number of buckets. At low skew values, almost all frequencies are identical and one or two buckets are enough. For medium values of skew, there are a sufficient number of distinct frequencies so that the histogram needs more buckets for accurately approximating the distribution.

[0147] For aggregate queries, FIG. 11 shows the percentage error on estimating counts as a function of space (in bytes) for range queries, and FIG. 12 shows the percentage error on estimating averages as a function of space (in bytes) for range queries. In FIG. 11, the percentage relative error for Sampling, Equi-Width, Equi-depth and MaxDiff are shown at 111-114, respectively. Also in FIG. 11, the Trivial technique incurred errors in excess of 50% and, consequently, is not shown. In FIG. 12, the percentage relative error for Trivial, Sampling, Equi-Depth, Equi-Width and MaxDiff are shown at 121-125, respectively.

[0148] The other techniques had reasonably small errors because the inaccuracies in approximating individual values are often mitigated when those values are aggregated. MaxDiff once again performed well, even for small amounts of space. The relative performance of various techniques is due to their effectiveness in approximating the entire relation accurately. Most techniques estimate average far more accurately than count. This fact, which is well known in statistical literature, was observed in all test runs.

[0149] The estimation of projections depends solely on the value domains of the attributes. Hence, the performance of various techniques are compared for two different value distributions: Uniform and Cusp-Max. For Non-Aggregate Queries, FIGS. 13 and 14 respectively show the errors in estimating the distinct values in attributes with Uniform and Cusp-Max distributions as a function of space. In FIG. 13, the set difference error for Trivial, Sampling, Equi-Width, Equi-Depth and MaxDiff are shown at 131-135, respectively. All histogram techniques, including Trivial, performed well for the Uniform distribution because the uniform spread assumption is perfectly accurate when the value domain is uniform, regardless of the groupings.

[0150] In FIG. 14, the set difference errors for Trivial, Sampling, Equi-Width, Equi-Depth and MaxDiff are shown at 141-145, respectively. For the Cusp-Max distribution, the more complex histogram techniques, particularly MaxDiff, performed better than Trivial because MaxDiff formed buckets so that the skew in the value domain did not significantly impact the uniform spread assumption. Most noticeably, Sampling performed poorly for projection result estimation. For background regarding projection result estimation, see, for example, P. J. Haas et al., Sampling-based estimation of the number of distinct values of an attribute, Proc. of the 21st Int. Conf on Very Large Databases, pp. 311-322, 1995; and J. F. Naughton et al., On estimating the size of projections, Technical Report 1021, University of Wisconsin-Madison, 1991, which are each incorporated by reference herein. This is particularly true for the Cusp-Max case because Sampling may completely miss values that are far from others, a concept that is penalized in the dissimilarity metric of the present invention (Eq. (1)). The jagged pattern in Sampling errors occurred because the errors depend significantly on the specific values that are sampled, which can vary from sample to sample.

[0151] For aggregate queries, FIGS. 15 and 16 respectively show the errors in estimating count and average for the Cusp-Max distribution as a function of space. In FIG. 15, the percentage relative errors for estimating counts for Trivial, Sampling, Equi-Width, Equi-Depth and MaxDiff are shown at 151-155, respectively. Histogram techniques performed well for count because the histograms accurately keep the number of distinct values, but Sampling degraded rapidly, as expected. In FIG. 16, the percentage relative error for estimating averages for Trivial, Sampling, Equi-Width, Equi-Depth and MaxDiff are shown at 161-165, respectively. All techniques, including Sampling, however, performed reasonably well for average.

[0152] In the final set of error studies, the performance of various techniques were evaluated for approximating the results of complex queries with selection and join operators. The R₁and R₂relations in the SJ-query contained 100,000 and 1000 tuples, respectively, and had 500 distinct values each. The range predicate had a selectivity of 10%. The value distributions of R₁and R₂were both Cusp-Max with z_fof 0.2 and 1, respectively. The frequency distributions of both relations had a skew of z_f=1.5.

[0153] For non-aggregate queries, FIG. 17 shows the errors in estimating the result set as a function of space, and FIG. 18 shows the errors in estimating the result set as a function of frequency skew. In FIG. 17, the set difference error for Equi-Width, Equi-Depth and MaxDiff are shown at 171-173. Strangely, Sampling performed significantly worse in this case (worse than even Trivial) and could not be plotted in the graph due to the resulting extremely high errors. The main reason for this is that the join of two samples often contains few or no tuples in the join because the small samples of the two relations are unlikely to contain any common values. In FIG. 18, the set difference error for Sampling, Equi-Width, Trivial, Equi-Depth and MaxDiff are shown at 181-185, respectively. Once again MaxDiff performed extremely well for the SJ-queries as well. The effects of space and frequency skew were identical to those observed for range predicates.

[0154] For aggregate queries, the relative performance of the various techniques for the aggregates on SJ-query results were similar to that for range predicates, with the difference that the errors due to Sampling were much higher (often exceeding 50%). Histograms, once again, due to their “holistic” approach to approximation, i.e., approximating the entire data, performed well for the SJ-queries as well (the maximum error was around 15%).

[0155] The times taken by various techniques for answering the SJ queries, running on a SUN SPARC machine with 250 MB of memory and more than 10 GB of disk space were measured. Because the time taken by the histogram-based techniques depends solely on the number of buckets, only the times for just one of the histograms (MaxDiff) are presented Table 8.

[0156] Evaluation of the exact answer for this query took about 248 seconds (averaged over 30 runs). Table 8 sets forth the times for different values of space. Max)iff took slightly more time than Sampling because a join query on histograms requires as many operations as the number of distinct values in the join attributes. Nevertheless, the times were still exceedingly small and were insignificant in comparison to the actual time of execution.

[0157] Based on the extensive set of experimental results, the following conclusions about the various techniques are derived. Trivial performed poorly in most cases. Sampling performed satisfactorily only for aggregate queries having no join or projection operators. Among aggregates, Sampling performed well for average and variance, but did poorly for count and sum. Sampling, however, offers probabilistic guarantees on for errors, which are essential in some applications. Furthermore, as new techniques are developed for handling the problem with join queries, sampling is likely to stay as a viable candidate for approximate query answering, at least for aggregate queries.

[0158] MaxDiff performed extremely well and was consistently better than the other techniques that were considered. This was observed for different kinds of data sets, query types (aggregate as well as set-valued, with selection, joins, or projections), and for various aggregate operations. Most of the time, the good performance could be traced to the fact that histograms approximate of the entire data well, and MaxDiff, unlike the other histograms considered, captures both frequency and value domain skews effectively.

[0159] FIG. 19 shows a program storage device 190 having a storage area 191. Information stored in the storage area in a well-known manner that is readable by a machine, and that tangibly embodies a program of instructions executable by the machine for performing the method of the present invention described herein for generating a histogram-based approximate response to a query. Program storage device 190 can be a magnetically recordable medium device, such as a magnetic diskette or hard drive, or an optically recordable medium device, such as an optical disk.

[0160] While the present invention has been described in connection with the illustrated embodiments, it will be appreciated and understood that modifications may be made without departing from the true spirit and scope of the invention.