Method for converting relational data into a structured document

[0002] This application claims priority under 35 USC § 119(e) to U.S. provisional patent application serial No. 60/181,400, filed on Feb. 9, 2000.

[0003] The present invention relates to the field of data exchange between applications on the Internet. More particularly, the present invention relates to converting relational data into XML (eXtensible Markup Language).

[0004] XML (eXtensible Markup Language) can serve many purposes. XML is a more expressive markup language than HTML (Hyper-Text Markup Language). XML may be an object-serialization format for distributed object applications. XML serves as the standard format for data exchange between inter-enterprise applications on the Internet. In data exchange, XML documents are generated from persistent data and then sent over a network to an application. To facilitate data exchange, numerous industry groups, such as healthcare and telecommunications groups, have been defining public document type definitions (DTDs) that specify the format of the XML data to be exchanged between their applications. The aim is to use XML as a “lingua franca” for data exchange between inter-enterprise applications. XML could make it possible for data to be exchanged regardless of the platform on which it is stored or the data model in which it is represented.

[0005] Most data is stored in relational or object-relational database management systems (RDBMS) or in legacy formats. To realize the full potential of XML, tools are needed that can automatically convert the vast stores of relational data into XML. Such tools should be general, dynamic, and efficient.

[0006] Relational data is tabular, flat, normalized, and its schema is proprietary, which makes it unsuitable for direct exchange. In contrast, XML data is nested and un-normalized, and its DTD is public. Thus, the mapping from relational data to an XML view is often complex, and a conversion tool must be general enough to express complex mappings. Existing commercial systems fail to be general, because they map each relational database schema into a fixed, canonical DTD. This approach is limited, because no public DTD will match exactly a proprietary relational schema. In addition, it is often desirable to map one relational source into multiple XML documents, each of which conforms to a different DTD. Hence a second step is required to transform the data from its canonical form in XML into its final XML form.

[0007] Also, the tools must be dynamic, i.e., only the fragment of the XML document needed by the application should be materialized. In database terminology, the XML view must be virtual. The application typically specifies in a query what data item(s) it needs from the XML document. Typically, these items are a small fraction of the entire data. Some commercial products allow users to export relational data into XML by writing scripts. However, these tools are not dynamic. Rather, they are general because the entire document is generated all at once.

[0008] Finally, to be efficient, such tools must exploit fully the underlying query engine of RDBMS whenever data items in the XML view need to be materialized. Query processors for ative XML data are still immature and do not have the performance of highly optimized DBMS engines.

[0009] Several commercial tools for exporting relational data into XML views exist today. The ODBC2XML, a product of Intelligent Systems Research (www.intsysr.com) tool allows users to define XML documents with embedded SQL statements, which permit the users to construct an XML view of the relational data. Such views are materialized, however, and cannot be further queried with an XML query language. Alternatively, Oracle's XSQL tool defines a fixed, canonical mapping of the relational data into an XML document, by mapping each relation and attribute name to an XML tag and tuples as nested elements. Such a view could be kept virtual, but this approach is not general enough to support mapping into an arbitrary XML format. IBM's DB2, XML Extender provides a Data Access Definition (DAD) language that supports both composition of relational data in XML and decomposition of XML data into relational tables. DAD's composition feature supports generation of arbitrary XML from relational data. However, the criteria for grouping elements is implicit in the DAD and DAD specifications cannot be nested arbitrarily. More significantly, XML Extender does not support query composition.

[0010] The present invention overcomes many of the shortcomings of the prior art. In addition, the present invention addresses the problem of automating the conversion of relational data into XML. According to the invention, a general, dynamic, and efficient tool for viewing and querying relational data in XML referred to as SilkRoute is provided. SilkRoute is general, because it can express mappings of relational data into XML that conform to arbitrary DTDs, not just a canonical mapping of the relational schema. The mappings may be referred to as views. Applications can express the data they need as an XML-QL query over the view. SilkRoute is dynamic, because it can materialize the fragment of an XML view needed by an application, and Silkroute is efficient, because it can fully exploit the underlying RDBMS (Relational DataBase Management Systems) query engine whenever data items in an XML view need to be materialized.

[0011] According to one aspect of the present invention a general framework is provided for mapping relational databases to XML views, to be used in data exchange. In another aspect of the invention, a new query language, RXL, for mapping relational sources to XML views, is provided. According to yet another aspect, the present invention provides a sound and complete query composition algorithm that, when given an RXL query and an XML-QL query, generates a new RXL query equivalent to their composition. In a still further aspect of the present invention, a technique is provided in which most of the work of an RXL query can be shipped to the underlying database engine.

[0012] Although the invention has been defined using the appended claims, these claims are exemplary and limiting to the extent that the invention is meant to include one or more elements from the apparatuses described herein in any combination or sub-combination. Accordingly, there are any number of alternative combinations for defining the invention that incorporate one or more elements from the specification (including drawings, claims, etc.) in any combinations or sub-combinations.

[0013] The present invention will now be described in more detail with reference to preferred embodiments of the invention, given only by way of example, and illustrated in the accompanying drawings in which:

[0014] FIG. 1 shows an illustrative architecture of SilkRoute according to the present invention.

[0015] FIG. 2 depicts an illustrative architecture of query composition according to the present invention.

[0016] The present invention includes SilkRoute, a general, dynamic, and efficient tool for viewing and querying relational data in XML. SilkRoute is a particular instance of a mediator system, as defined by Geo Wiederhold in “Mediators in the Architecture of Future Information Systems” IEEE Computer, pages 38-49, March 1992. In SilkRoute, data can be exported into XML in two steps. First, an XML view of the relational database may be defined using a declarative query language, called RXL (Relational to XML Transformation Language). The resulting XML view can be virtual. Second, some other application formulates a query over the virtual view, extracting some piece of XML data. For this purpose, an existing XML query language, XML-QL, may be used. Only the result of that XML-QL query can be materialized.

[0017] The core of SilkRoute is RXL, a powerful, declarative data-transformation language from flat relations to XML data. On the relational side, RXL has much of the power of SQL queries and can express joins, selection conditions, aggregates, and nested queries. On the XML side, RXL has the full power of XML-QL, and can generate XML data with complex structure and with arbitrary levels of nesting. It can also specify arbitrary grouping criteria, using nested queries and Skolem functions. Typical RXL queries are long and complex, because they express general transformations from the relational store to the XML view. RXL has a block structure to help users organize, structure, and maintain large queries.

[0018] Once the virtual XML view is defined, SilkRoute accepts XML-QL user queries and composes them automatically with the RXL query. The result of the composition is another RXL query, which can extract that fragment of the relational data that the user requested. In existing relational databases, composition can be straightforward and not considered a problem. For example, Ramakrishnan and Gehrke, in “Database Management Systems” McGraw-Hill, 2nd ed. 1999, describe how to reformulate SQL queries over SQL virtual views as SQL queries over base relations. For XML, however, this problem is more complex. The present invention provides a sound, complete, and conceptually simple algorithm that when given an RXL query and an XML-QL query, produces a new RXL query equivalent to their composition. Some restrictions can be placed on aggregate functions in RXL queries, but they can be used freely in XML-QL queries.

[0019] When an RXL query is evaluated, the underlying relational engine does most of the processing. To attain this result, the RXL query can be split into a collection of SQL queries, each of which produces a set of tuples. The SQL queries can be sent to the RDBMS, and their flat, sorted results can be merged in a single pass to construct the nested XML output.

[0020] An illustrative implementation of the present invention in electronic commerce, in which suppliers provide product information to resellers, will be described. For their mutual benefit, suppliers and resellers can agree to exchange data in a format that conforms to a particular DTD. An illustrative DTD of XML data exported by suppliers to resellers, supplier.dtd, is shown below.

[0021] The above code includes the supplier's name and a list of available products. Each product element includes an item name, a category name, a brief description, a retail price, an optional sale price, and zero or more trouble reports. The content of a retail or sale element typically is a currency value. A trouble report includes a code attribute, indicating the class of problem; the report's content may be the customer's comments. Most importantly, this DTD can be used by suppliers and resellers, and it can be a public document.

[0022] Consider now a particular supplier whose business data is organized according to the relational schema. An illustrative schema of a supplier's relational database (* denotes key) is depicted in the code below.

[0023] Clothing(*pid, item, category, description, price, cost)

[0024] SalePrice(*pid, price)

[0025] Problems(pid, code, comments)

[0026] The Clothing table contains tuples with a product id (the table's key), an item name, category name, item description, price, and cost. The SalePrice table contains sale prices and has key field pid and the Problem table contains trouble codes of products and their reports. The above code shows a third-normal form relational schema, designed for the supplier's particular business needs. The schema can be proprietary. For example, the supplier may not want to reveal the attribute cost in Clothing. The supplier's task is to convert its relational data into a valid XML view conforming to the DTD and make the XML view available to resellers. In this example, it is assumed that the supplier exports a subset of its inventory, in particular, its stock of winter outerwear that it wants to sell at a reduced price at the end of the winter season.

[0027] Once the XML views of a suppliers' data are available, the reseller can access that data by formulating queries over the XML view. Some examples of such queries may include: 1) retrieve products whose sale price is less than 50% of the retail price; 2) count the number of “defective” reports for a product; and 3) compute minimum and maximum cost of outerwear stock. As these queries might suggest, the reseller is typically interested only in a small subset of the information provided by the suppliers. Those skilled in the art will recognize that these queries could be formulated as SQL queries over the supplier's relational database, but relational schemas can differ from supplier to supplier and may not be accessible by the reseller.

[0028] FIG. 1 shows an illustrative architecture of SilkRoute according to the present invention. SilkRoute 100, serves as middleware between a relational database server (RDBMS) 110, and an application 120, accessing data over a distributed network, such as the Web/Intranet 130. The distributed network may be a public or private network. According to the invention, the database administrator starts by writing a view query that defines the XML virtual view of the database. In a preferred implementation of the present invention, the view query is an RXL query. The view query is typically complex, because it transforms the relational data into a deeply nested XML view. The resulting view query is virtual, meaning that it is not evaluated, but kept in source code.

[0029] Typically, applications contact SilkRoute 100, to request data. An application 120, only “sees ” the virtual XML view, not the underlying relational database. To access the data, the application 120, can formulate a user query in XML-QL over the virtual view and send it to SilkRoute 100. Together, the view query (e.g., RXL view query) and the user query (e.g., XML-QL user query) can be passed to the query composer module 102, in SilkRoute 100. The query-composer module 102, computes the composition and produces a new view query (e.g., RXL query), called the executable query. The answer to the executable query typically includes only a small fragment of the database, e.g., one data item, a small set of data items, or an aggregate value. The result of SilkRoute 100, is an XML document, as specified by the user query (e.g., XML-QL user query).

[0030] Once computed, the executable query is passed to the translator 104, which partitions the executable query into a data-extraction part, e.g., one or more SQL queries, and an XML-construction part, e.g., an XML template. The translator 104, also may take as an input a description of the relational schema and uses the relational schema to perform syntax checking of the RXL query (e.g., to ensure that the relations named in the RXL query exist in the relational database) and to determine the capabilities of the SQL dialect used by the relational database (e.g., does the SQL dialect support inner joins?).

[0031] Until now, SilkRoute 100, has manipulated only query source code, but no data. At this point, the data extraction part (e.g., SQL queries) is sent to the RDBMS server 110, which returns one tuple stream per each query (e.g., SQL query) in the data extraction part. The XML generator module 106, merges the tuple streams with the XML-construction part and produces the XML document, which is then returned to the application 120.

[0032] This scenario is probably the most common use of SilkRoute. However those skilled in the art will recognize that minor changes to the information flow in FIG. 1 can permit other scenarios. For example, the data administrator may export the entire database as one large XML document by materializing the view query. This can be done by passing the view query directly to the translator. In another scenario, the result of query composition could be kept virtual for later composition with other user queries. This is useful, for example, when one wants to define a new XML view from an existing composed view.

[0033] A. The View Query: RXL

[0034] Next, RXL (Relational to XML transformation Language) is described. RXL essentially combines the extraction part of SQL, i.e., a from and a where clause (possibly followed by sort by and/or group by clauses) with the construction part of XML-QL, i.e., the construct clause.

[0035] As a first example, consider this RXL query, which defines a fragment of an XML view:

[0036] from Clothing $c

[0037] where $c.category=“outerwear”

[0038] construct <product>

[0039] <name>$c.item</name>

[0040] <Category>$c.category</category>

[0041] <retail>$c.price</retail>

[0042] </Product>

[0043] Given a database like that in the supplier's schema, the query can produce an XML fragment like the following:

[0044] <product><name> . . . </name><category> . . . </category><retail>. . . </retail></product>

[0045] <product><name> . . . </name><category> . . . </category><retail> . . . </retail></product>

[0046] A root element is missing; later it will be explained how to add one.

[0047] As in SQL, the from clause declares variables that iterate over tables. Variable names start with a $. In this example, $c is a tuple variable that iterates over the Clothing table. The where clause contains zero or more filters (Boolean predicates) over column expressions. The column expression $c.item refers to the item attribute value of $c and in this case, requires that it equal the string “outerwear”. The construct clause specifies the XML value, called an XML template, in terms of the bound column expressions.

[0048] RXL has three powerful features that make it possible to create arbitrarily complex XML structures: nested queries, Skolem functions, and block structure. An example of a nested query is:

[0049] The outer query has no from or where clauses, only a <construct> clause for the root element <view>. The first sub-query builds one <product> element for each row in Clothing. Its inner sub-query creates zero or more <report> sub-elements, one for each report associated with that product. Those skilled in the art and familiar with SQL will recognize this as a left-outer join of Clothing with Problems followed by a group by on Clothing.

[0050] Skolem functions allow the way elements are grouped to be controlled. Recall that in XML an attribute with type ID contains a value that uniquely identifies the element in the document, i.e., a key. In RXL, the distinguished attribute ID always has type ID, and its value is a Skolem term, which is used to control grouping and element creation. For example, in the following:

[0051] from Clothing $c

[0052] construct<category ID=Cat($c.category) name=$c.category>

[0053] <product>$c.item</product>

[0054] </category>

[0055] Cat is a Skolem function and Cat ($c.category) is a Skolem term whose meaning is that only one <category>element exists for every value of $c.category, and it includes all products in that category:

[0056] <category><product>p1</product><product>p2</product></category>

[0057] <category><product>p3</product><product>p4</product></category>

[0058] . . .

[0059] Without the ID. attribute and its Skolem term, the query would create one <category> element for each row in Clothing:

[0060] <category><product>p1</product></category>

[0061] <category><product>p2</product></category> . . .

[0062] When Skolem terms are missing, RXL introduces them automatically. Since Skolem terms could be used to define arbitrary graphs, RXL enforces semantic constraints that guarantee a view always defines a tree, and therefore, a well-formed XML document. For example, the Skolem term of a sub-element must include all the variables of its the parent element.

[0063] Finally, the block structure allows RXL to construct parts of complex elements independently. The query below shows an illustrative multi-block RXL view query containing two blocks.

[0064] construct

[0065] <view ID=View( )>

[0066] {from Clothing $c

[0067] construct<product ID=Prod($c.item)>

[0068] <name ID=Name($c.item)>$c.item</name>

[0069] <Price ID=Price($c.item, $c.price)>$c.price</price>

[0070] </product>}

[0071] {from Clearance $d

[0072] where $d.disc >50,

[0073] construct <product ID=Prod($d.prodname)>

[0074] <name ID=Name($d.prodname)>$d.prodname</name>

[0075] <discount ID=Discount($d.prodname, $d.disc)>$d.disc</discount>

[0076] </Product>

[0077] </view>

[0078] The first block creates elements of the form:

[0079] <product><name>n</name><price>p</price></product>

[0080] for each product name in Clothing. The second block creates elements of the form:

[0081] <product><name>n</name><discount>d</discount></product>

[0082] for each product name in Clearance. It is to be assumed that Clearance(*prodname, disc) is part of the supplier's schema. When the same product name occurs both in Clothing and Clearance, then the two elements will have the same ID key and can be merged into:

[0083] <product><name>n</name><price>p</price><discount>d</discount></product>

[0084] Those skilled in the art and familiar with SQL will recognize this as an outer join.

[0085] The below code contains the complete view query, RXL view query (V), for the supplier relational schema example described above.

[0086] Lines 1, 2, and 26 create the root <supplier> element. Notice that the Skolem term Supp( ) as no variables, meaning that one <supplier> element is created. The outer-most clause constructs the top-level element supplier and its company child element. The first nested clause (lines 4-26) contains the query fragment described above, which constructs one product element for each “outerwear” item. Within this clause, the nested clause (lines 13-17) expresses a join between the Clothing and Sale Price tables and constructs a sale element with the product's sale price nested within the outer product element. The last nested clause (lines 18-24) expresses a join between the Clothing and Problem tables and constructs one report element containing the problem code and customer's comments; the report elements are also nested within the outer product element. Notice that the Skolem term of product guarantees that all product elements with the same identifier are grouped together. Usually Skolem terms can be inferred automatically, but they have been included explicitly, because they are relevant to query composition described herein.

[0087] B. The User Query: XML-QL

[0088] Applications do not access the relational data directly, but through the XML view. To do so, applications provide user queries in XML-QL, a query language specifically designed for XML. XML-QL queries contain a where clause followed by a construct clause. The where clause contains an arbitrary number of XML patterns and filters. The construct clause is identical to that in RXL.

[0089] In the example described herein, the reseller can retrieve all products with a sale price less than half of the retail price using the XML-QL user query (U) below:

[0090] The where clause includes a pattern (lines 3-10) and a filter (line 11). A pattern's syntax is similar to that of XML data, but also may contain variables, whose names start with $. Filters are similar to RXL (and SQL). The meaning of a query is as follows. First, all variables in the where clause are bound in all possible ways to the contents of elements in the XML document. For each such binding, the construct clause constructs an XML value. Grouping is expressed by Skolem terms in the construct clause. In this example, the construct clause produces one result element for each value of $company. Each result element contains the supplier's name and a list of name elements containing the product names.

[0091] In this example, the answer to the user query includes a small fraction of the relational database, i.e., only those products that are heavily discounted.

[0092] C. The Query Composer

[0093] The query composer module 102, of SilkRoute 100, takes a user query and the RXL view query and generates a new RXL query, which is equivalent to the user query evaluated on the materialized view. In the example described herein, the view query is the RXL view query (V) above, the user query is the XML-QL user query (U), and the composed query, RXL query (C) is shown below.

[0094] The composed query combines fragments of the view query and user query. Those fragments from the user query are highlighted. The composed query extracts data from the relational database in the same way as the view query. It also includes the user filter $s.price<0.5$c.retail and structures the result as in the user query. The details of the composition are subtle, and a complete description of the composition algorithm is described later herein.

[0095] The composed query is referred to as executable, because it is typically translated into SQL queries and sent to the relational database engine. The answer of the executable query is quite small—the same as that of the user query. In general, it is more efficient to execute the composed query, instead of materializing the view query, because composed queries often contain constraints on scalar values that can be evaluated using indexes in the relational database. Such indices are of little or no use when evaluating a view query. For example, consider a user query that specifies the condition: $s.price between 80 and 100. This condition is propagated into the executable query, and then into the SQL query, and can be evaluated efficiently if an index exists on price. In contrast, an index on price is useless when materializing the view query directly.

[0096] D. Translator and XML Generator

[0097] The translator 104, takes an RXL query and decomposes it into one or more SQL queries and an XML template. The SQL queries are executed by the RDBMS 110, server or engine, and their flat results (streams of tuples) are converted into XML by the XML generator 106.

[0098] The translator 104, also takes a source description, which is an XML document specifying systems information needed to contact the source: the protocol (e.g. JDBC), the connection string, and a source-specific query driver. The driver translates RXL expressions into the source's query language, which is typically a dialect of SQL. Although one skilled in the art will appreciate that other query languages can be supported. For example, the executable RXL query (C) is translated into the following SQL query:

[0099] select c.pid as pid, c.item as item

[0100] from Clothing c, SalePrice s

[0101] where c.category=“outerwear”,

[0102] c.pid=s.pid,

[0103] s.price<0.5*c.retail

[0104] sort by c.pid

[0105] and into the XML template:

[0106] <results>

[0107] <result ID=Result(“Acme Clothing”)>

[0108] <supplier>“Acme Clothing”</supplier>

[0109] <name ID=Name($pid, $item)>$item</name>

[0110] </result>

[0111] </results>

[0112] where the variables $pid and $item refer to the attributes pid and item in the SQL query's select clause; the template generation is described in more detail in section II, part A below. After translation, the SQL query is sent to the relational engine, RDBMS 110, and the resulting tuple stream is fed into the XML generator 106, which produces the XML output.

[0113] In this example, the translation requires only one SQL query. In general, there may be several ways to translate a complex RXL query into one or more SQL queries and to merge tuple streams into the XML result. Choosing an efficient evaluation strategy may be important when the RXL query returns a large result, e.g., if the entire XML view is materialized. SilkRoute can have one or more evaluation strategies, which can generate one SQL query for each disjunct of an RXL sub-query, which must be in disjunctive-normal form (DNF). Each SQL query has a sort by clause, making it possible for the XML generator 106, to merg the queries into an XML document in a single pass.

[0114] 5. Alternative Approaches

[0115] The above example of the present invention has been described in terms of a general approach for exporting relational data into XML. Other approaches are possible, and in some cases, may be more desirable.

[0116] Currently, the most widely used Web interfaces to relational databases are HTML forms with CGI scripts. A script can translate user inputs into SQL queries, and the query answers can be rendered in HTML. The answers could be generated just as easily in XML. Forms interfaces may be appropriate for casual users, but may not be appropriate for data exchange between applications, because they limit the application to only those queries that are predetermined by the form interface. Aggregate queries, for example, are rarely offered by form interfaces.

[0117] In another alternative implementation of the invention, the data provider can either pre-compute the materialized view or compute the view on demand whenever requested by an application. This alternative can be feasible when the XML view is small and the application needs to load the entire XML view in memory, e.g., using the DOM (document object module defined by the World Wide Web Consortium DOM Recommendation) interface. However, pre-computed views are not dynamic (i.e., their data can become stale) and are not acceptable when data freshness is critical.

[0118] Another alternative implementation of the present invention uses a native XML database engine, which can store XML data and process queries in some XML query language. XML engines will not replace relational databases, but a high-performance XML engine might be appropriate to use in data exchange. For example, one could materialize an XML view using SilkRoute and store the result in an XML engine running XML-QL, thus avoiding the query composition cost done in SilkRoute. However, XML engines may not match the performance of commercial SQL engines anytime soon. In addition, this approach can suffer from data staleness, and incur a high space cost (e.g., for disk space) because it duplicates the entire data in XML.

[0119] In this section, the query composition algorithm is described. As discussed previously, an RXL query, such as V, takes a relational database as an input and returns an XML document as an output. The XML-QL user query, such as U, which is written against V, takes an XML document as an input and returns an XML document. For any database D, the result of U can be computed by first materializing V(D), denoted as XMLD, and then computing U(XMLD). The query composition problem is to construct an equivalent RXL query C, where C=U V. In other words, it would be desirable to construct an RXL query C that is guaranteed to yield the same result as U and V for any database D, that is, C(D)=U(V(D)). C takes as an input a relational database and returns an XML document. With C, the construction of the intermediate result XMLD is skipped. As an example, RXL view query (V), and XML-QL user query (U) can be used with the result of the composition, C, being composed RXL query (C).

[0120] Before describing the details, a brief intuitive description is given. The key observation is that all XML components (tags, attributes, #PCDATA) present in XMLD are explicitly mentioned in the construct clause(s) of RXL view query (V). When XML-QL user query (U) is evaluated on XMLD, its patterns are matched with these components. The key idea is then to evaluate XML-QL user query (U) on the templates of RXL view query (V) directly, without constructing XMLD. During this evaluation only the patterns are considered and not the filters occurring in user query (U). In this example, user query (U) has a unique pattern that mentions <supplier>, <company>, <product>, <name>, <retail>, and <sale> with a particular nesting, and all these tags also occur in the templates of view query (V) under the same nesting. RXL view query (V) is shown again below, this time after the matching, with the matched tags being bolded.

[0121] That is, the RXL view query (V) is shown with patterns from the XML-QL user query (U) highlighted. Once the matching is done, the composed query (C) can be constructed in a second step, as follows. The construct clause of the composed query (C) is the same as the construct clause of the XML-QL user query (U), modulo variable renaming. The from and where clauses of the composed query (C) include both of the “relevant ” from and where clauses in the view query (V) and all the where filter conditions in the user query (U), modulo variable renaming. This completes the construction of composed query (C). In this example, the “relevant ” from and where clauses are:

[0122] from Clothing $c, SalePrice $s

[0123] where $c.category=“outerwear”, $s.pid=$c.pid

[0124] and the where filter condition in user query (U) is $sale<0.5*$retail which becomes the following after variable renaming:

[0125] where $s.price<0.5*$c.retail

[0126] Inspection of the composed RXL query (C) shown above indicated that the from and where clauses, together, form the from and where clauses of the composed query (C).

[0127] FIG. 2 depicts the architecture of query composition according to the present invention. The pattern matcher 140, implements a first step, which involves evaluating user queries (U) on view query (V) templates. During the first step, the user query (U) patterns are matched with view query (V) templates. The result is a solutions relation, R, in which each tuple represents one match. Multiple matches may occur if the patterns contain alternation, e.g., <company|organization>, or Kleene-star operators, e.g., <*.supplier>, or tag variables <$elm>. A rewriter 150, carriers out a second step by taking the remaining clauses (the from and where of the view query (V) and the construct of the user query (U)) and the relation R, and rewriting each solution tuple into one RXL clause. The result is the composed query C.

[0128] The illustrative query composition technique can be viewed as an example of partial evaluation, where the patterns are evaluated at composition time (a.k.a. compile time) on view query (V) templates, and the filters and constructors are evaluated at run time when the new RXL view is evaluated. Section II, parts A-D of the description describe the internal representation of view and user queries and provide a detailed description of an illustrative composition algorithm according to the present invention. A pseudo code version of the algorithm appears in section III.

[0129] A. Step 1: Pattern Matching

[0130] In Step 1, the solutions relation R that contains all matchings of user query (U) patterns with view query (V) templates can be constructed.

[0131] 1. Construct the View Tree.

[0132] For the composition algorithm, the view query V may be represented by a data structure called a view tree, which includes a global template and a set of datalog rules. The global template can be obtained by merging all view query (V) templates from all its construct clauses. Nodes from two different templates may be merged if and only if they have the same Skolem function. Hence, each Skolem function occurs exactly once in the view tree. The datalog rules are non-recursive. Their heads are the Skolem functions names, and their bodies include relation names and filters. The datalog rules can be constructed as follows. For each occurrence of a Skolem function F in a view query (V), one rule is constructed of the form F(x, y, . . . ):-body, where body is the conjunction of all from and where clauses in the scope where F occurs. When a rule is associated with a Skolem function, then that rule guards the Skolem function and its corresponding XML element. In both the template and datalog rules, the tuple variables used in RXL can be replaced by column variables.

[0133] Below is the template of the view tree for the RXL query on the left and the datalog rules of the view tree for the RXL query on the right according to the illustrative example of the present invention described herein.

[0134] The unique supplier element is guarded by the rule Supp( ):-true, which is always true, because no predicate expression guards the element's creation. The retail elements are guarded by the rule:

[0135] Retail($cpid, $cprice):-Clothing($cpid_, $category_, $cprice), $category=“outerwear”

[0136] which means that one retail element is created for each value of cpid and cprice that satisfies the table expression on the right-hand side. There is only one datalog rule for each Skolem function, because each function occurs once in the query view (V).

[0137] 2. Evaluate User View (U) on the View Tree.

[0138] Next, the patterns of user view (U) can be matched with the template of view query (V).

[0139] To simplify presentation, it is assumed that the user view (U) includes a single block as represented by Equation 1:

[0140] U=construct <elm> {where P, Wconstruct T} </elm>  (1)

[0141] where T denotes the template, P denotes all patterns, and W denotes all filters. New, temporary variables in U's patterns can be introduced, with one variable for the ID attribute of each element in the pattern. In this example, U has a single pattern and six new variables are added, one temporary variable for each element in the pattern, as shown below.

[0142] <supplier ID=$t1>

[0143] <company ID=$t2>$company</company>

[0144] <product ID=$t3>

[0145] <name ID=$t4>$name</name>

[0146] <retail ID=$t5>$retail</retail>

[0147] <sale ID=$t6>$sale</sale>

[0148] </product>

[0149] </supplier>

[0150] The necessity of these variables and how to handle multi-block user queries are described in section II, part A(3) below.

[0151] Next, U's patterns on V's template are evaluated in the standard way of evaluating patterns on a tree. In general, there may be zero, one, or more results. The results can be represented as a table R, with one column for each variable in U, and one row for each result. The values in the table are #PCDATA, Skolem terms, variables, tag names, attribute values, and attribute names, which occur in V's template. In this example, the step results in the following table R:

[0152] The column names correspond to the variables in U's single pattern shown above. The single row in R means that there exists only one matching of U's pattern with V's template. The row specifies that U's variable $name is bound to $citem in V, the variable $t3 is bound to the Skolem term Prod($cpid), and the variable $company is bound to the #PCDATA Acme Clothing.

[0153] B. Step 2: Query Rewriting

[0154] In Step 2, the table R can be used to construct the composed query C. Each row in R represents one match, and composed query C is the union of all possible matches. In particular, composed query C includes several parallel blocks, which denote a union in RXL. In each block, the from and where clauses contain the “relevant ” datalog rules, that is the rules for the Skolem functions in the corresponding row. The construct clause of the block contains the template of the user view U. Recall that U includes a single block (Eq. 1), and that T denotes its template, P its patterns, and W its filters. Let the rows in R be r₁. . . r_k. Then C includes several parallel blocks:

[0155] C=construct{<elm>{B₁} . . . {B_k} </elm>}

[0156] with one or more blocks corresponding to each row r_i. In the next section, how blocks corresponding to one row, r_i, in R are constructed is described.

[0157] 1. Construct One Block

[0158] To construct the from and where clauses of one block, the clauses are represented as one datalog rule. Then, the rule is converted into a from-where clause. Let F₁. . . F_n, be the Skolem functions that occur in the row r_i. Recall that the view tree associates one or more datalog rules to each Skolem function. Assume that there is a unique datalog rule for each Skolem function:

[0159] F₁:-body₁. . . F_n:-body_n.

[0160] The block's construct clause is S₀(T) where S₀is a variable substitution defined below. For each datalog rule F_i, one variable substitution S_iis applied. The body of the new datalog rule is the union of all bodies after variable substitution, plus S₀(W). Thus, the new rule has the form:

[0161] Q(S₀(x), S₀(y), . . . ):-S₀(W), S₁(body₁), . . . S_n(body_n) where x, y, . . . are the variables in U's template T. Next, Q is minimized, and rewritten as a from-where clause: all relation names appearing in the from clause, and all filters appearing in the where clause. This completes the construction of one block.

[0162] 2. Variable Substitutions

[0163] Next, the substitutions of S₀and S₁. . . S_nare defined. For all the datalog rules F₁, . . . F_n, the substitutions S₁. . . S_nare constructed so that the expressions S₁(body₁) . . . S_n(body_n) all have distinct variables, with one exception. For every two columns t_j, t_kin R, where the variable t_jcorresponds to an element that is the parent of t_k's element, all variables in S_j(F_j( . . .)) can be shared with S_k(F_k( . . . )). To compute S₀, the substitutions S₁. . . S_nare applied to the entire row r_iand drop all columns in r_ithat correspond to the temporary variables $t1 . . . $t2. The new row is S₀, which maps U's variables to variables, constants, and Skolem terms.

[0164] When there is more than one datalog rule per Skolem function, the resulting datalog program is converted into disjunctive normal form, i.e., a disjunction of multiple conjunctive datalog rules, before generating the RXL blocks. For each conjunctive rule, the construction above can be applied to obtain one block and take the union of all such blocks. In this case, more than one block for one row r_ican be obtained.

[0165] In this example, table R has one row that contains the Skolem terms Supp( ), Comp( ), Prod($cpid), Name($cpid, $citem), Retail($cpid, $cprice), and Sale($cpid, $spid, $sprice). Their corresponding datalog rules are shown in the view tree for the RXL query in section 11, part A(1) above. Next, the substitutions S₁, . . . , S₆are computed such that the rules have disjoint variables with the exception of variables that have parent/child relationships. In this example, the variable $t3 is the parent of variables $t4, $t5, $t6; see the pattern in section II, part A(2) above. Therefore the Skolem term Prod($cpid) shares the variable $cpid with that in Name ($cpid, $citem), Retail ($cpid, $cprice), and Sale ($cpid, $spid, $sprice). Otherwise, all variables must be distinct. The modified rules are:

[0166] The substitution S₀is obtained directly from the table R, by dropping all columns corresponding to the new variables $tl, . . . , $t6:

[0167] The template T of U is in the user query shown in section 1, part B. The filter W of U is $sale <0.5* $retail. Only the variables $company and $name occur in T, so S₀($company) and S₀($name) need to be included in the rule's head; $company, however, is a constant, therefore the rule becomes:

[0168] The last line is S₀(W). Minimizing Q, the following equivalent query is obtained:

[0169] Finally, the rule can be converted into from and where clauses, and a construct clause can be added whose template is S₀(T):

[0170] Lastly, column variables are replaced by tuple variables, and the single-block query C shown in section I, part C can be obtained.

[0171] C. Other Exemplary Implementations

[0172] The above-described exemplary implementation of the present invention illustrates a simple example of query composition. Below, several other exemplary implementations of the present invention that illustrate more complex cases are described.

[0173] 1. View Tree for Multi-block Query

[0174] Consider the two block RXL query in section I, part A. Below, on the left side is the view tree template for the two block query and on the right side is the datalog for the two block query.

[0175] In the RXL query, the Skolem functions Prod and Name occur twice. In the view tree, each function has two corresponding datalog rules, but in the template, they occur once.

[0176] 2. Multiple Rows

[0177] In general, R may contain multiple rows. To illustrate R with multiple rows, the query V described and shown in section II, part A(1) is employed. R is composed with the following XML-QL user query U′:

[0178] The regular expression supplier.product.(retail|sale) matches a retail or a sale element nested within a supplier and a product element. It is analogous to the XPath expression /supplier/product /retail|sale. There are two matches of U with V, which produce two rows in R:

[0179] The temporary variables $t1, $t2, $t3, are for supplier, product, and retail|price, respectively. The composed query C has two blocks:

[0180] C=construct <results>{B1}{B2} </results>

[0181] The relevant datalog rules for the first row are those for Supp, Prod and Retail of view query described in section II, part A(1). No variables are renamed, because $t2 is the parent of $t3. The generated datalog rule after minimization is:

[0182] Q($cprice):-Clothing($cpid,_, $category,_,$cprice), $category=“outerwear”

[0183] and it produces C's first block BI:

[0184] B1=from Clothing $c

[0185] where $ c.category=“outerwear”

[0186] construct <price>$c.price</price>

[0187] The relevant datalog rules for the second row are those for Supp, Prod, and Sale. As before, no variables are renamed, and the datalog rule is:

[0188] Q($cprice):-Clothing($cpid,_, $category,_, _, $category=“outerwear”, SalePrice($spid, $sprice), $cpid=$spid

[0189] which produces C's second block B2:

[0190] B2=from Clothing $c, SalePrice $s

[0191] where $c.category=“outerwear”, $c.pid=$s.pid

[0192] construct <price>$c.pid</price>

[0193] 3. Adding Template Variables

[0194] The temporary variables $t1, $t2, etc. added to U's patterns play an important role, as revealed by the next example. The query V, written directly with column variables, is:

[0195] and the two XML-QL queries U, U′ can be considered:

[0196] Both return pairs of <b> values, but the first query returns pairs where both <b>'s are in the same <a> element. Without temporary variables in U's patterns, the relation R would be the same for U and U′. After introducing the new variables, the two relations R have different column names, and as expected, they produce two distinct composed queries.

[0197] 4. Renaming variables in Datalog Rules

[0198] Continuing with the previous example, the need for the substitutions S₁, S₂, . . . First, V's view tree is constructed:

[0199] Next, the composition with U′ is illustrated. Five temporary variables are added and U's pattern becomes:

[0200] Matching the pattern with the template produces one row in R:

[0201] Intuitively the variable $y in the $z1 column is different from $y in the $z2, column, because they match different <b> elements, possibly in different <a> elements. This distinction is made precise by the renaming step. Thus, after variable renamings, the five relevant datalog rules become:

[0202] and the composed query C, after query minimization, is:

[0203] construct <results> from T($x1, $y1), T($x2, $y2)

[0204] construct <result><z1>$y1<z1><z2>$y2</z2></result>

[0205] </results>

[0206] 5. XML-QL Queries With Block Structure.

[0207] In general, U may have several blocks, both nested and parallel. For multi-block user queries, a different table R for each block in U can be constructed, in the same way in which the XML-QL query processor handles multiple blocks. Tables corresponding to parallel blocks are independent; for nested blocks, there is a distinct inner table that corresponds to each row in the outer table. The composed query C follows the same block structure, except that one block in U may generate multiple parallel blocks in C, as described above early in this section.

[0208] 6. Query Minimization

[0209] Query minimization eliminates redundancies in queries, such as duplicate conditions. Query minimization can be expensive, because it is NP-complete. That is, the complexity of query minimization is exponential in the number of variable in the query. Commercial database systems often do not perform minimization, because users typically do not write redundant queries. In SilkRoute, the composed query C can be generated automatically. One condition in a view query V may appear in multiple datalog rules, and, hence be propagated as multiple copies in the generated query C. To avoid query minimization, one could trace these repetitions to the original RXL query, but care is needed to deal with variable renamings. For RXL queries with large parallel blocks, however, query minimization may be unavoidable.

[0210] D. Aggregation Queries

[0211] Briefly, it is described how aggregations in XML-QL queries can be “pushed ” into composed RXL views and evaluated by the target RDBMS according to the present invention. In both XML-QL and RXL, Skolem terms can be used to specify the values by which aggregate expressions are grouped.

[0212] Suppose a reseller wants to count the total number of reports for each defective product. This can be expressed in XML-QL as follows:

[0213] where <supplier.product ID=$pid>

[0214] <name>$n</>

[0215] <report>$r</>

[0216] </> in “http://acme.com/products.xml”

[0217] construct <product ID=F($pid)>

[0218] <name ID=G($pid, $n)>$n</>

[0219] <totaldefects ID=H($pid)>count(*)</>

[0220] </>

[0221] The Skolem term F($pid) in <product ID=F($pid)> asserts that all bindings for the variables $pid, $n and $r are grouped by $pid's value. Similarly, the Skolem term H($pid) specifies the grouping attributes for the aggregate function count(*), which counts the total number of bindings. This idea is similar to the GROUP BY construct in SQL. XML-QL and RXL's semantics guarantee that only one element is produced for each value of a Skolem term, e.g., one name element is emitted for each value of $n.

[0222] A simple extension to datalog that accommodates aggregate functions can be used. An example of a datalog rule that can use a “generator ” to count values is:

[0223] C(p, q, COUNT(*)):-R(p, q)

[0224] Only the last argument in the head can be an aggregate function; the other arguments specify the grouping attributes. The meaning is that C contains the set of triples (p, q, r) where r is the number of tuples in the group corresponding to values (p, q) in the relation R.

[0225] Using this composition algorithm, the XML-QL query above can be rewritten as:

[0226] from Clothing $c, Problems $p

[0227] where $c.pid=$p.pid

[0228] construct <product ID=F($c.pid)>

[0229] <name ID=G($c.pid, $c.item)>$c.item</>

[0230] <totaldefects ID=H($c.pid)>count(*)</>

[0231] </product>

[0232] Note that the aggregate function can be “pushed ” into the RXL view. When this view is materialized, the aggregation can be evaluated by the relational engine. Significantly, this query can be evaluated efficiently, because commercial database systems are often highly optimized for aggregation queries.

[0233] In the formal description of the algorithm above, a notation for describing the types of values that are manipulated, e.g., view trees, XML-QL blocks, are needed. Types are denoted by grammar rules, such as the following:

[0234] These rules specify that a view tree Node is composed of a tag, a rule, and a list of children nodes. A Rule is composed of a Skolem term (its head) and a conjunctive list of conditions (its body). A Condition is either a table expression, a filter expression, or the disjunction of two lists of conjuncts.

[0235] An XML-QL block is represented by a list of patterns, a list of filters, and a template. An RXL block is represented by a list of conditions and a template:

[0236] A template is either: a constant string; a variable; an element, which includes a tag and list of nested templates; or a nested query. To simplify presentation, templates are polymorphic, i.e., an XML-QL template contains only a nested XML-QL block and similarly, for an RXL template.

[0237] Finally, a canonical pattern is represented by the head variable (that occurs on the right-hand side of in), a regular-path expression over strings, and the target variable (that occurs in the body of an element):

[0238] Pattern:-Var, RegPathExpr, Var

[0239] In this example, each regular-path expression is one string atom, but in general, strings can be combined with the alternation (|), concatenation (.), and Kleene-star (*) operators, similar to those used in regular expressions.

[0240] The composition function compose, as shown below, takes two environments, which are lists of (variable, value) pairs. Shown below is an illustrative top-level compose function for a composition algorithm:

[0241] The initial environment (X_env) maps the distinguished variable $viewtree to the root of the view tree referenced by the query. The initial variable-substitution S that maps XML-QL variables to RXL expressions is empty, and X_block is the top-level XML-QL block (lines 1-3). In this example, $viewtree is bound to the root of the tree defined in the RXL query in section II, part A(1). The result of compose is a list of RXL blocks. In the pseudo-code, XML-QL expressions are prefixed by X_ and RXL expressions by R_

[0242] Function compose (line 7) decomposes X block into its patterns, filters, and template, and rewrites each nested pattern in a canonical form as a list of unnested patterns. New temporary variables are introduced to represent the intermediate nodes in the nested pattern.

[0243] On line 13, the patterns are evaluated in the current environment, producing R, which maps XML-QL variables to nodes and constants in the view tree. Each tuple in R represents one possible rewriting of the XML-QL query over the view. For each tuple ri the current environment is extended with the new variable bindings (line 19).

[0244] Function newVariables (line 22) computes the new mappings of XML-QL and RXL variables to common RXL variables. In summary, newVariables recovers the correspondence between Skolem terms that share a common ancestor in the XML-QL pattern; this correspondence determines the mappings for RXL variables. For XML-QL variables, the mapping is simple. If the corresponding value is a leaf node or constant value, the variable is replaced by its value in the substitution mapping S described in section II, part B. If the corresponding value is an internal node, the variable is replaced by the complete RXL expression that computes that element under the substitution S. Lastly, function oneSolution (line 25) takes the new environment and computes the new RXL blocks, which are appended to the list of other potential solutions.

[0245] The composition algorithm for the oneSolution function in the following code constructs the RXL block in three steps.

[0246] First, for each XML-QL variable X_v in X_block, it projects Xv's value from the solution tuple r_i. Its value is a view-tree element and an associated rule, whose head and body are projected in R_head and R_body, a list of conditions. Function makeCopy (line 13) assigns fresh variable names to all free variables in R_condition, i.e., those that do not occur in the rule's head. Function rewriteR (line 15) rewrites the new rule, using the variable mapping S. The new condition is added to the conjunctive conditions in R_conditions. Second, the function rewriteX (line 20) rewrites the XML-QL filters in X_filters and adds those to R_conditions. Third, the function to_DNF (line 22) puts the new conditions in disjunctive normal form. On line 23, rewriteX rewrites the XML-QL template to produce the new RXL template. Finally, one new RXL block is created for each list of conjuncts in R_disjuncts, and the union of all these blocks is returned.

[0247] The rewriteX and rewriteR functions in the composition algorithm of the rewrite function below 14, replace XML-QL and RXL variables by their new names in S.

[0248] The “helper ” functions substX and substR perform the variable substitutions. Note that rewriteX calls compose recursively to rewrite a nested XML-QL block into an equivalent nested RXL block.

[0249] IBM's DB2, XML Extender provides a Data Access Definition (DAD) language that supports both composition of relational data in XML and decomposition of XML data into relational tables. DAD's composition feature, like RXL, supports generation of arbitrary XML from relational data. Unlike RXL, the criteria for grouping elements is implicit in the DAD and DAD specifications cannot be nested arbitrarily. More significantly, XML Extender does not support query composition, however, DAD could be used as a view-definition language in a SilkRoute architecture.

[0250] SilkRoute is a general, dynamic, and efficient framework for viewing and querying relational data in XML. SilkRoute is an XML-export tool that can support arbitrarily complex, virtual views of relational data and support XML user queries over virtual views. The ability to support arbitrary views is critical in data exchange between inter-enterprise applications, which must abide by public DTDs and cannot reveal the underlying schemas of their proprietary data. SilkRoute has many benefits. For example, the fragment of the relational data requested by a user query need only be materialized; that requested data can be produced on demand; and the relational engine can perform most of the computation efficiently.

[0251] SilkRoute has one translation strategy, which generates one SQL query for each RXL sub query, which must be in disjunctive-normal form (DNF). In practice, RXL view queries can be arbitrary boolean combinations of table and filter expressions; for example, parallel. RXL blocks often construct parts of complex elements independently, i.e., they express unions. User queries over such views often produce composed queries with many unions. Any RXL sub-query can be normalized into multiple sub-queries in DNF, which can result in a quadratic increase in the number of sub-queries to evaluate. In practice, multiple queries in DNF can be translated directly into SQL, for example, by using SQL's union-join constructs. Similarly, nested RXL queries often express left outer joins, e.g., the parent sub-query is the left relation and the child sub-query is the right relation. Two SQL queries can be generated, one for parent and child, but one SQL query suffices. In addition to reducing the number of SQL queries, each individual RXL sub-query can be minimized, i.e., redundant expressions can be eliminated, so that the resulting SQL query is also, minimal. Techniques exist for query minimization, but general algorithms are NP-complete. Heuristic algorithms are projected to be effective for RXL queries, because RXL's nested block structure can help identify those expressions that most likely are redundant. XML-QL and SilkRoute can be implemented in Java. SilkRoute has drivers for Oracle and MySQL database servers.