Creation of pre-filters for more efficient X-path processing

[0001] The present invention generally relates to computer implemented database management systems, and particularly to systems and methods for more efficient XPath/XQuery processing.

[0002] The extensible markup language (XML) is a flexible tag-based markup language suitably used to store data for posting on the Internet or local intranets, wide area networks, or the like. XML is increasingly being used as a native language for data storage in database management systems. In these and other XML applications, it is advantageous to have a flexible XML query language for creating and modifying XML documents, for efficiently and selectively retrieving data from XML documents or collections of XML documents, for sorting data, for inputting data into XML documents, and for otherwise manipulating XML items and data. Various XML query languages have been developed, including XML-QL, XQL, Quilt, and XQuery.

[0003] XML employs markup tags to classify, group, interrelate, or otherwise provide metadata regarding data stored in XML documents. An XML query can be viewed as producing streams of sequences of items. In a tabular notation using one column, each separately processed sequence comprising an XML item or a concatenation of XML items is suitably viewed as a row, while each XML item within a row is suitably viewed as an XML item, fragment, or row element. Such terminology is used herein to provide a convenient tabular visualization of the data; however, the use of such terminology does not imply that the data is stored or processed in tabular format.

[0004] In typical query processing, an XML query is constructed by a user, a software module, or the like, and is converted from a textual format to a data flow model. At the data flow model level, query rewrites of identified inefficient data flow structures are performed to optimize the data flow model. A query rewrite is a query transformation that produces a more efficient query without changing the query output. The optimized data flow model is compiled into executable instructions. Optionally, query rewriting is performed at a lower level than the data flow model, such as at the executable instructions level. It will be appreciated that the executable instructions substantially correspond to a data flow model written in a specific executable instruction set. Similarly, the query text substantially corresponds to a data flow model written in a high level text-based language, and so optionally query rewrites are performed at the text query level. Typically, however, the XML query is converted into an intermediate data flow model which formats the XML query in a manner well-suited for efficient XML query rewrite processing.

[0005] Regardless of the processing level at which query rewrites are performed, the query rewrites perform various optimization tasks such as reducing XML item sequence construction, reducing usage of memory for temporary data storage, promoting data flow pipelining, improving index usage and improving I/O behavior. For XQuery, heuristics may be applied, such as “try to express the whole query with as few FLOWR expressions as possible” or “apply filters and extractions early during data processing” in the form of rewrite rules (or rewrites). However, there are problems which arise when queries are rewritten in order to evaluate XPath expressions as early as possible in order to filter out unwanted items in a more efficient way.

[0006] Some of these problems are associated with a query rewrite technique known as extraction push down of Xpath in which an XPath extraction is moved/pushed into a lower (closer to base table) query block without changing the semantics of the query, but possibly changing the result of the lower query block. One example is known as the multiple consumer problem, which occurs when an extraction pushdown cannot be performed since other parts of the same query refer to the result of the lower query block that it is being pushed into.

[0007] Accordingly, there is a need for systems and methods for increasing the efficiency of the processing of XML queries. There is also a need for systems and methods for rewriting queries to filter out unwanted items more efficiently while avoiding issues such as the multiple-consumer problem.

[0008] To overcome the limitations in the prior art briefly described above, the present invention provides a method, computer program product, and system for querying an XML document.

[0009] In one embodiment of the present invention an XML query rewrite method comprises identifying an XPath within an XML query; and transforming the XPath into a pre-filter of the XML query.

[0010] In another embodiment of the present invention, an XML query rewrite method for processing an XML query comprises: identifying an XPath within the XML query; transforming the XPath into a pre-filter; executing the pre-filter; and executing the XML query after the execution of the pre-filter.

[0011] In a further embodiment of the present invention an XML compilation processor comprises: an execution compiler transforming an XML query into an executable XML query plan; a query rewrite processor performing query transformation on the XML query, the query transformations including transforming an XPath within the XML Query into a pre-filter; and transforming the XML query into a transformed XML query including the pre-filter.

[0012] In an additional embodiment of the present invention comprises an article of manufacture for use in a computer system tangibly embodying computer instructions executable by the computer system to perform process steps for querying an XML document, the process steps comprising: identifying an XPath within the XML query; transforming the XPath into a pre-filter; executing the pre-filter; and executing the XML query after the execution of the pre-filter.

[0013] Various advantages and features of novelty, which characterize the present invention, are pointed out with particularity in the claims annexed hereto and form a part hereof. However, for a better understanding of the invention and its advantages, reference should be made to the accompanying descriptive matter together with the corresponding drawings which form a further part hereof, in which there is described and illustrated specific examples in accordance with the present invention.

[0018] The present invention overcomes the problems associated with the prior art by teaching a system, computer program product, and method for efficient processing of XML queries using pre-filters. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. Those skilled in the art will recognize, however, that the teachings contained herein may be applied to other embodiments and that the present invention may be practiced apart from these specific details. Accordingly, the present invention should not be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features described and claimed herein. The following description is presented to enable one of ordinary skill in the art to make and use the present invention and is provided in the context of a patent application and its requirements.

[0019] The present invention provides a system and method of using pre-filters in an XQuery processor. The invention addresses problems which arise when queries are rewritten in order to evaluate XPath expressions as early as possible in order to filter out unwanted items in a more efficient way. The invention does this by executing a pre-filter before the execution of the original XPath expression. The execution of the pre-filter removes unwanted data early on. As a result, other costly operations are performed on a reduced set of data and run-time performance may be significantly improved. Also, the techniques of the present invention may improve index usage.

[0020] FIG. 1 shows a block diagram of an XQuery processor 10 in accordance with one embodiment of the invention. A query is performed on a data stream of items of an XML data type produced by an extensible markup language (XML) source. The data stream of XML items may be extracted from an Internet service 11, from one or more XML documents 12, or from another source of XML items. Although XML documents 12 are shown in FIG. 1, the data stream processed by the query may be extracted from a document formatted in another tag-based markup language, such as a data stream of hypertext markup language (HTML) items extracted from one or more HTML documents. Similarly, the Internet service may supply a data stream of HTML items. A textual XML query 14 is constructed by a user, by an application program, or the like.

[0021] The textual XML query 14 is preferably constructed in accordance with XQuery, which is a commonly used XML query language. However, other XML query constructions can be employed in which XML documents are processed as data streams of sequences of XML fragments or XML items. In XQuery, for example, data streams of XML items and sequences of items are constructed using FLWOR expressions in which FOR clauses produce data streams of individual XML items, LET clauses produce concatenated sequences of XML items, combination of FOR and LET clauses cause a Cartesian product of the produced streams, WHERE clauses provide filtering based on predicate expressions, and the RETURN clause causes the computation of the result that is flowed to a subsequent operation. The FLWOR expression is used as an example herein; other XQuery expressions also can create LET and FOR data flow, henceforth called LET and FOR quantifiers. Example data flow stream transformations are described herein with reference to the preferred XQuery syntax; however, these exemplary data flow stream transformations are readily adapted to other XML query languages that may employ different syntax, clause names, or the like for producing data streams of single XML items, concatenated sequences of XML items, or combinations thereof.

[0022] The textual XML query 14 is processed by a compilation processor 20 to produce an executable XML query plan 22, which, when executed, accesses the Internet service 11 to receive, sort, search, transmit, or otherwise process data content from or to the Internet service 11, accesses the XML documents 12 to store, read, sort, search, or otherwise process data content of the XML documents 12, or otherwise interacts with a source or repository of XML items.

[0023] In a read operation, an XML items collection engine 24 formats a data stream of XML items extracted from the Internet service 11, the XML documents 12, or other source of XML items into a query data model or other suitable format that is received by the executing XML query plan 22. In a write operation, the XML items collection engine 24 formats a stream of XML items output from the executing XML query plan 22 into a format suitable for receipt by the Internet service 11, for storage in the XML documents 12, or for storage or receipt by another repository of XML items. Rather than storing or transmitting the output of the executing XML query plan 22, the output can be displayed on a display 26 or otherwise communicated to a user.

[0024] More specifically, the compilation processor 20 includes various components for transforming and optimizing the textual XML query 14 to produce the executable XML query plan 22. In the exemplary illustrated embodiment, a textual XML query parser 30 parses the textual XML query 14 to produce a parsed XML query from which a data flow modeler 32 produces an equivalent data flow model. The data flow model format is convenient for performing query rewrite and optimization of the XML query prior to conversion to executable code.

[0025] The data flow model is analyzed by a query rewrite processor 34 which employs heuristic, cost-based, or other methods to identify query rewrites that improve or are likely to improve efficiency of the XML query. The query rewrite processor 34 performs identified query rewrites if it can determine that the rewrite is safe, that is, if it can determine the making the rewrite will not alter the output of the query.

[0026] In accordance with embodiments of the invention, a pre-filter 35 determines an expression that will act as a pre-filter for the XML Query. The function of the pre-filter is based on the consideration that, given almost any XPath inside of XQuery, one can determine an expression which will act as a pre-filter. By annotating the original query with this pre-filter, the semantic remains unchanged. Given some of the unique properties of the pre-filter 35, the query rewrite processor 34 can apply certain rewrites in a much more powerful way. This enables better index usage and better execution behavior.

[0027] The concept of the pre-filter 25 begins with an observation in XPath:
$i[e1]/e2==$i[e1][e2]/e2  (1)
where e1 is a filter and e2 is a deterministic XQuery expression. Note that e1 is not required to be present leading to $i/e2==$i[e2]/e2).

[0028] Now we can express $i[e1]/e2 as:
$i[e1]/e2===>
1. for $imp in $i[e1][e2]return $imp/e2, if $i is provably singleton or
2. let $return :=$i[e1]/e2 where fn:exists($i[e1][e2])], otherwise.  (2)
Hence, this gives us a pre-filter $i[e1][e2] which can be optionally applied. Also, note that
fn:exists($i[e1]/e2)===>fn:exists($i[e1][e2])  (3)
is a correct rewrite using just the pre-filter and dropping the original expression. The significance of expressions I, 2 and 3 is that the pre-filter is an XPath of the form $i[filter] which enables a far more aggressive push down behavior since the pre-filter will only cause $i to be discarded. In other words, it does not change the items being flowed.

[0029] Consider the following example (omitting other rewrites for simplicity):

[0030]

	for $i in for $j in T/a return $j
	for $j in $i/b[c = 10]
	for $k in $i/d[e > 5]
	return $k

Note that we cannot push $i/b[c=10]nor $i/d[e>5] into the nested FLOWR since both of the two would change the data flow the other expression is dependent upon. However, using a pre-filter as described above one can rewrite the query into (with subsequent other rewrites applied):

[0031]

	for $i in for $j in T/a[b[c = 10] and d[e > 5]] return $j
	for $j in $i/b[c = 10]
	for $k in $i/d[e > 5]
	return $k

thus enabling potential index usage. Also, other rewrites might also benefit from this.

[0032] Expression 3 above allows us to rewrite fn:exists queries and XMLEXISTS queries into the form of fn:exists($i[fn:exists([e1]/e2)]) which allows a much better early-out behavior since we do not have to construct any items dictated by e2.

[0033] In another example consider the XML Query:
$i/person/phone[ac=408]==
$i[person/phone[ac=408]]/person/phone[ac=408]  (4)
In expression (4) by using the pre-filter
person/phone[ac=408]
all phone numbers not having 408 as the area code are filtered out. This may be done using an index. Without the pre-filter all area codes would have to be navigated. By removing the unwanted area codes the performance is significantly improved. The performance improvement can be significant because, between the execution of the pre-filter and the original expression, there may be other costly operations which benefit from a reduction of data being flowed early on. In addition, this is accomplished without raising the multiple consumer problem.

[0034] Referring again to FIG. 1, the query rewrite processor 34 will use the pre-filter 35 to create a pre-filter using the above-described methods. An index matcher 37 will also be used by the query rewrite processor 34 to make use of any appropriate indices that are available. Once the query rewrite processor 34 identifies a data stream that is indicated as being advantageously transformed, it examines the expression that receives the data stream to determine whether or not the conversion can be safely performed. This expression examination suitably may include analyzing the evaluation of the expression and its resulting data stream using an evaluation properties calculator 36 to compute a plurality of properties that indicate what type or types of quantifier transformations can be safely performed. That is, a determination is made as to whether the data stream transformation can be performed without possibly corrupting the XML query by changing the result of the evaluation of the expression under consideration. Also, the upper join operates under potentially significant less data, since rows not adhering to the filter conditions are discarded earlier during processing.

[0035] With continuing reference to FIG. 1, the query rewrite processor 34 performs one or more query rewrite operations on the data flow model representing the XML query. Preferably, the query rewrite processor 34 operates iteratively, in that after the data flow model is analyzed and one or more query rewrite operations are performed, the analysis of the data flow model is repeated to determine whether the previous query rewrite operations have produced a transformed data flow model in which additional query rewrites are indicated.

[0036] Once the query rewrite processor 34 determines that further query rewrite operations are not indicated, the data flow model, as modified by processing performed by the query rewrite processor 34, is input to a query optimizer 40. The query optimizer 40 performs optimizations based on a costing analysis of physical operators of the data flow model to provide greater XML query efficiency. An execution compiler 42 compiles the data flow model after optimization into the executable XML query plan 22 which is suitable for execution.

[0037] The components of the XML query compilation processor 20 are exemplary only. Those skilled in the art can readily modify the described XML query compilation processor 20 to suit specific XML query syntaxes or languages, specific applications, specific database management system configurations, and the like. For example, if the XML query is provided to the XML query compilation processor in a non-textual format rather than as the textual XML query 14, then the text parser 30 is suitably omitted.

[0038] In another contemplated embodiment, the data flow modeler 32 is omitted, in which case the query rewrite processing is suitably performed on the parsed XML query or on the executable XML query plan. Alternatively, a plurality of data flow modeling processors can be employed to implement a plurality of data flow model formats. For example, another data flow modeler is optionally inserted between the query rewrite processor 34 and the query optimizer 40 to optimize the data flow model format for processing by the query optimizer 40.

[0039] Still further, although the XML query rewrite processor 34 is shown in FIG. 1 as a separate component, it is contemplated to integrate XML query rewrite operations into the data flow modeler 32, to integrate the XML query rewrite processor 34 and the query optimizer 40 into a single component that performs both query rewrites and query optimization, or to otherwise modify the XML query compilation processor 20.

[0040] The compilation processor 20 is typically implemented using one or more computer programs, each of which executes under the control of an operating system, such as OS/2, Windows, DOS, AIX, UNIX, MVS, or so forth, and causes a computer (not shown) to perform the desired XML query compilation processing as described herein. Thus, using the present specification, the compilation processor 20 may be implemented as a machine, process, or article of manufacture by using standard programming and/or engineering techniques to produce software, firmware, hardware or any combination thereof.

[0041] FIG. 2 is a flow chart of a method for XQuery processing in accordance with an embodiment of the invention. In particular, a process 44 begins with the input of an XML query in step 46. The XPath in the XML is then identified in step 48. The XPath is then transformed into a pre-filter in step 50. The pre-filter is then executed as indicated by step 52. The XML query is then navigated in step 54 and an XML query result is then generated in step 56. By executing the pre-filter in step 52, before the original XML query, in step 54, unwanted data is removed early on. This may benefit other costly operations by reducing data being flowed early on. The result may be a significant run-time improvement.

[0042] The invention can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment containing both hardware and software elements. In a preferred embodiment, the invention is implemented in software, which includes but is not limited to firmware, resident software, microcode, etc.

[0043] Furthermore, the invention can take the form of a computer program product accessible from a computer-usable or computer-readable medium providing program code for use by or in connection with a computer or any instruction execution system. For the purposes of this description, a computer-usable or computer readable medium can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.

[0044] The medium can be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system (or apparatus or device) or a propagation medium. Examples of a computer-readable medium include a semiconductor or solid state memory, magnetic tape, a removable computer diskette, a random access memory (RAM), a read-only memory (ROM), a rigid magnetic disk and an optical disk. Current examples of optical disks include compact disk-read only memory (CD-ROM), compact disk-read/write (CD-R/W) and DVD.

[0045] A data processing system suitable for storing and/or executing program code will include at least one processor coupled directly or indirectly to memory elements through a system bus. The memory elements can include local memory employed during actual execution of the program code, bulk storage, and cache memories which provide temporary storage of at least some program code in order to reduce the number of times code must be retrieved from bulk storage during execution.

[0046] Input/output or I/O devices (including but not limited to keyboards, displays, pointing devices, etc.) can be coupled to the system either directly or through intervening I/O controllers.

[0047] Network adapters may also be coupled to the system to enable the data processing system to become coupled to other data processing systems or remote printers or storage devices through intervening private or public networks. Modems, cable modem and Ethernet cards are just a few of the currently available types of network adapters.

[0048] FIG. 3 is a high level block diagram showing an information processing system useful for implementing one embodiment of the present invention. The computer system includes one or more processors, such as processor 60. The processor 60 is connected to a communication infrastructure 62 (e.g., a communications bus, cross-over bar, or network). Various software embodiments are described in terms of this exemplary computer system. After reading this description, it will become apparent to a person of ordinary skill in the relevant art(s) how to implement the invention using other computer systems and/or computer architectures.

[0049] The computer system can include a display interface 64 that forwards graphics, text, and other data from the communication infrastructure 62 (or from a frame buffer not shown) for display on the display unit 66. The computer system also includes a main memory 68, preferably random access memory (RAM), and may also include a secondary memory 70. The secondary memory 70 may include, for example, a hard disk drive 72 and/or a removable storage drive 74, representing a floppy disk drive, a magnetic tape drive, an optical disk drive, etc. The removable storage drive 74 reads from and/or writes to a removable storage unit 76 in a manner well known to those having ordinary skill in the art. Removable storage unit 76, represents a floppy disk, a compact disc, magnetic tape, optical disk, etc. which is read by and written to by removable storage drive 74. As will be appreciated, the removable storage unit 76 includes a computer readable medium having stored therein computer software and/or data.

[0050] In alternative embodiments, the secondary memory 70 may include other similar means for allowing computer programs or other instructions to be loaded into the computer system. Such means may include, for example, a removable storage unit 78 and an interface 80. Examples of such may include a program cartridge and cartridge interface (such as that found in video game devices), a removable memory chip (such as an EPROM, or PROM) and associated socket, and other removable storage units 78 and interfaces 80 which allow software and data to be transferred from the removable storage unit 722 to the computer system.

[0051] The computer system may also include a communications interface 82. Communications interface 82 allows software and data to be transferred between the computer system and external devices. Examples of communications interface 82 may include a modem, a network interface (such as an Ethernet card), a communications port, a PCMCIA slot and card, etc. Software and data transferred via communications interface 82 are in the form of signals which may be, for example, electronic, electromagnetic, optical, or other signals capable of being received by communications interface 82. These signals are provided to communications interface 82 via a communications path (i.e., channel) 84. This channel 84 carries signals and may be implemented using wire or cable, fiber optics, a phone line, a cellular phone link, an RF link, and/or other communications channels.

[0052] In this document, the terms “computer program medium,” “computer usable medium,” and “computer readable medium” are used to generally refer to media such as main memory 68 and secondary memory 70, removable storage drive 74 and a hard disk installed in hard disk drive 72. These computer program products are means for providing software to the computer system. The computer readable medium allows the computer system to read data, instructions, messages or message packets, and other computer readable information from the computer readable medium. The computer readable medium, for example, may include non-volatile memory, such as a floppy disk, ROM, flash memory, disk drive memory, a CD-ROM, and other permanent storage. It is useful, for example, for transporting information, such as data and computer instructions, between computer systems. Furthermore, the computer readable medium may comprise computer readable information in a transitory state medium such as a network link and/or a network interface, including a wired network or a wireless network, which allows a computer to read such computer readable information.

[0053] Computer programs (also called computer control logic) are stored in main memory 68 and/or secondary memory 70. Computer programs may also be received via communications interface 82. Such computer programs, when executed, enable the computer system to perform the features of the present invention as discussed herein. In particular, the computer programs, when executed, enable the processor 60 to perform the features of the computer system. Accordingly, such computer programs represent controllers of the computer system.

[0054] In accordance with the present invention, we have disclosed systems and methods for creating pre-filters to achieve more efficient Query processing. Those of ordinary skill in the art will appreciate that the teachings contained herein can be implemented using many kinds of software and operating systems, including, but not limited to, XML-enabled database systems such as relational databases and native XML databases as well as middleware such as query gateways, federation, and information integration. References in the claims to an element in the singular is not intended to mean “one and only” unless explicitly so stated, but rather “one or more.” All structural and functional equivalents to the elements of the above-described exemplary embodiment that are currently known or later come to be known to those of ordinary skill in the art are intended to be encompassed by the present claims. No claim element herein is to be construed under the provisions of 35 U.S.C. section 112, sixth paragraph, unless the element is expressly recited using the phrase “means for” or “step for.”

[0055] While the preferred embodiments of the present invention have been described in detail, it will be understood that modifications and adaptations to the embodiments shown may occur to one of ordinary skill in the art without departing from the scope of the present invention as set forth in the following claims. Thus, the scope of this invention is to be construed according to the appended claims and not limited by the specific details disclosed in the exemplary embodiments.