Method and system for improved access to non-relational databases

[0002] The present invention relates to computer systems, and more particularly to the access of databases in the computer system.

[0003] Databases are an important tool for the storage and management of information for businesses. Both relational and non-relational databases exist for this purpose. Examples of relational databases include ORACLE, DB2, AND INFORMIX. Examples of non-relational databases include custom databases created with the 4690 operating system, developed by INTERNATIONAL BUSINESS MACHINES CORPORATION. The 4690 operating system allows programmers to created custom databases which support keyed, sequential, and binary file types. Database Management Systems (DBMS) provide users the capabilities of controlling read/write access, specifying report generation, and analyzing usage.

[0004] Some businesses, particularly large organizations, maintain multiple database types, both relational and non-relational. The simultaneous access to these databases then becomes an issue. For example, a company may wish to combine and compare customer preference information in the marketing department's ORACLE database with the current customer issues tracked in the customer service department's DB2 database. One conventional way of accessing these databases is through the DB2 DATAJOINER (“DataJoiner”) products, developed by INTERNATIONAL BUSINESS MACHINES CORPORATION. DataJoiner is a multi-database server which uses the Structured Query Language (SQL) and Open Database Connectivity (ODBC), to provide client access to diverse data sources that reside on different platforms.

[0005] FIG. 1 illustrates a computer system which uses DataJoiner. The computer system 100 comprises clients 102 in various platforms, a DataJoiner server 104, and data sources 106 of various types. The Data Joiner server 104 allows the clients 102 to transparently access data from the different data sources 106.

[0006] FIG. 2 is a block diagram illustrating the basic architecture of DataJoiner. DataJoiner 202 is a DBMS which includes ODBC software and drivers for a variety of data sources. DataJoiner 202 allows for transparent access to multiple data sources in a heterogeneous environment. The Generic Access Applications Programming Interface (API) 204 allows for customized access to home-grown or other data sources 208 not already supported by DataJoiner. Generic Data Access Modules 206 containing Custom ODBC compliant drivers for these data sources may be written and installed on the same system as DataJoiner 202 and used by DataJoiner 202 to access the data sources 208. The Generic Access API 204 defines the calls which must be provided by the custom ODBC drivers in order for DataJoiner 202 to access the custom data source 208.

[0007] However, DataJoiner is not able to provide certain features for access to the non-relational databases. In particular, DataJoiner 202 does not provide (1) a two-phase commit, (2) performance of SQL operations from within a single application across heterogeneous file systems, (3) referential integrity, and (4) indices for queries across heterogeneous file system which include non-relational data sources, such as 4690 keyed, sequential, and binary file systems.

[0008] Two-Phase Commit

[0009] DataJoiner can perform data changes across multiple databases sources within a single operation, while guaranteeing that changes to one source can be rolled back if changes to another source within the same transaction fail. The two-phase commit is the mechanism for implementing this. For example, someone may be trying to wire some money electronically to another location. A debit would need to be written against one account maintained by a first DBMS, and a credit would have to be reflected in another account maintained by a second DBMS. The two-phase commit protocol ensures that either all of the DBMS involved commit the changes required by the transaction or none do. However, keyed, sequential, and binary file systems cannot participate in this function.

[0010] SQL Operations from within a Single Application

[0011] Conventional relational DBMS's employ two underlying data storage concepts in order to facilitate relational operations on that data: (1) data is stored in tables with each table having a fixed number of columns; and (2) system catalogs, or data dictionaries, are generated for each table. These data dictionaries (metadata) describe the column layout, index structure, and other data access information. These concepts allow one application to create data, and other applications totally unrelated to the first, to analyze and update the data.

[0012] However, with keyed, sequential, and binary file systems, there are several inhibitors that prevent these file systems from having the above flexibility. These include: (1) data is not held in a fixed table format; (2) data access is via a buffer that passes the entire record, with no public or obvious field delineation available; and (3) the structure, or field layout, of keyed, sequential, and binary records is not stored in a publicly available catalog. With these inhibitors, relational operations cannot be performed on the keyed, sequential, and binary file systems, making SQL operations from within a single application across heterogeneous file systems which include these file systems difficult or impossible.

[0013] Referential Integrity

[0014] Referential integrity is a technique where the consistency of data within a DBMS is ensured by referring to one piece of data during an operation on another piece of data. An example would be to refer to a department table to ensure that a valid department number is being used when a new employee is being added to the database. Referential integrity is incorporated in many of the conventional relational DBMS and is being enforced at the database level to ensure that all applications adhere to any constraints and thus ensure the integrity of the data.

[0015] However, the concept of referential integrity, and a file system mechanism to enforce it, does not exist for keyed, sequential, and binary file systems. Without this function, heterogeneous DBMS can not implement referential integrity between the relational and non-relational data. Using the above example, if the valid department list is in a keyed file system and the new employee data was to be stored in a relational database, there would be no way that the underlying data stores could enforce the referential integrity during the employee add operation.

[0016] Indices for Queries

[0017] A need often arises to access keyed and sequential data based on data selection criteria different from the key or record number traditionally used to access the data. Queries based on this type of selection criteria are referred to as ad-hoc queries. For instance, instead of retrieving an employee record by an employee number there may be a need to retrieve all of the employees who are employed in department “X”. The employee records are keyed by an employee identification number yet the selection criteria is based on the department number in which the employee is employed.

[0018] For relational databases, ad-hoc queries may be efficiently satisfied since indices of the data may be created for data elements other than the key or record number. Once created; these indices may be used as an alternative form of data access which provides a more efficient processing of the query.

[0019] However, to satisfy queries using conventional file access mechanisms available for hash-keyed or sequential files, every record in the file must be read and compared to the selection criteria since indices are not available for these files. This causes performance problems both in accessing the file and in creating the network traffic when the querying application resides on a machine other than the source data files.

[0020] Accordingly, there exists a need for a method and system for improved access to non-relational databases in a computer system. The method and system should increase the flexibility and efficiency of access to the databases. The present invention addresses such a need.

[0021] The method and system of the present invention provides improved access to databases in a computer system. The present invention includes a plurality of data sources, the data sources including at least one non-relational data source; a server coupled to the plurality of data sources, the server capable of providing client access to the plurality of data sources; and a store place data access layer (SPDAL) coupled to the server and the at least one non-relational data source, the SPDAL providing relational features to the at least one non-relational data source. In the preferred embodiment, the relational features provided include a two-phase commit process; a process for performing Structured Query Language (SQL) operations from within a single application; a referential integrity process; and a process for providing indices for non-relational data source files. These features allow the computer system to perform data changes across multiple database sources within a single operation; perform SQL operations from a single application across heterogeneous file systems; provide referential integrity of data in both relational and non-relational databases; and perform ad-hoc queries of data and other queries efficiently in non-relational databases. By adding these features, significant flexibility is added to the computer system.

[0022] FIG. 1 is a diagram illustrating a computer system which uses DataJoiner.

[0023] FIG. 2 is a block diagram illustrating the basic architecture of DataJoiner.

[0024] FIG. 3 is a block diagram illustrating a preferred embodiment of the architecture of the method and system in accordance with the present invention.

[0025] FIG. 4 is a block diagram illustrating the X/OPEN DTP model used in accordance with the present invention.

[0026] FIG. 5 is a flow chart illustrating the two-phase commit process provided by the method and system in accordance with the present invention.

[0027] FIG. 6 is a block diagram illustrating an example of a two-phase commit performed in accordance with the present invention.

[0028] FIG. 7 is a preferred embodiment of metadata tables provided for non-relational databases in accordance with the present invention.

[0029] FIG. 8 is a flow chart illustrating the referential integrity process provided by the method and system in accordance with the present invention.

[0030] FIG. 9 is a block diagram illustrating a preferred embodiment of the Index Manager in accordance with the present invention.

[0031] FIG. 10 is a flow chart illustrating the creation of indices by the Index Manager in accordance with the present invention.

[0032] FIG. 11 is a flow chart illustrating the use of the indices in satisfying a user query in accordance with the present invention.

[0033] The present invention relates to a method and system for providing improved access to non-relational databases in a computer system. The following description is presented to enable one of ordinary skill in the art to make and use the invention and is provided in the context of a patent application and its requirements. Various modifications to the preferred embodiment will be readily apparent to those skilled in the art and the generic principles herein may be applied to other embodiments. Thus, the present invention is not intended to be limited to the embodiment shown but is to be accorded the widest scope consistent with the principles and features described herein.

[0034] The present invention provides improved access to non-relational databases through an expansion upon the capabilities of DataJoiner. FIG. 3 is a block diagram illustrating a preferred embodiment of the architecture of the method and system of the present invention. The Store Place Data Access Layer (SPDAL) architecture 300 comprises an Open Database Connectivity Applications Programming Interface 302 (ODBC API) which functions with DataJoiner 304. The SPDAL ODBC API 302 works with the Generic Access API 306 of DataJoiner 304 to provide a single access point for heterogeneous data source manipulations. An ODBC Driver 308 may be created to function with the SPDAL ODBC API 302 to access keyed, sequential, and binary file types, such as a 4690 data source 316. The ODBC Driver 308 comprises an ODBC Driver Manager 310, a Generic Driver 312, and a SQL Engine 314. In the preferred embodiment, the ODBC Driver Manager 310 is an ODBC Driver Manager, known in the art.

[0035] SPDAL 300 improves access to non-relational databases by providing the following capabilities: (1) true two-phase commit, (2) performance of SQL operations from within a single application across heterogeneous file systems, (3) referential integrity, and (4) indices for queries.

[0036] Two-Phase Commit

[0037] A two-phase commit process for keyed, sequential, and binary file systems is provided by the preferred embodiment of the method and system of the present invention. The two-phase commit is provided with an X/OPEN Distributed Transaction Processing (DTP) model using “XA interface” in SPDAL 302. X/OPEN, and its XA interface, is a standard protocol for access to databases.

[0038] FIG. 4 is a block diagram illustrating the X/OPEN DTP model used in accordance with the present invention. The X/OPEN DTP model comprises three software components:

[0039] An application program 402 which defines transaction boundaries and specifies actions that constitute a transaction.

[0040] Resource managers 404 which provide access to shared resources.

[0041] A transaction manager 406 which assigns identifiers to transactions, monitors their progress and takes responsibility for transaction completion and for failure recovery.

[0042] In the preferred embodiment, the XA Interface 408 is implemented with the X/OPEN DTP model as an intercept layer in the Generic Driver 312. The intercept layer would act as a resource manager for each file system, i.e., it would assume the role of a data source. DataJoiner 304 would act as the transaction manager. This would make the keyed, sequential, or binary file system appear to DataJoiner 304 as a fully functional relational database implementing an XA interface 408. The XA Interface 408 implements XA calls which allow for interaction between the transaction manager/DataJoiner 304 and the data source 316. This two-phase commit protocol provided by the present invention can be used to manage data changes across multiple data base sources within a single operation.

[0043] FIG. 5 is a flow chart illustrating the two-phase commit process provided by the method and system in accordance with the present invention. For this illustration, assume that a first database is a non-relational database and a second database is a relational database and a request requires a two-phase commit between them. First, DataJoiner 304, as the transaction manager, sends a request to the non-relational database, via step 510, through intercept layer in the Generic Driver 312. The intercept layer implements the appropriate XA calls to facilitate interaction between DataJoiner 304 and the non-relational database. The non-relational database replies to the request and waits, via step 520. DataJoiner 304 also sends a request to the relational database, via step 530. The relational database replies to the request and waits, via step 540. If both the non-relational and relational databases commit to the request, via step 550, the request is performed, via step 560. Otherwise, the transaction fails, via step 570.

[0044] Examples of functions for each XA API call comprise the below:

[0045] xa_close: Frees any dedicated two-phase commit resources.

[0046] xa_commit: Unlock the section of the file previously locked, this will make the data changes available to other applications.

[0047] xa_complete: Used by the transaction manager to wait for asynchronous operations. Return control to the transaction manager when the operation identified by the handle is complete.

[0048] xa_end: Free any dedicated resources acquired during xa_start processing.

[0049] xa_forget: Release resources associated with a xid.

[0050] xa_open: Initialize any dedicated two-phase commit resources.

[0051] xa_prepare: Obtain a write lock on the portion of the keyed, sequential, or binary file that will be changed. By acquiring a write lock, the resource manager intercept layer can guarantee that the subsequent write and commit will occur. The prepared write instruction (location in the file and buffer to write) should be saved in case a rollback is required.

[0052] xa_recover: Return to the transaction manager any prepared but not committed transactions.

[0053] xa_rollback: Undo the write operation staged with xa_prepare.

[0054] xa_start: Initialize any dedicated resources required for a transaction.

[0055] FIG. 6 illustrates an example of a two-phase commit performed in accordance with the present invention. In this example, assume that a request is received by DataJoiner to transfer $1000 from a first account at a first bank to a second account in a second bank. The first bank maintains its accounts in a non-relational database 602 (DS1), such as a 4690 data source. The second bank maintains its accounts in a relational database 604 (DS2), such as a DB2 data source. The banks do not want to perform the credit without also performing the debit and vice versa. In completing this transaction, first the transaction manager, DataJoiner 304, sends a request to the 4690 Keyed File, DS1602, to debit $1000 from the first account, via step 510. The request is sent through the Generic Access API 306 to the Intercept Layer in the Generic Driver 312. The Intercept Layer implements the appropriate XA calls to facilitate the portion of the transaction involving DS1602. DS1602 replies to the XA calls and waits, via step 520. DS1's 602 reply is sent to the transaction manager by the Intercept Layer. The transaction manager also sends a request to DS2604 to credit the second account $1000, via step 530. DS2604 replies and waits, via step 540. This ends phase one of the two-phase commit process.

[0056] If either DS1602 or DS2604 indicates that they are unable to make the requested credit/debit, i.e., if either data source fails to commit, via step 550, the transaction fails, via step 570, and the transaction manager moves on to the next transaction. If both DS1602 and DS2604 indicate they are able to make the requested credit/debit, i.e., both data sources commit, via step 550, then the credit and debit are performed, via step 560. This ends phase two of the two-phase commit process.

[0057] In the above manner, a two-phase commit process is provided by the method and system of the present invention.

[0058] SQL Operations from within a Single Application

[0059] With the preferred embodiment of the method and system of the present invention, the ability to perform SQL operations from within a single application is provided. The present invention implements this capability by providing additional metadata tables in a database in addition to the metadata tables of DataJoiner 304. These additional metadata tables describe the keyed, sequential, and binary file content such that the files within those file systems can be accessed by the SPDAL ODBC API 302 of the present invention, and manipulate it using relational operations. In addition, the metadata content itself is described and held as a relational table, so that this data is publicly available to all users and applications, and can be used in conjunction with DataJoiner 304 to perform single relational operations across relational data sources and keyed, sequential, and binary file data concurrently.

[0060] The metadata information of the present invention is used by DataJoiner 304 via the SPDAL ODBC API 302 to service requests from applications, thus removing the burden of individual applications to code to the specifics of the data's physical design and location.

[0061] A summary of the metadata tables of the present invention follows. A more detailed description of fields in the metadata tables are provided in FIG. 7:

[0062] LogicalEntity Table: maps a LogicalEntityName to one or more physical entities. (See Table 1, FIG. 7)

[0063] PhysicalEntity Table: describes metadata information that is defined for each relational database table or keyed, sequential, and binary file. (See Table 2, FIG. 7)

[0064] RecordDescription Table: describes the metadata information for records that exist within keyed, sequential, or binary files. (See Table 3, FIG. 7)

[0065] Attribute Table: describes the metadata information for all fields in a keyed, sequential, binary file. The definition of a field provides the physical layout structure that an application would use to process the data within the field. (See Table 4, FIG. 7)

[0066] ValidAttributeValuesTable: describes the discrete values that are valid within an attribute. (See Table 5, FIG. 7)

[0067] ForeignKey Table: describes the metadata information for parent-child relationships. These relationships are used to enforce referential integrity across all SPDAL data, as will be described in more detail later in this specification. This includes relational databases as well as non-relational databases. Relationships can be established between attributes within the same file system (for example between two relational tables) or between attributes across different file. systems (for example between a relational table and a keyed file). (See Table 6, FIG. 7)

[0068] Index Table: describe, set, and maintain indexes on keyed, sequential, binary files. The use of this table will be described in more detail later in this specification. (See Table 7, FIG. 7)

[0069] Although the present invention is described with the metadata tables as illustrated in FIG. 7, one of ordinary skill in the art will understand that other access mechanisms for keyed, binary, and sequential file types may be used without. departing from the spirit and scope of the present invention.

[0070] When DataJoiner 304 receives a SQL query which involves both a relational and a non-relational data source, Datajoiner 304 “splits” the query. It passes “as is” to the SPDAL ODBC API 302 the portion of the query which deals with the non-relational data source 316, and it passes the other portion to the relational data source through its native ODBC driver. The SQL engine 314 then interprets the SQL query into the low level calls for access to the non-relational data source 316. The requested data is returned in response to these low level calls. This result is then returned to DataJoiner 304, which combines it with the result returned from the relational data source. DataJoiner 304 then returns the combined result to the SQL query.

[0071] By providing the metadata in accordance with the present invention, advanced relational concepts, such as referential integrity, can be used across relational database management systems, and keyed, sequential, and binary file systems in the same operation. In addition, the metadata content structure is described as standard relational tables, making the metadata publicly available to any user or application. Finally, the metadata allows for a logical view for access to the underlying keyed or sequential file data. This enables an application to code to a logical model without having to be concerned with the underlying physical data store, or changes to the underlying physical data store.

[0072] Referential Integrity

[0073] The method and system of the present invention provides for referential integrity across both relational and non-relational databases. In the preferred embodiment, referential integrity is implemented in the SPDAL ODBC API 302 by combining the metadata described above with a relational database engine that can process the results of relational operations from outside data sources to ensure referential integrity across both the relational database and keyed, sequential, and binary file systems. The relational database engine is a SQL engine 314 which has the ability to perform queries which implement referential integrity, as described below.

[0074] The metadata is used to describe the referential constraints within the keyed, and binary file systems, and between these file systems and the relational file systems. The metadata is implemented by providing the following information in its ForeignKey Table:

[0075] Parent/child relationships between files

[0076] Primary key fields

[0077] Cardinality between the parent and child

[0078] List of fields that comprise a foreign key

[0079] Delete rules

[0080] Table 6 in FIG. 7 provides in more detail the above listed information.

[0081] FIG. 8 is a flowchart illustrating the referential integrity process provided by the method and system of the present invention. In the preferred embodiment, the database access is through ODBC or X/OPEN CLI interfaces. Both are widely accepted and implemented by conventional competitive relational database management systems. First, the SPDAL ODBC API 302 captures an end user query before it is passed to DataJoiner 304, via step 810. The query is preprocessed in the SPDAL ODBC API 302 based on information in the metadata tables, via step 820, specifically, the information in the ForeignKey Table. In the preprocessing, the SPDAL ODBC API 302 determines if the end user query requires any referential integrity processing, via step 822. Is so, the SPDAL ODBC API 302 would formulate the queries to execute and enforce the referential constraints, via step 824. Then, the end user query is passed to DataJoiner 304 for processing, via step 840. DataJoiner 304 determines if the query requires non-relational data, via step 850. If not, DataJoiner processes the query for the relational data, via step 860, and returns the result, via step 870. If so, then the query is processed on the non-relational data source, via step 880. In this processing, the queries which execute and enforce the referential contraint is passed to the SQL Engine 314, via step 826. The SQL Engine 314 translates these queries to commands compatible with the non-relational data source 316, via step 828. The SQL Engine 314 next executes these commands, via step 830. DataJoiner also processes the query for the relational data, via step 860. The results from both the relational and non-relational data sources are returned, via step 870.

[0082] The following is a summary of ODBC calls which may be made by the ODBC API 302 to implement referential constraints in accordance with the method and system of the present invention:

[0083] DELETE: Use the delete rules specified in the metadata to delete restrict, set null, or cascade. The ODBC driver of the present invention may need to make several calls into the relational DBMS in order to complete the requested action.

[0084] INSERT: Issue a SELECT ODBC call on the parent table to ensure that each row conforms to the referential constraint prior to passing the INSERT to the original relational DBMS ODBC driver. If the constraint does not pass, then the ODBC driver of the present invention will generate an error message and return control directly to the application program. From the application's viewpoint, it looks as if the underlying relational DBMS detected the referential constraint violation and returned the error.

[0085] UPDATE: Issue a SELECT ODBC call on the parent table to ensure that each row conforms to the referential constraint prior to passing the UPDATE to the original relational DBMS ODBC driver. If the constraint does not pass, then the ODBC driver of the present invention will generate an error message and return control directly to the application program. From the application's viewpoint, it looks as if the underlying relational DBMS detected the referential constraint violation and returned the error.

[0086] In this manner, referential integrity is provided within a keyed, sequential, or binary file system in accordance with the method and system of the present invention. The present invention provides dynamic real-time enforcement of the referential integrity constraints, at the data store level, across both relational databases and keyed, sequential, and binary file systems.

[0087] Indices for Queries

[0088] The method and system of the present invention provides a more efficient performance of queries, including ad-hoc queries, for non-relational databases. The efficiency is increased through the creation of an Index Manager which allows for the creation of indices for hash-keyed and sequential files. The indices are created based on the data selection criteria commonly used to satisfy user-defined ad-hoc queries. The user decides, based on the frequency that the query will be issued, along with the required response time to obtain the result set, which indices should be created.

[0089] As illustrated in FIG. 9, the Index Manager 902 is located next to the SQL Engine 314. The Index Manager 902 is accessed by the SQL Engine 314 during the satisfying of an end user query. In the preferred embodiment, the Index Manager comprises the following:

[0090] Index Manager API 904: allows users to maintain their indices and access the data contained with in the indices. This interface is available for use by external application programs that require access to the indices.

[0091] Index Manager Hash-keyed and Sequential File Interface 906: provides a means of updating indices to reflect additions, updates, and deletions of data from indices when the corresponding source data files change. This interface is context sensitive in that it depends on the file maintenance mechanism available on the data source.

[0092] Index Manager B-Tree Function 908: index entries are maintained in a B-Tree structure within the Index Manager to maintain the index entries. B-Tree structures are well known in the art and will not be discussed further here.

[0093] Index Manager Support Functions 910: include error logging and audit logging facilities, as well as a means of obtaining information about the data being indexed, i.e., the meta data.

[0094] FIG. 10 is a flowchart illustrating the creation of indices by the Index Manager 802 in accordance with the present invention. First, hash-keyed and/or sequential files are created, via step 1010. Next, the Index Manager API .904 is used to specify an index or indices to be created on a field or fields within a file, via step 1020. The Index Manager 902 then creates an index on the specified field or fields, via step 1030. This index is kept up to date when changes are occurring in the source data files via the Index Manager Hash-keyed and Sequential File Interface 906.

[0095] FIG. 11 is a flowchart illustrating the use of the indices in satisfying a user query in accordance with the present invention. When the user executes a query, via step 1110, the query is sent to the SQL Engine 314, via step 1120. The SQL Engine 314 then accesses the Index Manager 902 via the Index Manager API 904 to see whether an index exists that helps satisfy the query, via step 1130. If so, the Index Manager API 904 makes function calls to retrieve the indexed data, via step 1140. If not, then every record in the file needs to be read and compared to the selection criteria, via step 1150. Once the index data is retrieved, the Index Manager 902 determines if the index alone satisfies the query, via step 1160. If it does, then the result of the query is returned to DataJoiner 304 via the SQL Engine 314, via step 1170. If not, the rows based on the keyed file keys or binary sequential file offsets found in the index are also retrieved via the SQL Engine 314, via step 1180. Then the result is returned, via step 1170.

[0096] With the Index Manager 902 of the present invention, indexing capabilities are provided to hash-keyed, sequential and binary files. The efficiency in the execution of other queries is also greatly improved. By allowing the query application processing to only retrieve data that is known based on indices to satisfy the user defined query, there will be less network traffic when the application issuing the query resides on a machine other than the one where the source data files are stored. In addition, the cost of disk space to store indices is lowered.

[0097] A method and system for improved access to non-relational databases in a computer system has been disclosed. The method and system provides capabilities for a two-phase commit, performance of SQL operations from within a single application across heterogeneous file systems, referential integrity, and indices for ad-hoc and other queries for non-relational databases. These capabilities allow the computer system to perform data changes across multiple database sources within a single operation; perform SQL operations from a single application across heterogeneous file systems; provide referential integrity of data in both relational and non-relational databases; and perform ad-hoc queries of data and other queries efficiently when some or all of the data is stored in non-relational databases. By adding these capabilities, significant flexibility is added to the computer system.

[0098] Although the present invention has been described in accordance with the embodiments shown, one of ordinary skill in the art will readily recognize that there could be variations to the embodiments and those variations would be within the spirit and scope of the present invention. Accordingly, many modifications may be made by one of ordinary skill in the art without departing from the spirit and scope of the appended claims.