System and method for data integration using multi-dimensional, associative unique identifiers

[0001] This application claims the benefit of U.S. Provisional Application No. 60/486,780, entitled METHOD FOR CREATING MULTIDIMENSIONAL GLOBALLY UNIQUE IDENTIFIERS, and filed on Jul. 11, 2003. U.S. Provisional Application No. 60/486,780 is incorporated by reference herein.

[0002] In general, the present application relates to computer software and data processing, and in particular, to a system and method for identifying data using multi-dimensional, associative unique identifiers.

[0003] Generally described, large scale enterprise computer systems can include a number of data objects that are stored in various databases, software applications, and/or other data stores. In one aspect, the data objects can correspond to encapsulations of data relating to one or more business applications, which are generally referred to as business objects. Business objects can be in the form of structured or unstructured data relating to physical objects, such as equipment, personnel, etc. Additionally, business objects can also relate to abstract objects, such as database entities, textual descriptions of concepts, meta data, etc.

[0004] Generally described, business objects can be identified in accordance with conventional data processing approaches with an independently assigned domain specific unique identifier, such as a database key constraint. One skilled in the relevant art will appreciate that domain specific unique identifiers are not dependent on the specific business object data they represent. Instead, domain specific unique identifiers are limited to representing specific business object data and the identifier cannot be recreated for each instance of data representing a business object. Accordingly, a particular instance of a business object, such as a representation of an individual, may be associated with a number of unique identifiers. As the number of data sources containing business object data increases, such as independent databases, traditional data processing approaches become deficient in associating all business object data from each independent source.

[0005] Thus, there is a need for a system and method for uniquely representing business objects that is dependent on specific business object attributes and that is invariant with specific values of business object attribute data.

[0006] A system and method for associating data objects utilizing unique identifiers is provided. Data objects are modeled utilizing a data object ontology. Unique identifiers for instances of each data object are calculated utilizing a selection of unique attributes of the data object ontology. Data objects from multiple data sources can be integrated utilizing the unique identifiers for each data object.

[0007] In accordance with an aspect of the present invention, a method for processing data is provided. A data integration application associates a data object with a data object ontology corresponding to a hierarchy of data object dimensions. Additionally, the data integration application calculates a unique identifier for each instance of data object based upon a selection of a subset of data object dimensions.

[0008] In accordance with another aspect of the present invention, a method for processing data is provided. A data integration application associates a data object with a data object ontology corresponding to a hierarchy of data object dimensions. The data object ontology includes a invariance strength identifier for a source and target dimension in the hierarchy of data object dimensions. The data integration application also calculates a unique identifier for each instance of data object based upon a selection of a subset of data object dimensions having a high affinity metric.

[0009] In accordance with a further aspect of the present invention, a system for processing data is provided. The system includes a number of data sources including data object dimension data corresponding to instances of data objects. The system also includes a data integration application for obtaining the data object dimension data. The data integration application integrates the data object dimension data according to a unique identifier calculated from a hash of a selection of a subset of data object dimension data.

[0017] Generally described, the present invention relates to a system and method for representing data. More specifically, in one aspect, the present invention relates to a system and method for representing data objects based upon unique identifiers corresponding to specific data object attribute values. In another aspect, the present invention relates to a system and method for identifying and integrating data from multiple data sources utilizing unique identifiers.

[0018] In an illustrative embodiment of the present invention, the data objects can correspond to physical objects in a defined domain, such as physical items associated with an enterprise. Additionally, the data objects can correspond to abstract objects, such as meta data, within the same domain. Although the present invention will be described in relation to data objects and data object attributes associated within a business context (e.g., business objects), one skilled in the relevant art will appreciate that the disclosed embodiments are illustrative in nature and should not be construed as limiting.

[0019] FIG. 1 is a block diagram of illustrative of a business object ontology 100 for generating a unique identifier for a business object. In an illustrative embodiment of the present invention, the business object ontology 100 corresponds to a hierarchy of dimensions can be representative of a particular business object concept. The values associated with each dimension for a particular instance of the ontology is then utilized to generate a unique identifier, as will be described in greater detail below. Based on the selection of specific attributes for the business object ontology 100, attribute data from any number of data sources should generate a similar unique identifier. Thus, the incoming data can be associated with the particular business object.

[0020] In an illustrative embodiment of the present invention, the business object ontology 100 corresponds to a particular business object concept. For example, the business object concept can be a representative of an aspect of physical items within an enterprise, such as a customer, a piece of equipment/machinery. A particular business object can be represented by a single business object ontology, such as a business ontology for representing each piece of machinery in a plant. Alternatively, business objects can be represented by multiple business object ontologies corresponding to different aspects of the business object. For example, one business object ontology may correspond to a representation of a customer from a financial perspective while another business object may correspond to a representation of the customer from a physical presence perspective (e.g., geographic-based location). Still further, a business object ontology may consist of a collection of other defined business ontologies. With reference to the previous example, a more global customer business ontology can incorporate the financial business ontology and the physical presence ontology.

[0021] With reference to FIG. 1, the business object ontology 100 corresponds to a multi-level hierarchy of dimensions that are representative, or otherwise associated with, the business object. The business object ontology 100 includes a root dimension 102 that is representative of the business object concept represented by the business object ontology. The business object ontology 100 also includes one or more dimensions 104, 106, 108 and 110 that are dependent on the root dimension 102. The root dimension, e.g., dimension 102, is considered to be a source dimension, while the dependent dimensions 104, 106, 108 and 110 are considered to be target dimensions. The business ontology 100 can also include another level of dimensions 112, 114, 116 and 118 that are dependent on the dimensions 104 and 108 respectively. In this respect, dimensions 104 and 108 are source dimensions, while dimensions 112-118 are target dimensions. Although the business ontology 100 is shown with three levels, one skilled in the relevant art will appreciate that alternative business ontologies can have any number of levels. Further, one skilled in the relevant art will appreciate that a business ontology can have any number of dimensions at each level in the hierarchy.

[0022] With continued reference to FIG. 1, in an illustrative embodiment of the present invention, the business object ontology 100 can include an invariant strength identifier for each source to target dimension relationship. The invariant strength identifier corresponds to an estimated correlation between the source dimension and the target dimension. The invariant strength identifier can be specified as a value in a range from 0.00 to 1.00, where 1.00 indicates that the value of the source will be invariant if the value of the target is invariant. As illustrated in FIG. 1, each link between a source dimension and a target dimension includes an invariant strength identifier.

[0023] In an illustrative embodiment of the present invention, an invariant strength identifier can be calculated manually by identifying the one or more dimensions that are not likely to change for a particular business object, such as a birth date. Additionally, the invariant strength identifier may be calculated based upon a statistical analysis of an enterprise data. In this approach, the invariant strength can be calculated in a manner defined below:
temporal invariance of (dimension 1, object A)=num A/N  (1)

[0024] where

[0025] num A=the maximum number of times a value of the dimension for an object appears in a snapshot of all records over a period of time; and

[0026] N=total number of data records analyzed.
temporal invariance of (dimension 1, object)=min_i [temporal invariance of (dimension 1, object A)] for each object instance in the records.  (2)

[0027] In an illustrative embodiment of the present invention, each dimension in the business object ontology 100 can be associated with specific attribute data. The data can include a dimension type identifier, a dimension version identifier, a timestamp identifier for the unique id, and a value for the dimension. In an illustrative embodiment of the present invention, the dimension type identifier identifies which business object ontology and the particular dimension to which the value corresponds. The version information indicates a version for the business object ontology if more than one version of the ontology is present. The timestamp identifier can include one or more timestamps relating to when the business ontology was created, modified and/or when it will expire. The value for the dimension can be in the form of a textual string that represents a value for the dimension (e.g., textual string including the name of the information). One skilled in the relevant art will appreciate that the business ontology may include additional or alternative dimension data.

[0028] With reference now to FIG. 2, an illustrative business object ontology 200 for a customer business object will be described. The business object ontology 200 includes a root dimension 202, which in the illustrative example, corresponds to the representation of a customer in the define domain. The root dimension 202 includes three target dimensions 204, 206 and 208, which corresponds to an identification dimension, a financial dimension, and a location dimension, respectively. With continued reference to FIG. 2, dimension 204, name, has a number of additional target dimensions 210-212, such as legal name and nickname. Similarly, dimension 208, location, has a number of target dimensions 214-216, such as postal address and geographic location. As described above, each of the source/target dimensions in the business object ontology 200 includes an invariant strength identifier.

[0029] In accordance with an aspect of the present invention, at least a portion of the values of the dimensions in the business ontology can be utilized to calculate a unique identifier for data associated with a business object. FIG. 3 is a block diagram illustrative of the generation of a unique identifier from multiple object dimensions in accordance with the present invention. As illustrated in FIG. 3, a selection of values for three dimensions of a business ontology 100 (FIG. 1) can be used to generate a unique identifier 120 for the particular business object.

[0030] In accordance with the present invention, the dimensions selected to generate the unique identifier 120 are most likely to have a unique value for each instance of an object in a particular domain. Thus, no two instances of a business object will be associated with the same unique identifier. To select which dimensions should be include in the calculation of a unique identifier, an affinity metric is calculated for each dimension in the business object ontology. The affinity metric consists of a uniqueness metric and the temporal invariance for each dimension, which will be described below.

[0031] In an illustrative embodiment of the present invention, the uniqueness metric is a measure of the uniqueness of dimension values for all records to be identified in a given domain. The uniqueness metric can be determined as follows:
μ (affinity metric)=Σ (max |S|/N) for K records  (3)

[0032] where

[0033] S=subgroups that contain records having the same value for a particular dimension; and

[0034] N=number of groups of K records having the same value for a dimension.

[0035] Based upon the calculation of a uniqueness metric and the invariant strength of the source/target dimensions, the affinity metric for dimensions in a business ontology can be calculated as follows:
affinity metric=α*μ+β*temporal invariance  (4)

[0036] where

[0037] α,β are weighting factors and

[0038] 0<=α<=1;

[0039] 0<=β<=1

[0040] α+β=1.

[0041] In an illustrative embodiment of the present invention, a unique identifier for an instance of a business object is composed of a hash of a number of dimension values having the highest affinity metric values. In an illustrative embodiment of the present invention, the md5 hashing algorithm may be used to generate a unique representation of data. The md5 hashing algorithm is well known in the relevant art and will not be described in greater detail. However, one skilled in the relevant art will appreciate that alternative hashing algorithms may be utilized in accordance with the present invention. Equation 5 defines the calculation of the unique id as follows:
unique id=hash (values D₁−D_K)  (5)

[0042] where

[0043] D₁−D_K are dimension values having the highest K affinity metrics.

[0044] In accordance with another aspect of the present invention, the unique identifiers calculated as described above, may be utilized to integrate and process data from a plurality of sources. FIG. 4 is a block diagram illustrative of the integration of business object data from a plurality of data sources utilizing unique identifiers in accordance with the present invention. As illustrated in FIG. 4, a data integration application 402 collects data from a number of data sources 404, 406, 408. The data integration application 402 may be remote from one or more data sources 404, 406, 408. Alternatively, the data integration application 402 may be integrated as component within one or more the data sources 404, 406, 408. Each data source includes some type of business object data that is accessible by the data integration application 402. The business object data for each data source 404, 406, 408 may be of the same type of data or of different data types.

[0045] In an illustrative embodiment of the present invention, the data integration application 402 calculates a unique identifier for the incoming data based upon a selected business ontology 100 (FIG. 1) for the business object data and the selected affinity metrics, as described above. Because the unique identifier can typically be associated with only one instance of a business object (e.g., a specific piece of equipment), the data integration application 402 can then map all incoming data to a specific business object. Accordingly, the data integration application 402 can associate different types of business object data from a variety of data sources to the same instance of a business object.

[0046] FIGS. 5A-5C are block diagrams of a table 500 utilized to associated incoming business object data from a variety of data sources in accordance with the present invention. With reference to FIG. 5A, the table 500 includes a number of rows corresponding to each unique identifier in the defined domain of data sources. The table 500 also includes a number of columns corresponding to each unique data source that will be integrated. As a source is integrated, a mapping of a unique id, such as id 502 and 506, is correlated to the identification/location of the data in the data sources. As illustrated in FIG. 5A, identification/locations 504, and 508 are correlated to ids 502, 506, respectively.

[0047] With reference to FIG. 6, a routine 600 for integrating data from a variety of data sources utilizing unique ids will be described. At block 602, the next set of data to be integrated is obtained. In an illustrative embodiment of the present invention, the next set of data may be a sequential process from each data source. Alternatively, a prioritization may be utilized to select specific data to be integrated. At decision block 604, a test is conducted to determine whether a unique id exists for the data to be integrated. As described above, the data integration application 402 (FIG. 4) calculates a unique id based upon a selection of a subset of dimension data from a business object ontology. If a unique id does not exist in the map table 500 (FIG. 5A), the data integration application 402 creates an entry for the new unique id at block 606 and adds it to the table 500 at block 608. Once the entry has been added to the table at block 606 or if an entry already exists at decision block 604, at block 610, the data integration application adds the data to the table 500. At decision block 612, a test is conducted to determine whether additional data is to be integrated. If additional data is to be integrated, the routine 600 returns to block 602. Alternatively, if no additional data is to be integrated, the routine 600 terminates at block 614.

[0048] With reference now to FIG. 5B, data identification 510 has been added to table 500 to correspond data source 2 to unique id 502. Likewise, data identifications 512 and 514 have been added to table 500 to correspond data sources 2 and 3 to unique ids 502, 504. With reference to FIG. 5C, unique id 516 has been added to table 500 to account for a new unique id and data identification 518 has been added to the table 500 to corresponds data source 3 to unique id 516.

[0049] While illustrative embodiments of the invention have been illustrated and described, it will be appreciated that various changes can be made therein without departing from the spirit and scope of the invention.