Method and apparatus for structuring, maintaining, and using families of data

[0001] This non-provisional application takes priority from U.S. Provisional Application No. 60/234,015 filed on Sep. 20, 2000.

[0002] 1. Field of the Invention

[0003] This invention relates to the field of computer software. More specifically the invention relates to an improved method and apparatus for structuring, maintaining, and using families of data.

[0004] 2. Background Art

[0005] Many companies use catalogs to convey information about the products they sell. The organization and layout of each catalog that is published is important because the catalog must quickly convey information to the purchaser about the products the company offers for sale. For instance, when publishing the contents of a catalog, product information should be organized into a more detailed arrangement than that provided by the categories of a typical classification scheme. A detailed arrangement groups items according to the category value and other criteria. For example, products in a certain category, such as paintbrushes, may also be grouped by manufacturer. These groupings are referred to as families. Generally speaking, a family can be defined as a group of records, in a table, related by one or more common fields having the same value. These families may also have additional fields of common information, such as of common information, such as images, logos paragraphs of descriptive text, bullets of specifications, and other data. Families provide a way of identifying groupings by fixing one or more common fields and/or attribute values. Existing methods use data structures to store and retrieve these families of records. However, these methods present several problems with defining structures. To educate the reader, a brief description of some of the problems with arranging records in families follows.

[0006] For illustration purposes a brief example of a family will follow. Initially, the data to be illustrated in a catalog (or any other type of data in a database) is represented in a classification scheme called a taxonomy. The taxonomy provides for the partitioning of a table and its records into multiple categories, with or without a hierarchy, along with the assignment of attributes to each of a number of categories. In Table 1, a taxonomy is used where a table and its records are partitioned into categories, with or without a hierarchy, where each category comprises a set of common attributes. A category's attributes may not be physically part of a record but instead can be considered part of the definition of the record, where the record contains a reference to the category's attributes.

[0007] The examples that follow will be based on the taxonomy and data displayed in Tables 1-4:

[0008]

TABLE 1
Category ID	Category	Parent ID	Position
1	Printers	0	0
2	Daisy Wheel Printers	1	0
3	Dot Matrix Printers	1	1
4	Inkjet Printers	1	2
5	Laser Printers	1	3

[0009]

TABLE 2
Attribute ID	Attribute	Type
1	Pages Per Minute (ppm)	Numeric
2	Color	Text

[0010]

	TABLE 3
	Attribute ID	Feature ID	Feature
	2	1	Color
	2	2	Black & White

[0011]

	TABLE 4
	Category ID	Attribute ID
	1	1
	1	2

The four tables above define the following taxonomy:

[0012] Printers (ppm, color)

• • [0013] Daisy Wheel Printers
  • [0014] Dot Matrix Printers
  • [0015] Inkjet Printers
  • [0016] Laser Printers

[0017] The taxonomy provides an example of a category hierarchy with five categories, a root category (a node that has no parent), identified as “Printers”, and four remaining child (and leaf node) categories associated with the “Printers” category. The “Printers” category may have two attributes “ppm” and “color”.

[0018]

TABLE 5
PRINTERS:
ID	Model	Manufacturer	Category ID	Description	Price
1	ALP1	Acme	5	8 pages per minute. black & white	$500
2	AIJP1	Acme	4	3 pages per minute ink, black & white	$150
3	ALP2	Acme	5	8 pages per minute; color	$4000
4	ADMP1	Acme	3	3 pages per minute; black & white	$100
5	BLP1	Best	5	20 pages per minute, color	$5000
6	BLP2	Best	5	20 pages per minute, black & white	$1000
7	BIJ1	Best	4	4 pages per minute; color	$250
8	BDWP1	Best	2	2 pages per minute, black & white	$75

[0019] The first table (Table 1), or category table, defines categories within the taxonomy. The category table includes a “Parent ID” field that may be used to define a hierarchy and, more particularly, a category's level within a category hierarchy. An attributes table (Table 2) defines attributes that may be included in a category. Table 3, a feature-values table, may be used to define enumerated values of an attribute of the attributes table. In the example, the feature values table identifies two enumerated values for the “color” attribute. Table 4, a category-attribute table, identifies the attributes that are associated with a record of the category table. Inheritance may be used to allow child categories to inherit attributes that are associated with a parent category. The families, in the examples, will be defined by the combination of manufacturer and category. The fifth table (Table 5) shows a list of data entries for printers. The “Position” field identifies a position within a hierarchical level for a given category. Each of the records in a uniform fields table (i.e., Table 5) references a category record in the category table (Table 1) that defines additional data elements (or attributes) of the referencing record.

[0020] Several solutions may be used to partition the data (e.g., in Table 5) into families. A brief description of some of these solutions and the problems associated with them follows.

The “Table Per Family” Approach

[0021] The “table per family” approach partitions the records into families by storing the records of each family in its own table (e.g., Tables 6-11).

[0022]

TABLE 6
ID	Model	Manufacturer	Category ID	Description	Price
1	ALP1	Acme	5	8 pages per minute; black & white	$500
3	ALP2	Acme	5	8 pages per minute; color	$4000

[0023]

TABLE 7
ID	Model	Manufacturer	Category ID	Description	Price
2	AIJP1	Acme	4	3 pages per minute ink; black & white	$150

[0024]

TABLE 8
ID	Model	Manufacturer	Category ID	Description	Price
4	ADMP1	Acme	3	3 pages per minute; black & white	$100

[0025]

TABLE 9
ID	Model	Manufacturer	Category ID	Description	Price
5	BLP1	Best	5	20 pages per minute; color	$5000
6	BLP2	Best	5	20 pages per minute; black & white	$1000

[0026]

TABLE 10
ID	Model	Manufacturer	Category ID	Description	Price
7	BIJ1	Best	4	4 pages per minute;	$250
				color

[0027]

TABLE 11
ID	Model	Manufacturer	Category ID	Description	Price
8	BDWP1	Best	2	2 pages per minute; black & white	$75

This approach provides for efficient storage of the data. However, as the number of families increases, so does the number of tables. Data management and searching for records then becomes increasingly complex and time-consuming because additional tables must be accessed. Furthermore, changes to the family definition require complex restructuring of the tables and reorganization of the records contained within them. For example, if families were changed to be defined as the combination of the category and the color attribute, then six new tables (Laser/Color, Laser/B&W, Inkjet/Color, Inkjet/B&W, Dot Matrix/B&W, and Daisy Wheel/B&W) would need to be created and populated, and the old tables would have to be destroyed.
The “Table Lookup” Approach

[0028] The “table lookup” approach typically requires three steps. First, a table containing a record for each of the families must be created (e.g., Table 12). Second, a lookup field for the family must be added to the partitioning table. Third, the identifier (ID) of the proper family record, in the family table, must be placed into this field for each record of the partitioning table to create a relationship between each record and its corresponding family (e.g., Table 13).

[0029]

TABLE 12
Family ID	Description
1	Acme Laser Printers
2	Acme Inkjet Printers
3	Acme Dot Matrix Printers
4	Best Laser Printers
5	Best Inkjet Printers
6	Best Daisy Wheel Printers

[0030]

TABLE 13
ID	Model	Manufacturer	Category ID	Description	Price	Family ID
1	ALP1	Acme	5	8 pages per minute; black & white	$5500	1
2	AIJP1	Acme	4	3 pages per minute ink; black & white	$150	2
3	ALP2	Acme	5	8 pages per minute; color	$4000	1
4	ADMP1	Acme	3	3 pages per minute; black & white	$100	3
5	BLP1	Best	5	20 pages per minute; color	$5000	4
6	BLP2	Best	5	20 pages per minute; black & white	$1000	4
7	BIJ1	Best	4	4 pages per minute; color	$250	5
8	BDWP1	Best	2	2 pages per minute; black & white	$75	6

[0031] This approach has several major drawbacks. First, the manual process of assigning the family identifiers is time-consuming, error-prone and extremely tedious. Second, changes to the record do not result in the product being properly reassigned to the correct family. Third, changes to the families may require that some or all of the records of the family be reassigned.

The “Stored Query” Approach

[0032] Because the related records in a family have the same fixed values for a set of field values, they can be identified by a query specifying these common values. This query can be stored and later referenced to identify and locate the records for the family.

[0033]

TABLE 19
Query Name	Query
Acme Laser	Manufacturer = Acme; Category = Laser Printers
Printers
Acme Inkjet	Manufacturer = Acme; Category = Inkjet Printers
Printers
Acme Dot Matrix	Manufacturer = Acme; Category = Dot Matrix Printers
Printers
Best Laser	Manufacturer = Best; Category = Laser Printers
Printers
Best Inkjet	Manufacturer = Best; Category = Inkjet Printers
Printers
Best Daisy Wheel	Manufacturer = Best; Category = Daisy Wheel Printers
Printers

[0034] This approach also has several shortcomings. First, there are a variety of problems setting up and maintaining the queries. Setting up the queries is time-consuming and error-prone, because each must be manually done. Each query must be given a name or identifier so that it can be referenced and, with a large number of families, it quickly becomes difficult to organize and manage the set of family queries. There is no way to guarantee that the set of queries will contain the entire set of records, while also ensuring that each record belongs to exactly one query; that is, some queries may inadvertently overlap so that a single record belongs to multiple families, or the queries may not provide adequate coverage, so that some records may not belong to any family. The relationship between the families is not visually obvious from the queries, nor is there any single structure that identifies, illustrates, or maintains these relationships. Finally, while the queries identify which records belong to the family, they fail to provide an efficient way to determine to which family a particular record belongs. Finding the family for a particular record would require examining each of the queries, one at a time, to see if the record matched the criteria for that query.

Storing Common Information For Family

[0035] Another common data storage problem concerns the need of a database to store fields of common information that relate to a family of related records rather than just a single record. The challenge is to store information in a way that is efficient, easy to implement for existing data, and easy to maintain, as additional records are added to the database.

Single Table Approach

[0036] Existing solutions use a “Single Table approach” or a “Multiple Table approach”. In the “Single Table” approach, all of the data values for a main table record, including the common information that applies to an entire family of records are stored, within the record itself in the single main table. As a result, the table structure is very simple but, at the same time, it is both wasteful of storage because the common data values are duplicated in multiple records, and wasteful of effort because each of the values must be entered manually and repetitively for each of the multiple records in a family. In addition, a change to any of the common data values is not automatically propagated through the entire family of records; rather, the data value must be updated in each of the multiple records that contain the value, introducing the potential for inconsistency and error.

Multi-Table Approach

[0037] The “Multi-Table” approach is consistent with the relational data model and uses multiple tables to store related information. The primary table stores the specific information about each main table record while a lookup table contains a record for each family that stores the fields of common information. Records in the tables are linked by placing an identifier in both tables that links each record in the primary table to the corresponding record in the lookup table. The advantage of this approach is that the common data values are stored only once in a single record in the lookup table, eliminating duplication and saving space; additionally, changes to the single copy of the common information are automatically reflected in all the records of a family. The drawback of this approach is that the link between each record in the primary table and corresponding record in the lookup table still needs to be defined manually; similarly, new records that are added to the database must be manually linked to the common information by the user rather than automatically linked by the system. In addition, if there are many different fields of common information, but only some of them are used for each family, the columns that store the information will be sparse.

Publishing

[0038] A third aspect related to data storage and retrieval relates to publishing catalogs of product information in paper and electronic media. Publishing catalogs of product information in paper and electronic media historically has been two very different and distinct processes, with a very different level and type of effort involved, and very different standards and expectations for quality. The challenge is to eliminate the distinctions between paper and electronic output and combine the best of both media in a way that brings to electronic catalogs the structure and high standard of quality typical of paper catalogs and, at the same time, dramatically reduces the cost of laying out paper catalogs by flexibly, programmatically, and automatically generating page layouts in real time.

[0039] Known solutions present several shortcomings. Paper catalogs are meticulously laid out, with existing page layout programs, a page at a time. Tables are formatted individually by manually populating page layouts with product data, a process that is time-consuming, tedious and very, very expensive. There is also no simple way to experiment with different tabular layout formats and views of the data. Once a page has been laid out, it is difficult to add or remove records from tables without destroying the structure of the page and requiring that it be laid out again (sometimes from scratch), which discourages updates with the result that catalog pages tend to quickly become out-of-date. The upside of this complex process, however, is that manual page layout usually results in high page density, flexible and well-structured tabular layout formats using pivots to eliminate redundant information, and a very high overall standard of quality. Notwithstanding the high level of quality, however, it remains difficult to enforce a uniform look throughout a publication because more than one person is usually involved in the page layout process, and each lays out pages somewhat differently.

[0040] By contrast, electronic catalog pages are typically database-driven and generated programmatically in real-time. Since page layouts do not actually exist until the electronic catalog page is displayed, new products can be added and old products removed without disturbing the system or the published output. Unfortunately, the downside of this flexibility is that automatically generated electronic catalog pages are usually no more than wide, ugly, “spreadsheet-style” tables of data with redundant information, very little structure, and none of the sophisticated tabular layout formats that are standard for paper pages. With category-specific attributes and a large number of categories, it is even more impractical to have a customized hand-coded display for each family, so generic unstructured presentations are even more the norm.

[0041] Moreover, when publishing to multiple media, none of the effort invested in meticulously laying out paper pages can be leveraged for the electronic catalog, since both the structure of the tabular layout formats as well as the product data are typically trapped within the page layout itself, while the electronic catalog requires that the data be stored and managed in a database to be searchable and generated in real-time. Thus the worlds of the two media are completely distinct and non-overlapping, very difficult to integrate, and require two distinct publishing efforts.

[0042] The invention is a method and apparatus for structuring, maintaining, and using families of data. Each family of data represents a group of records in a database table. Records in the group of records are related by one or more common field values. In an embodiment of the invention, the fields and attributes are combined to construct family items. Each family item is stored in a family table (or partitioning table). A family item refers to one field value item or to a combination of field values. Each family possesses a description and is characterized by the values of the fields it comprises. Fields used to construct families possess relationships with each other. Using these relationships, one is able to link field values in a hierarchy. A hierarchy can be defined when a group of records (having a set of field values) comprises all the characteristics of a second group of records, and further comprises one or more extra field values. In this vase, the first group is called the parent family and the second one is called the child family. Thus, with more than 2 fields one may build a hierarchical tree comprising multiple levels of inheritance. An embodiment of the invention uses the fields to generate taxonomy, where each family is identified through a combination of a unique set of field values.

[0043] An embodiment of the invention uses family identifiers to label each record in the family table with a unique identifier. The identifiers are also used to populate a field, in the table, reserved for holding the identifier of the parent family, thus allowing for traversing hierarchical trees in both ascending and descending orders. Embodiments of the invention partition the records of a table according to the set of families constructed. Partitioning the table records may be performed by setting the value of a field in the family table, reserved for the purpose, to the value of the family to which the record in the main data table belongs.

[0044] The invention offers means to manage and update family structures. For example, embodiments of the invention may reconstruct the family structure upon insertion or deletion of one or more records in the database partition table.

[0045] By partitioning the records and storing the family information in a separate table, the invention offers methods to enhance data retrieval, and allows for dynamically changing the data structure for database output. An embodiment, of the invention provides means for storing formatting data along with the data in the database. In this case, database driven document generation depends more on the formatting stored in the database and less on the rendering programs that generate the documents.

[0053] An embodiment of the invention comprises a method and apparatus for structuring, maintaining, and using families of data. In the following description, numerous specific details are set forth to provide a more thorough description of embodiments of the invention. It will be apparent, however, to one skilled in the art, that the invention may be practiced without these specific details. In other instances, well known features have not been described in detail so as not to obscure the invention. A description of some relevant database terminology can be found in Section A.

Structuring and Creating Partitioning Families

[0054] Each family of data represents a group of records in a database table that are related by one or more common field having the same value, and that may also have additional fields of common information (e.g., images, logos paragraphs of descriptive text, bullets of specifications and other data). Families are used to indirectly partition the records in a database. A partition is the division of a group of records into one or more subgroups, each of which is defined by a set of records from that group that have a fixed set of values for one or more field values. The partition is specified by the set of fields whose values or value combinations will define the subgroups. Each field can include category specific attributes. The main table of records that is to be divided into partitions is divided according to a partitioning table, e.g., the family table.

[0055] To generate a family structure in an existing database, the existing taxonomy (e.g., classification structure) in a database is assumed. Further information about family structures can be found in patent application entitled “DATA INDEXING USING BIT VECTORS”, U.S. Ser. No. 09/643,207, which is incorporated herein by reference. The taxonomy represents the partitioning of a table into multiple categories, with or without a hierarchical structure, along with the assignment of attributes to each category. A category is a subset of the records of a table that has a set of common field values or combination thereof. Each record in a table belongs to exactly one category.

[0056] Embodiments of the invention take advantage of the fact that each family is defined by fixing a set of common values for one or more fields.

[0057] In an embodiment of the invention, families are organized into a partitioning hierarchy. FIG. 1 is a diagram illustrating how lists of field values can be arranged in a hierarchical structure to build partitioning families in an embodiment of the invention. In FIG. 1, P₀ 110 is the root node in the tree. P₀ 110 may for example be a field describing a set of records in the database. In the example provided above, all printers possess a “Printer” designation. The designation “Printer” is used as a root in a hierarchy of families to specify that all products contained in the hierarchy must be printers.

[0058] In an embodiment of the invention, the list of field values 100 (e.g. {A₁, A₂, A₃} and {B₀, B₁, B₂}) are selected by a user, or automatically generated using the data records present in the database. The lists of field values are used to build families in a hierarchy. Each field value may be a node in the hierarchy tree. In the example provided above, Color, Laserjet and Printer are descriptions of a product combined to build the family of “Color laserjet printer”. Another example of a family is “Color inkjet printer”. Each path of the hierarchy tree may be used as an entry in a family table that is also referred as the partitioning table of the partitioning hierarchy. A partitioning hierarchy of a partitioning table is a hierarchy in which the nodes of the hierarchy represent partitions of the partitioning table. In FIG. 1, the path 120 is typically a family in the partitioning hierarchy. In addition some tree paths are also considered partitioning nodes (e.g. 130) because of the lack of records matching the family specification. In the example provided above, if no manufacturer in the database makes a network dot matrix printer, then the partitioning hierarchy may not include a “network dot matrix printer” family. Therefore, a partitioning node is a node in the partitioning hierarchy that corresponds to a particular family of records. Since a partition simply divides a group of records into sub-groups, the set of records represented by a partitioning node is exactly the set of records represented by combining the sets of records represented by each of the descendants of that partitioning node. The root partitioning node (or root partition) represents the entire set of records of the partitioning table; each sub-node represents only those records which have a fixed set of field values defined by the partitions, starting at that sub-node and tracing ancestors back up to the root. The entire set of leaf partitioning nodes (or leaf partitions) represents the entire set of records and each record of the partitioning table belongs to one and only one leaf partitioning node. In what follows, a base family will refer to a family that corresponds to a leaf partitioning node. Also, the base family set will refer to the complete set of base families that corresponds to the complete set of leaf partitions in a partitioning hierarchy. The base family set is useful because each record of the partitioning table belongs to exactly one base family.

[0059] FIG. 2 shows a flowchart diagram illustrating the overall steps involved the method of building the family-based partitions in an embodiment of the invention. In step 210, one or more sets of field values are collected. This process may involve user intervention through a user interface (see below, in the example of implementation), an automatic process for determining a set (or sets) of field values or categories (see below, in the example of maintaining family-based partitioning), or a combination of both user input and an automatic process. In an embodiment of the invention, in step 220, the set (or sets) of field values and categories are used to specify product families. Since a category and field-based taxonomy already exists, it would be beneficial to layer the partitioning hierarchy on top of it, so as to leverage the work already done to create the taxonomy. This can be accomplished by using the category field to define the first partition (e.g. P₀ in FIG. 1) in the partitioning hierarchy. At first this might appear to be the same as the Taxonomy approach presented above. The difference lies in the fact that the partitioning hierarchy is layered on top of the existing taxonomy, rather than incorporating the family information directly into the taxonomy. The steps involved in building a set of family items further described in FIG. 3.

[0060] In an embodiment of the invention, each family item in a set of family items is associated with an identifier (e.g. step 230). The identifier allows subsequent location of the family item in the partitioning table, and use of the inheritance tree in the hierarchical partitioning. In step 240, inheritance relationships are defined and implemented. For example, an embodiment of the invention uses a descriptor (e.g. a field in a database table) to hold the identifier of the parent in the hierarchy. In step 250, partitions are built using each valid path in the partitioning tree.

[0061] FIG. 3 shows a flowchart illustrating the steps involved in obtaining family items from one or more sets of field values in an embodiment of the invention. In step 310 one or more sets, containing one or more field values, are loaded through a user interface, or through a process for automatically selecting field values. In an embodiment of the invention, each combination of field values is used to form a database query. In step 320, the query is submitted to the database. The test in step 330 indicates whether a combination of field values have any associated records in the database as a result of the query. If the query points to existing records in the database, the combination of field values is retained and processed to produce a family item in step 340. Otherwise the field values are ignored in step 350. In step 360 a search is performed to check whether all field values in the sets of fields were processed. The processing continues until all field values in the set are processed. In the case of a hierarchical relationship, this is equivalent to traversing all the possible paths in the hierarchy tree.

[0062] FIG. 4 shows a flowchart illustrating the steps involved in building a hierarchy between family items in an embodiment of the invention. In step 410 each family item is assigned an identifier (see above). In step 420, for each node, the identifier of the parent of the node is determined. The parent node identifier is associated with the node in step 430, for example, by entering the identifier of the parent in the Parent ID field corresponding to the record of the node. In step 440 a position of the field value in the set of field values is provided. The family item is then associated with the position in step 450.

[0063] In an embodiment of the invention, the partitioning hierarchy is stored as a hierarchical structure. An additional table is used to store the fixed field values that define the partitions.

[0064] In the example provided below, the table contains fields that provide information on the identifier of the partitioning node, the field value that is being partitioned, and positional information to allow for combining and nesting partitions. Rather than storing the partitioning information directly as part of the hierarchy table, an additional table is used because there may be multiple fields that define a partition. For example, a partition could be defined based on the combination of a field (such as the manufacturer) and an attribute (such as color).

[0065]

TABLE 20
Family ID	Family	Parent ID	Position
1	Printers	0	0
2	Daisy Wheel Printers	1	0
3	Best Daisy Wheel Printers	2	0
4	Dot Matrix Printers	1	1
5	Acme Dot Matrix Printers	4	0
6	Inkjet Printers	1	2
7	Acme Inkjet Printers	6	0
8	Best Inkjet Printers	6	1
9	Laser Printers	1	3
10	Acme Laser Printers	9	0
11	Best Laser Printers	9	1

[0066]

TABLE 21
Family ID	Field
1	Manufacturer

[0067] Table 20 defines the following family partitioning hierarchy:

• [0068] Printers (ppm, color)
  • [0069] Daisy Wheel Printers
    • [0070] Best Daisy Wheel Printers
  • [0071] Dot Matrix Printers
    • [0072] Acme Dot Matrix Printers
  • [0073] Inkjet Printers
    • [0074] Acme Inkjet Printers
    • [0075] Best Inkjet Printers
  • [0076] Laser Printers
    • [0077] Acme Laser Printers
    • [0078] Best Laser Printers

[0079] Table 21 shows an additional partition layered on the top of the previous taxonomy describing using the manufacturer field.

[0080] In this example, notice that the family partitioning hierarchy has the same initial structure of the taxonomy, but additional nodes are added to it. These nodes are created because a partitioning exists at the Printers node that is defined to partition according to manufacturer. This causes all leaf nodes under this node to be further partitioned by manufacturer. The initial leaf nodes were Daisy Wheel Printers, Dot Matrix Printers, Inkjet Printers, and Laser Printers. Under each of these, additional nodes will be added for each manufacturer that has products defined by the query constructed by taking all of the criteria defined by the ancestor nodes in the family partitioning hierarchy. Since this is the first partition, the criteria are simply the category for each of the initial leaf nodes. Notice that a node is not added for all manufacturers, only those that correspond to actual records in the database.

[0081] Whenever a new category is added to or removed from the taxonomy, the corresponding portion of the family partitioning hierarchy must also be adjusted in the same manner. This is an important constraint on this approach and will result in a change of base families.

[0082] This idea can be extended to reflect changes in the possible values for other fields in the family partitioning hierarchy. Thus, when a value is added/removed from the set of possible values for a particular partition, the corresponding node will be added/removed from the family partitioning hierarchy. This is illustrated below where, in addition to partitioning on the category (the initial taxonomy) and manufacturer (the additional nodes added to account for the manufacturer), a partition by Color is also performed on the Laser Printers node.

[0083]

TABLE 22
Family ID	Family	Parent ID	Position
1	Printers	0	0
2	Daisy Wheel Printers	1	0
3	Best Daisy Wheel Printers	2	0
4	Dot Matrix Printers	1	1
5	Acme Dot Matrix Printers	4	0
6	Inkjet Printers	1	2
7	Acme Inkjet Printers	6	0
8	Best Inkjet Printers	6	1
9	Laser Printers	1	3
10	Acme Laser Printers	9	0
11	Best Laser Printers	9	1
12	Color Acme Laser Printers	10	0
13	B&W Acme Laser Printers	10	1
14	Color Best Laser Printers	11	0
15	B&W Best Laser Printers	11	1

[0084]

TABLE 23
Family ID	Field
1	Manufacturer
9	Manufacturer
9	Color

Table 22 defines the following family partitioning hierarchy:

• [0085] Printers
  • [0086] Daisy Wheel Printers
    • [0087] Best Daisy Wheel Printers
  • [0088] Dot Matrix Printers
    • [0089] Acme Dot Matrix Printers
  • [0090] Inkjet Printers
    • [0091] Acme Inkjet Printers
    • [0092] Best Inkjet Printers
  • [0093] Laser Printers
    • [0094] Acme Laser Printers
      • [0095] Color Acme Laser Printers
      • [0096] B&W Acme Laser Printers
  • [0097] Best Laser Printers
    • [0098] Color Best Laser Printers
    • [0099] B&W Best Laser Printers

[0100] In this example, notice that partitioning information for Color has been added to the Laser Printers node, and that only descendants of that node are affected (see Table 23). Also, notice that a second occurrence of a manufacturer partition has been added. The reason is that descendant nodes inherit partition information. In other words, all descendant nodes of a particular “ancestor” node are automatically assigned the same partition information that is assigned to the ancestor, which makes setting up and maintaining partitions much more efficient. However, if there were no way to override the partition settings of an ancestor node, inheritance would always affect all descendant nodes. To get around this problem, inheritance does not affect a node that has any partitions defined nor does it affect any of its descendants; rather the descendants inherit the override partition settings. In order to obtain the partition defined for an ancestor as well as a custom partition, a node must define both partitions. If the second occurrence of the manufacturer partition had not been added, then the family partitioning hierarchy would be as follows:

[0101]

TABLE 24
Family ID	Family	Parent ID	Position
1	Printers	0	0
2	Daisy Wheel Printers	1	0
3	Best Daisy Wheel Printers	2	0
4	Dot Matrix Printers	1	1
5	Acme Dot Matrix Printers	4	0
6	Inkjet Printers	1	2
7	Acme Inkjet Printers	6	0
8	Best lnkjet Printers	6	1
9	Laser Printers	1	3
10	Color Laser Printers	9	0
11	B & W Laser Printers	9	1

[0102]

TABLE 25
Family ID	Field
1	Manufacturer
9	Color

Table 24 defines the following family partitioning hierarchy:

• [0103] Printers
  • [0104] Daisy Wheel Printers
    • [0105] Best Daisy Wheel Printers
  • [0106] Dot Matrix Printers
    • [0107] Acme Dot Matrix Printers
  • [0108] Inkjet Printers
    • [0109] Acme Inkjet Printers
    • [0110] Best Inkjet Printers
  • [0111] Laser Printers
    • [0112] Color Laser Printers
    • [0113] B&W Laser Printers

[0114] Table 25 shows, in addition to the manufacturer partitions from the previous example, a third partition (Color).

[0115] In this example, there is also a difference based on the ordering of the partitions. Had the Manufacturer Name partition been added after the Color partition, then the result would be as above with two nodes added under each of the Color Laser Printers and B&W Laser Printers nodes for the Acme and Best manufacturers.

[0116] Partitioning by multi-valued fields is given special treatment to ensure that a record belongs to exactly one family. The combination of values is treated as a distinct unit when determining the unique set of values for the field. For example, if there was a partition on a multi-valued field of color and one of the records had Blue/Green as the value for that field, then the record would be placed in the Blue/Green family, and not in the Blue family or Green family.

[0117] In an embodiment of the invention, in order to find the records that belong to a particular family, a query can be constructed by setting constraints for each value from the fixed set of common values for that family. Executing that query will locate a set of records that belong to that family.

[0118] Since the partitioning hierarchy is organized so that each branch, from a node to its sub-nodes, differ in the value or value combination on which the node is partitioned, each of the leaf partitioning nodes will differ by at least one value or value combination. Thus the queries constructed for each of the base families will also differ by at least one constraint. The result is that each query is guaranteed to return a non-overlapping set of records.

[0119] The linkage between families and records is accomplished automatically by constructing queries with the appropriate constraints for that family, as opposed to the manual process of linking each record to the proper family. As an added benefit, when new records are added or existing records are modified, they will belong to the proper family automatically. Also, if the partitioning hierarchy is restructured so that family definitions change, each record of the partitioning table will automatically belong to its proper family.

[0120] There are several advantages provided by embodiments of the method described above. Databases implementing the method provide support for partitioning based on product families and the product families hierarchy. Embodiments of the invention provide for efficient storage for families allowing products to be found from families, and, conversely, families to be found from products. The layering of partitioning hierarchy on top of a category based taxonomy leverages existing taxonomies. Embodiments of the invention provide a method for automatically creating new families as the set of actual field values is changed. Embodiments of the invention provide for ensuring that product records automatically belong to the proper family, even as new records are added and existing records are modified. The method provides for the ability to partition at any level in the partitioning hierarchy, so that different nodes within a single partition can be partitioned differently. Other embodiments of the invention implement inheritance and the overriding of the inheritance of partition information in the partitioning hierarchy

[0121] FIG. 6 shows a flowchart illustrating the steps involved in obtaining a family item from a record in the partitioning table in an embodiment of the invention. To build a family item a process starts at the root node of the family tree in step 610, and identifies the partition field value in step 620. The process continues fetching child nodes using the value of partition field value in step 630. A relevant child node should have a value equal to the partition field value in the record. The process checks that the node is a leaf node in 640. If the node is a leaf node (i.e. has no child nodes), the result of the combination of field values is the family item. The search is finished in step 650. Otherwise, the process continues searching for child nodes by traversing the partitioning tree.

[0122] FIG. 7 shows a flowchart illustrating the steps involved in obtaining all the records for a given family item in an embodiment of the invention. The process of obtaining records in a family involves two major steps: 1) building a list of constraints and 2) running a query with those constraints against a database. In an embodiment of the invention, a list of constraints is built starting with the loading of the family node in step 710. The node's parent is fetched, and the value of the partition field value by which the parent node is partitioned is added to the constraint list in step 720. The search continues by traversing the tree, searching for each current node's parent in step 730. The root node is found by checking each current node tested in step 740. When the root node is found, a query comprising the list of constraints is ran against the database in step 750. The result of such a query returns the records that are part of the family for which the search started.

Maintaining Product Families

[0123] After the family partitioning hierarchy has been created, it must be maintained when there are changes to the taxonomy structure or to the domain of fields values used for partitioning. Changes to the taxonomy structure that require updates to the partitioning hierarchy include adding, removing, moving, and modifying a category. Changes to the domain of a partitioning field include adding, removing and modifying a field value, while changes to the feature domain for a partitioning attribute include adding, removing and modifying a feature value.

[0124] A second problem arises as a result of an optimization that avoids creating a family partitioning hierarchy that contains a high percentage of families with no records. In the previous section, we had assumed that the set of possible values and value combinations and the set of actual values and value combinations in existing main table records were identical. The optimization recognizes that this is not likely to be the case, and that in fact, the number of actual values and value combinations will be substantially less than the number of possible values and value combinations.

[0125] Note that using the set of possible values and value combinations when creating families causes the partitioning hierarchy to become unnecessarily large because it will contain many families that contain no records. To illustrate this point, consider a catalog with 200 categories, 500 manufacturers, and 10,000 products. If category were to be partitioned by manufacturer, the “cross-product” approach of using the possible value combinations would create 100,000 families in the partitioning hierarchy, even though the main table contains only 10,000 product records! Most of these families would in fact contain no records, since for a particular category, only a small subset of manufacturers offers products (and conversely, each manufacturer offers just a small number of categories of products).

[0126] By contrast, using only the set of actual value combinations that occur in the main table records reduces the number of families dramatically to precisely those containing records (and certainly no more than the number of products in the main table) and results in a much more compact partitioning hierarchy. A consequence of this optimization, however, is that the partitioning hierarchy must now be maintained not only across changes to the taxonomy structure and domains of partitioning field values, but also across changes to main table records. These changes include adding, removing, or modifying main table records.

[0127] Embodiments of the invention provide a solution to automatically adjust the partitioning hierarchy when the taxonomy structure, the domain of a partitioning field, or the main table records are modified.

[0128] Since the partitioning hierarchy is layered on top of the taxonomy, changes to the structure of the taxonomy hierarchy require updates to the partitioning hierarchy. In particular, nodes that are added, removed, modified, or moved in the taxonomy must be similarly added, removed, modified, or moved in the partitioning hierarchy. In addition, many of the advanced features for in-place schema and data manipulation such as splitting and merging fields can also require updates to the partitioning hierarchy.

[0129] Since the partitioning hierarchy depends on the existence of values in actual product records, changes to the main table records may require updates to the partitioning hierarchy. When records are added to the main table, new families must be created if the records contain a value not yet used in any of the fields that are used in defining the family partitions. Similarly, if a record is deleted from the main table and that record is the only record in the main table to contain a particular value for one of the family partitioning fields, the corresponding partitioning node must be removed. Modification of a main table record can have effects similar to those of adding a new record or deleting an existing one since a new value assigned to a field or record could be a value not yet used in one of the family partitioning fields and the value replaced could have been the only occurrence of a particular value in the family partitioning field value. The merging field values in the taxonomy has the same effect as modifying the main table records by replacing the original filed values with the merged field value and can require updates to the partitioning hierarchy.

[0130] Note that updates to the partitioning hierarchy to reflect changes to the domain of a partitioning field are automatically handled through the handling of changes to the main table records. This is because changes to a domain no longer affect the partitioning hierarchy unless the added, removed or modified value is actually in use in the main table records.

[0131] FIG. 5 shows a flowchart illustrating the steps involved in automatically updating the family partitioning in an embodiment of the invention. In step 510, new records are detected (e.g. receiving a data insertion query in by the database, or upon an alteration of existing records). Records are checked in step 520 to test whether a new field value is introduced. If a new field value (relevant to the family partitioning) is detected, the family partitioning is reorganized in step 530. The method provides a mechanism for automatically maintaining product families and the partitioning hierarchy. For example, the method provides a mechanism for detecting when the partitioning hierarchy needs to be updated due to modifications of the taxonomy or main table records. Partitioning nodes may be created based on the actual set of values and value combinations used in main table records rather than the possible set of values and value combinations. The method also provides way to detect if a field value disappears (e.g., upon a deletion of records or alteration of the records in the database). The method provides a way to check whether a field value was deleted from the database in step 550. If such event occurs, family partitioning is reorganized to optimize the family partitioning. An embodiment of the invention checks user input instructions to modify the family partitioning in step 570. If a user inputs data to modify the family partitioning the latter is reorganized to optimize family partitioning.

Maintaining Common Information for Families

[0132] An embodiment of the invention provides an improved solution for storing data, allowing maintenance of all the benefits of the multi-table approach, while eliminating the need for a lookup field in the primary table whose value identifies the identifier of the corresponding record in the lookup table. This method simultaneously eliminates the need for the user to manually place the identifier of the lookup record into this lookup field in each primary table record. Instead, the improved solution layers on top of the family partitioning hierarchy in such a way that the system creates and maintains all of the relationships automatically based on the membership of each group of primary table records in each family in the family hierarchy.

[0133] In an embodiment of the invention, after partitions have been defined by the user and the family partitioning hierarchy has been generated by the system, the user assigns the common information for each family to the families corresponding to leaf nodes of the family hierarchy in the next step. Under this scheme, records in the primary table have already been grouped together into families and common information is then easily assigned to each family. Each new record in the primary table is then automatically linked to the correct common information by virtue of its membership in the proper family. Moreover, for efficiency in storage, rather than store the data values in a fixed set of fields that exist for every family record, the data values are stored in a related, secondary table only on an as-needed basis so that, like attributes, they only take up space if they exist.

[0134] Embodiments of the invention provide for means to easily link common information to families and link common information to each family rather than to the main table records by utilizing the family partitioning hierarchy. Other embodiments include automatically creating and maintaining all of the relationships between existing and new main table records and common information based on family membership.

Media-Independent Publishing

[0135] Embodiments of the invention provide a solution to improve media publishing. An embodiment of the invention provides a method for a layer on top of the structure to automatically format and publish data from a database. All of the layout formats that are typically stored in the page layout are captured and stored in the database alongside the product data itself. In this scheme, the searchable, database-driven electronic catalog can not only serve up the product data but also the formatting data. The method allows for rendering in real-time by a report writer (such as ASP-generated HTML). The rendering is done independently of data types, even in a catalog with many categories and category-specific attributes. The report writer code itself (or HTML) need only handle the preprocessed pivot tables and requires no complex code for pivoting tabular layout formats, no special coding for each category or family, and no intelligence about the underlying data. More information about pivot tables can be found in co-pending patent application entitled “METHOD AND APPARATUS FOR DYNAMICALLY FORMATTING AND DISPLAYING TABULAR DATA IN REAL TIME”, filed on Sep. 20, 2001, Ser. No. to be assigned, which is incorporated herein by reference. Using the structure described in the previous section, electronic catalogs for the first time can now have the density and layout quality of paper catalog pages while maintaining their database-driven search ability.

[0136] Embodiments of the invention provide improved solutions for publishing database content that substantially eliminate the manual process of page layout for publishing paper catalogs. For example, the time and effort invested on defining the appropriate tabular layout formats are substantially reduced, since the tabular layout formats are set only once and do not have to be repeated for each family to publish catalogs. As opposed to existing methods that require users to manually populate page layouts with product data, the invention provides solutions that automatically generate page layouts by combining product data and formatting data from the database. Embodiments of the invention use the API of the page layout program (e.g., for programs such as QuarkXPress or Adobe InDesign), or an intermediate ASCII file format (for programs such as Xyvision XPP) to render pages automatically. The invention provides solutions that result in many ways in reducing the publishing cost. For example, embodiments of the invention allow changes to the product data to be reflected immediately in subsequently generated output. Embodiments of the invention support the on-demand generation of custom catalogs on product subsets with no additional effort. Other embodiments of the invention produce a more uniform look throughout the publication, since every page is generated dynamically and automatically by the system.

[0137] In an embodiment of the invention, each paper publication starts out as a snapshot of the family partitioning hierarchy and its associated formatting information. Any of the formatting specifications, defined and stored in the family partitioning hierarchy and used for electronic catalog publishing, can be changed in any way for each paper publication. This provides almost unlimited flexibility to create custom paper catalogs, each of which is based upon the electronic standard but is laid out in a fashion that is as similar to, or as different from, any other catalog as necessary. In addition, the system offers the following per-publication flexibility:

• • [0138] A product mask can be applied when the snapshot is taken to limit the set of products appearing in the paper publication, so that each publication can have a different, custom subset of the entire product set (masks can also be applied electronically, and/or search parameters specified, to limit the set of products appearing in electronic output).
  • [0139] The order of the partitions in a publication can be rearranged when the snapshot is taken and set in any order (by contrast, partitioning order is fixed in the family partitioning hierarchy).
  • [0140] The sequence of the families in a publication can be rearranged in any order (by contrast, the family sequence is fixed in the family partitioning hierarchy).
  • [0141] A family can be copied from the family partitioning hierarchy into the publication to include families that were not initially included in the publication.
  • [0142] Each family can appear in multiple locations in the publication, can be individually formatted, can include a different subset of the columns and common information, and can contain a different subset of the records in the family (by contrast, each family in the family partitioning hierarchy can appear only once, contains a fixed subset of the columns and common information, and contains all of the records).

[0143] Additional features for paper publishing that allow publication-specific restructuring and reformatting of each family as well as the entire publication are listed in the table below:

[0144]

TABLE 26
Feature	Description
Layout Detail	Change any or all of the tabular layout format
	settings of the current node
Column Names	Change any of the display names for the current node
Records	Exclude any of the records of the current node, or
	include any that had been masked out
Family Data	Exclude any common information of the current node
Refresh Options	Include or exclude new records, columns, or common
	information
Detail	Display the criteria for the current node
Format	Specify additional formatting options

[0145] Embodiments of the inventions offer several improved solutions over existing methods for database driven publishing. Embodiments of the invention provide means for layering both the electronic and paper publishing process on top of the same extended taxonomy structure for automatically formatting and publishing database data.

[0146] The invention uses tabular layout formats that are captured and stored in the database alongside the product data itself, instead of storing the formatting in the page layout. The invention provides means for publishing high-quality output to the web using layout information stored in the database. In addition, the invention uses the API of the page layout program (or intermediate ASCII file format) to render pages automatically. Further, the invention allows for applying a product mask when the publication is first created. Finally, it also allows the layout detail, column names, set of records, and common information to be individually customized for each family of a particular publication.

[0147] An embodiment of the invention is implemented in a database system to build a catalog manager. A detailed description of a catalog manager is provided in Section B.

Section A

[0148] The following definitions will assist in understanding the discussion contained in this application:

[0149] A database is a logical collection of interrelated information, managed and stored as a unit.

[0150] A record is a representation of a real-world object such as a person, a product, or a company. A record consists of one or more individual data elements.

[0151] A field describes one of the data elements of a record and is common to all the records in a table.

[0152] A table is a simple, rectangular, row/column arrangement of related data values. Each horizontal row in the table represents a single record and consists of the same set of fields. Each vertical column of the table represents one field that is stored for each row in the table.

[0153] A relational database is a database in which all data is organized into tables that may be related by matching columns.

[0154] A hierarchy is a table in which the records have parent/child relationships. A node is another term for a record in a hierarchy.

[0155] The root node of a hierarchy is a node that has no parent.

[0156] An internal node of a hierarchy is a node that has at least one child.

[0157] A leaf node of a hierarchy is a node that has no children.

[0158] An attribute is a data element that is not common to all the records in a table.

[0159] A category is a subset of the records of a table that has a set of common attributes. Each record in a table must belong to exactly one category.

[0160] A taxonomy is the partitioning of a table and its records into multiple categories, with or without hierarchy, along with the assignment of attributes to each of the categories.

[0161] A family defines a group of records in a table which are related by one or more common fields and/or attributes that have the same value, and which may also have additional fields of common information, such as an image, a logo, a paragraph of descriptive text, bullets of specifications, and so on.

[0162] A partition is the division of a group of records into one or more subgroups, each of which is defined by the set of records from that group that have a fixed set of values for one or more fields and/or attributes. The partition is specified by the set of fields and/or attributes whose values or value combinations will define the subgroups.

[0163] The partitioning table is also known as the family table and defines the partitions for the main table of records that is to be divided into partitions.

[0164] A partitioning hierarchy of a partitioning table is a hierarchy in which the nodes of the hierarchy represent partitions of the partitioning table. A partitioning node is a node in the partitioning hierarchy that corresponds to a particular family of records. Since a partition simply divides a group of records into sub-groups, the set of records represented by a partitioning node is exactly the set of records represented by combining the sets of records represented by each of the descendants of that partitioning node.

[0165] The root partitioning node (or root partition) represents the entire set of records of the partitioning table; each sub-node represents only those records which have the fixed set of field values defined by the partitions starting at that sub-node and tracing ancestors back up to the root; the entire set of leaf partitioning nodes (or leaf partitions) represents the entire set of records; and each record of the partitioning table belongs to one and only one leaf partitioning node. A base family is a family that corresponds to a leaf partitioning node.

[0166] Thus a method and apparatus for structuring, maintaining, and using families of data has been described. The invention, however, is defined by the claims and the full scope of any equivalents.