Method and system for object replication in a content management system

[0001] The present application claims priority under 35 U.S.C. §119(e) of Provisional U.S. Patent Application Ser. No. 60/265,187 filed Jan. 30, 2001.

[0002] The present invention relates generally to a content management system and particularly to a system for object replication in such a system.

[0003] Content management systems provide many services for the management of digital content. The basic client functions are logon, logoff; create folder, item or part; index item or part; search indexes; retrieve folders, item or parts; store parts; replace parts; and delete parts. In addition, a plurality of other features could be provided such as encryption, system administration, streaming of audio and video, caching functions or other types of management functions.

Content Management System Block Diagram

[0004] FIG. 1 illustrates the basic data flow in a content management system 100. The system includes a client 102, library server 101 and a plurality of object servers 103a–103c. The client 102 comprises an application and a toolkit. The toolkit includes a daemon process 106 that sends and receives data on behalf of the client. The client makes requests to the library server 101 and receives results. Objects are transported between a plurality of object servers 103a–103c and a plurality of the daemon process of the client as directed by the library server 101.

[0005] A given object is defined by an entry in the index or list of objects by a unique identifier and coupled with searchable attributes including the file or object server identifier and a collection identifier. The collection identifier describes how the object is to be managed for storage. A collection is a unit of storage conceptually a cabinet where objects are placed. It may consist of many volumes of various storage media and a set of rules as to how the actual objects are stored and handled. The library server 101 and each of the plurality of object servers 103a–103c are utilized in the conventional content management system 100 to manage digital content. Their functions are described below.

[0006] For a further description of the basic functions of the library server 101 and one of the plurality of object servers 103, refer now to the following discussion in conjunction with the accompanying figures.

Library Server 101

[0007] FIG. 2 is a diagram that illustrates the various elements of a conventional library server 101. The library server 101 holds index, attribute and content information in a searchable form within a relational database or through auxiliary servers. In a preferred embodiment, the library server 101 contains a foldering system and references to data objects that may be stored on an object server or other external file systems. The data objects may be any type of digitized information. The library server 101 also typically contains a workflow system.

[0008] As is seen, the conventional library server comprises a command monitor 120, a jobber 122, a plurality of child processes 124a–124e, and a database 126. The function of each of these elements will be described hereunder.

Command Monitor 120

[0009] The command monitor 120 is the main line for server code. The command monitor 120 also provides server control logic and starts other processes.

Jobber 122

[0010] The jobber 122 builds static access modules for the database to improve query performance.

Child Processes 124a–124e

[0011] The child processes 124a–124e perform requests from the clients (over the network, for example). The requests include but are not limited to query, add, update, attribute data, passes on store, retrieve, replace requests for objects to object server. The number of child process is configurable.

Database 126

[0012] The database 126 stores attribute and server control information. The database 126 of the library server 103 is accessed as needed by the child processes.

[0013] The library server 101 also includes a plurality of tables. The tables include a part table, object server table, and a collname table.

Parts Table 127

[0014] The function of the parts table 127 is described hereinbelow. One row of the table exists for each part. Parts are stored on object servers. The row identifies the item part and maps its location to an object server collection.

Object Server Table 129

[0015] The object server table 129 maintains information concerning the plurality of object servers.

Collname Table 131

[0016] The collname table 131 maintains the names of each collection for each object server.

Object Servers 103a–103c

[0017] Each of the object servers 103a–103c holds objects as files or references to other storage systems. The object server provides for name translation from library server name to file system name/location and for hierarchical storage management and transport of objects. Each of the object servers 103a–103c in a preferred embodiment also stores meta information in a relational database and in transaction log files. Finally, each of the object servers 103a–103c in a preferred embodiment also stores objects in files or other storage subsystems.

[0018] FIG. 3 is a diagram, which illustrates the various elements of a conventional object server 103. As is seen, the conventional object server comprises a command monitor 105, a purger 107, a destager 109, a migrator 111, child processes 112a–112e, a staging area 114, a plurality of volumes 116, and a database 118. The functions of these elements are described below.

Command Monitor 105

[0019] The command monitor 105 is a main line for server code, provides server control logic and starts other processes.

Purger 107

[0020] The purger 107 cleans the cache and removes least recently used items.

Destager 109

[0021] The destager 109 moves objects from cache to first storage class. The destager 109 maps a storage class to one or more volumes or to another object server. In the destager 109 mapping information is encoded in the database.

Migrator 111

[0022] The migrator 111 is an object server process that implements the storage manager activity moving objects from initial permanent storage to subsequent storage. The migrator 111 moves objects from one storage class to another storage class. Movement is defined by time and sequence as part of a management class.

Child Processes 112a–112e

[0023] The child processes 112a–112e perform the requests passed from the library server 103 to the client 102 daemon processes (over the network). The child processes 112a–112e store, retrieve, and replace requests for objects to object server. The number of child processes 112a–112e is configurable.

Staging Area 114

[0024] The staging area 114 is a cache area for object storage.

Volumes 116a–116d

[0025] Volumes 116a–116d are permanent storage media. The volumes 116a–116d may be disk, tape, optical or any type of storage subsystem.

Database 118

[0026] The database 118 holds object location and name mapping, and the system managed storage information and replication work requests and server configuration information.

[0027] A feature within the object server 163 is an object server table 121. The function of the object server table 121 is described below.

Object Server Table 121

[0028] The object server table 121 provides the objects that are stored and managed by that object server. One row within the table exists for each object stored and managed by the object server. The row identifies the object and maps its identifier to a local filename.

Functional Description

[0029] The function of the conventional content management system 100 (FIG. 1) is typically transactional in nature. A typical process for a transaction in a content management system is an object store process. In an object store process objects are stored in the appropriate locations within the system. FIGS. 4–6 are diagrams that illustrate conventional process for storing an object in a content management system.

[0030] First, in a begin transaction (FIG. 4), the client calls the application programming interface to store a part, via step 402. A memory pointer is then passed to the daemon within the client for use when an object server requests the part, via step 404. The store request contains the item and part information.

[0031] FIG. 5 illustrates a retrieve process. After receiving the store request from the client, the library server validates the store request and determines the destination information for this part, via step 502. The library server also inserts a row into the parts table. The library server then sends the store request to the selected object server, via step 504. This store request contains the part name, collection name, object size, daemon address and port, and time information.

[0032] After the object server receives the store request, the selected object server validates the store request and determines a storage location for the part, via step 506 (possibly a cache). The object server reserves file space for the object. The object server also logs the file location and cleans up resources. The object server then requests from the client a daemon to allow for sending the object, via step 508.

[0033] After receiving the request, the client daemon validates the object request and matches the object request to information passed from the client, via step 510. Then the daemon sends a response with the object appended thereto to the object server, via step 512. The object server places the object in prepared file space. Then the object server inserts a row into the object table. Finally, the object server sends a store response to the library server, via step 514. The library server then checks the response and sends a store response to the client, via step 516.

[0034] Thereafter the end transaction process is initiated as illustrated by FIG. 6. In the end transaction process the client receives the store response from the library server, via step 602. Then, the client sends an end transaction commit request to the library server, via step 604. The library server then sends an end transaction commit request to each object server contacted in this unit of work, via step 606.

[0035] Then, each of the object servers contacted adds a commit record to its respective transaction log and commits the database changes, via step 608. Each of the object servers then sends an end transaction response to the library server, via step 610. Each of the object servers processes their transaction logs.

[0036] After the library server receives the end transaction response from each of the object servers, the library server checks for any response errors, and sends an end transaction response to the client, via step 612. The transaction is now completed and the client inspects the results, via step 614.

[0037] Although this typical process is utilized extensively to manage data, it is oftentimes desired that multiple replicas of an object or different object servers be resident within the system. Replication provides for reliability in a variety of ways. For example, it can be utilized as part of a comprehensive data security model to provide offsite storage. In addition, replicated parts lost due to a hardware, software or administrative error can be recovered by a utility if a copy exists. It also provides for availability of objects. For business, legal or regulatory reasons (depending on locale) objects (possibly legal documents) may be required to reside on certain classes of media, such as optical.

[0038] The performance characteristics of this media may conflict with the desired access rate for a given customer. Replication to a faster media can enable keeping a copy on the legal storage medium and a copy on fast access medium over a longer defined period than normal caching practices permit. Replication also provides backup redundancy, the ability to maintain a copy at two or more sites. Higher availability of objects is also achieved via multiple peer copies.

[0039] However, in conventional content management systems replication systems have not been implemented. One way of replicating objects is to replicate an entire database in a plurality of object servers. This would greatly increase the complexity and could affect the performance of the content management system. Such a system would require significant “intelligence” in each of the object servers to identify which object server has copies such that one object server would have to be able to identify if another object server has the desired object therewithin.

[0040] Accordingly, what is needed is a system and method for object replication, which does not significantly affect the cost and efficiency of the content management system. The present invention addresses such a need.

[0041] A system and method in accordance with the present invention provides for replication in a content management system. Replication is provided by utilizing the library server to track the objects to be replicated within the system. The replication is accomplished by adding two tables to the library server, a copies table and replicate table. The copies tables includes three new parts flags. A first flag identifies an ending as a copied part. A second flag identifies that other copies of this part exist. Finally a third flag identifies that a part is not yet available.

[0051] The present invention relates generally to a content management system and particularly to a system for object replication in such a system. The following description is presented to enable one of ordinary skill in the art to make and use the invention and is provided in the context of a patent application and its requirements. Various modifications to the preferred embodiment and the generic principles and features described herein will be readily apparent to those skilled in the art. Thus, the present invention is not intended to be limited to the embodiment shown but is to be accorded the widest scope consistent with the principles and features described herein.

[0052] A system and method in accordance with the present invention provides for replication in a content management system. The library server tracks the objects to be replicated within the system. In a preferred embodiment, a three valued logic is utilized for replication of an object: (1) part does not exist; (2) part will exist; and (3) part exists. The replication is accomplished by adding two tables to the library server, a copies table and replicate table. The copies tables includes three new parts flags. A first flag identifies that a copied part exists. A second flag identifies that other copies of this part exist. Finally a third flag identifies that a part is not yet available. The flag additions also apply to the parts table.

[0053] All the operations are kept simple and between only two servers at a time using a two-phase commit protocol. This reduces the complexity and allows operations to proceed where they would be unable to proceed if requiring 3 servers and one was not available.

[0054] A content management system that includes replication in accordance with the present invention utilizes many of the same elements as described in FIGS. 1, 2 and 3. To provide replication functionality is provided in the library server. Utilizing this functionality, the entire database of the object servers does not have to be replicated, thereby minimizing the complexity and intelligence required in each of the object servers. In so doing, a replication system is provided which is simple, easy to implement and compatible with existing content management systems.

[0055] To describe the key features of the present invention in more detail refer now to the following description in conjunction with the accompanying drawings.

Library Server 501

[0056] FIG. 7 is a diagram of a library server 501 in accordance with the present invention. Library server 501 has elements that are similar to those described in the library server 101 (FIG. 3) of the conventional content management system 102 (FIG. 1). In addition to the functions already described with reference to FIG. 2, the library server 501 includes additional tables and modifications to existing tables to support replication. These changes will be described hereinbelow.

Copies Table 503

[0057] In a preferred embodiment of the copies table 500, one row of the table exists for each replicated part. Replicated parts are stored on object servers as peers and are then managed as individuals. Three new part flags are added:

[0058] IS_A_REPLICA

[0059] This flag identifies an entry as a copied part.

[0060] HAS_REPLICAS

[0061] This flag identifies that other copies of this part exist.

[0062] REPLICATION_PENDING

[0063] This flag identifies that a copy of the part is not available.

Replicate Table 505

[0064] In a preferred embodiment of the replicate table 505, one row of the table exists for each defined replication of a collection. A replication is defined from a source object server collection to a target object server collection. There is no long-term relationship. The replication information is inherited by each part at first store time. The theoretical design puts no limits on the number of replicas but the table limit is the size of a small integer and the internal implementation has a limit of 128 replicas per object server collection. The source object server collection cannot equal the target object server collection.

Object Server Table 507

[0065] In a preferred embodiment of the object server table 507 within the library server 501, one row of the table exists for each defined object server (referenced by the replicate table 505).

Collname Table 509

[0066] In a preferred embodiment of the collection (collname) table 504, one row of the table exists for each defined collection (referenced by replicate table 505).

Object Server 503

[0067] FIG. 8 illustrates an object server 503 in accordance with the present invention. Each of the plurality of the object servers 503 has similar elements to those described with the object server 103 of FIG. 3 but also includes one additional element, one additional table and a modification to the object server table to support the replication request.

[0068] The additional element comprises a replicator 510. Its function will be described below.

Replicator 510

[0069] The replicator 510 provides for movement of objects from one collection to another collection from a list of requests received on behalf of the library server. The replicator informs the library server on successful completion of these tasks. The replicator 510 moves objects from source collection to target collection.

[0070] In addition there is one new table in the object server, a replication table.

Replication Table 512

[0071] One row of the table exists for each pending replication request within the table. Actions defined by replicate type include but are not limited to, for example:

[0072] “A” pending restore from a sync utility.

[0073] “D” delete this replica.

[0074] “N” insert replica information into library server copies table. This action is used to force object replication.

[0075] “L” inform library server replication for this object is complete.

[0076] “R” replicate this object to target.

[0077] “S” restore this object from a copy.

Object Server Table 514

[0078] A collection ID is provided to accommodate replication with the object server table 514.

[0079] To illustrate how a content management system in accordance with the present invention provides for replication, an object store process, which includes replication, will be described hereinbelow.

Object Store Process

[0080] FIG. 9 is a flow chart, which illustrates an object store process, which includes replication. First, the begin transaction is initiated, via step 702. The library server searches for any replicas defined using an object server node and Collection name tuple from the replication table, via step 702. If one or more replications are defined then the HAS_REPLICAS flag is set. The library server inserts a row for each copy in the COPIES table with the IS_A_REPLICA, HAS_REPLICAS, REPLICATION PENDING flags set, via step 705. The library server name, number of replicas and an object server name and collection tuple is sent for each replica. Then the conventional steps for the store process are performed, via step 706. Thereafter the end transaction process is initiated.

[0081] In the end transaction process, the object server inserts a row for each replica, which can include in a preferred embodiment object name, library server name, target object server name, target object server collection, file size, timestamps, and an action code into the object server replication table, via step 708. Then the object server logs a start replicator event in the transaction log, via step 710. After the object store process has completed, replication processing can occur, via step 712. Replication processing is described in detail hereinbelow.

Replication Processing Phase 1

[0082] FIG. 10 illustrates the first replication-processing phase, i.e., replicating an object between a source object server to a target object server. First, a request is received by a source object server from a replicate table, via step 801. The object when replicated is sent along with request to the target object server, via step 802. Next, an object reply is received by the source object server from the target object server, via step 804. If reply is OK a row is marked as valid in the replication table source object server.

[0083] Steps 802 and 804 are repeated a predetermined number of times to sort replications for optimized access order.

[0084] Thereafter, end transaction commit signal is sent to the target server, via step 806. Finally, the end transaction reply is received by the source object server and if the reply is OK, replica is committed, via step 808.

Replication Processing Phase 2

[0085] FIG. 11 illustrates the second phase of replication; i.e., the source object server informing the library server that replication for the object is complete. First, an object store update request is sent to the library server from the source object server, via step 902. Next, update reply from the library server is received by the source object server, via step 904. If reply is valid, a row in replication table of source object server is deleted. Steps 902 and 904 are repeated to sort replications for optimized access order.

[0086] Next, an end transaction commit signal is sent to the library server, via step 906. Finally, end transaction reply is received, via step 908, and if OK, the database changes are committed and the replication is complete.

[0087] There are a plurality of replication operations supported by the above-identified system. They include but are not limited to retrieve, replacement, and delete operations. Each of these operations is described below.

Replication Operations

[0088] 1. Retrieve Operator

[0089] If primary part in parts table is unavailable and the part has the HAS_REPLICAS flag set, the library server inspects the copies table for the list of replicated parts.

[0090] 2. Replacement Operator

[0091] If the original part has copies the replacement request contains replication information to allow the distribution of the new copies. If the original is not available the Library Server can use a copy.

[0092] 3. Delete Operator

[0093] If the original part HAS_REPLICAS flag is set the copy information is delivered along with the delete request.

Conclusion

[0094] A system and method in accordance with the present invention has the following advantages:

Replication is Performed at a Part Level

[0095] Replications requests are safe and guaranteed at transaction commit time. Accordingly, replications are performed outside the transaction scope as a long running transaction and replications are done in their own transaction scope.

[0096] For example, the replication process can be controlled, using a weekly schedule; each day may have a single period of time that will allow the replication process to run.

[0097] Defined Replications are an Attribute of the Collection

[0098] The replication attributes of a collection are inherited by the entities (parts) at initial store time. This is essential to guard against circular replications. The replication attributes of a collection may be changed at any time. In a preferred embodiment, up to 128 replicas may be defined per collection.

[0099] Replicated Parts are Treated as Peers by Object Server

[0100] Replications may be added for any part and replications may be coalesced into a single part via the normal migration process.

[0101] Although the present invention has been described in accordance with the embodiments shown, one of ordinary skill in the art will readily recognize that there could be variations to the embodiments and those variations would be within the spirit and scope of the present invention. Accordingly, many modifications may be made by one of ordinary skill in the art without departing from the spirit and scope of the specification.