System and method of configuring a database system with replicated data and automatic failover and recovery

[0001] The present patent application is closely related to and is being filed on even date with U.S. Ser. No. 60/741,141, George H. Claborn, et at., AUTOMATIC FAILOVER TO A STANDBY DATABASE WITH NO DATA LOSS, assigned to the assignee of the present patent application. U.S. Ser. No. 60/741,141 is incoporated by reference herein for all purposes.

[0002] Not applicable.

[0003] Not applicable.

[0004] 1. Field of the Invention

[0005] The present invention relates generally to database management systems and more particularly to a method of configuring a database management system for automatic failover from a primary database server to a standby database server and subsequent recovery of the failed primary database server.

[0006] 2. Description of Related Art

[0007] As government and business store increasing amounts of data in database systems, there are increasing demands to have such data always available, even in the face of catastrophic failure of computer hardware, network outage, disastrous data corruption, etc. To meet these requirements database system engineers have developed a number of features to have database data replicated in a number of different computer systems. Once data is replicated from one database system to another, if the first database system fails or otherwise becomes unavailable, the second database is used for processing database requests. The process of switching from an unavailable first database system to a second database system is commonly known as failover. Replication features such as those just described are available under the name Oracle Data Guard in relational database systems manufactured by Oracle Corporation of Redwood City, Calif.

[0008] FIG. 1 shows a database system that uses Data Guard to replicate data to multiple standby databases across a network. Replicated database system 101 contains primary database 103 and two standby databases 113 and 121. Primary database 103 contains database information including database tables and meta-data. Updates made to the primary database 103 are transmitted via network 105 to replication system 108, which replicates the updates in database 113 and/or to replication system 110, which replicates the updates in database 121. In both replication systems, what is transmitted via network 105 is updates in the form of redo-data 107. The redo-data is then stored in archived redo log files 109. Redo log files 109 are files that contain redo-data records. Redo-data records record data that the database system can use to reconstruct all changes made to the primary database 103, including changes that have not yet been committed (made permanent). For example, if a balance value in a bank_balance table changes, the database system generates a redo-data record containing a change vector that describes the change to the database. When the redo-data is used to recover the database system, the database system reads the change vectors in the redo-data records and applies the changes recorded in the vectors to the database.

[0009] In replication system 108, redo log files 109(i) are applied at 111 against physical standby database 113. Physical standby database 113 provides a physically identical copy of primary database 103, with on-disk database structures that are identical to the primary database 103 on a block-for-block basis. The database schema, including indexes therein, is the same. A physical standby database 113 is said to be synchronized with the primary database when all of the redo data produced by the primary database has been received in replication system 108.

[0010] In replication system 110, redo log files 109(ii) are applied against logical standby database 121. Logical standby database 121 contains the same logical information as the primary database 103, although the physical organization and structure of the data can be different.

[0011] An Oracle database system 101 using Data Guard can be run in three distinct protection modes:

• • [0012] Maximum protection
  • [0013] This mode offers the highest level of data protection. Redo-data 107 is synchronously transmitted (SYNC) to standby database system 108 or 110 from the primary database 103, and transactions are not committed on primary database 103 unless redo-data 107 is available to at least one standby database 113 or 121 configured in this mode. If the last standby database system configured in this mode becomes unavailable, processing stops on primary database 103. This mode guarantees no data loss because the primary database 103 and standby database 113 or 121 are, and remain, synchronized with each other with respect to the redo-data that is available to each.
  • [0014] Maximum availability
  • [0015] This mode is similar to the maximum protection mode, including the guarantee of no data loss at least so long as primary database 103 and standby database 113 or 121 remain synchronized with each other with respect to the redo-data that is available to each. However, if standby database system 108 or 110 becomes unavailable (for example, due to network connectivity problems), processing continues on primary database 103. Thus the primary and that standby are no longer synchronized with each other—the primary has generated redo-data that is not yet available to the standby. When the fault is corrected, standby database 113 or 121 is resynchronized with primary database 103. If there is a need to failover before the standby database is resynchronized, some data may be lost.
  • [0016] Maximum performance
  • [0017] This mode offers slightly less data protection to primary database 103, but higher potential performance for the primary than does maximum availability mode. In this mode, as primary database 103 processes transactions, redo-data 107 is asynchronously transmitted (ASYNC) to standby database system 108 or 110. The commit operation on primary database 103 does not wait for standby database system 108 or 110 to acknowledge receipt of redo-data 107 before completing write operations on primary database 103. If any standby destination 113 or 121 becomes unavailable, processing continues unabated on primary database 103. There is little impact on primary database 103 performance due either to the overhead of asynchronously transmitting redo-data or to the loss of the standby.

[0018] In Oracle Data Guard, automatic failover is termed Fast-start Failover or FSFO. Configuring a replicated database system 101 for Fast-Start Failover requires that the database administrator perform a series of discrete steps:

• • [0019] 1. Upgrade the protection mode of the database configuration to be maximum availability;
  • [0020] 2. Configure flash recovery areas for all databases in the configuration;
  • [0021] 3. Enable flashback logging on all databases in the configuration;
  • [0022] 4. Create standby redo log files for all databases in the configuration;
  • [0023] 5. Change the log transport mode of the failover target standby database to be synchronous (SYNC);
  • [0024] 6. Restart the primary, the standby, or both databases;
  • [0025] 7. Enable Fast-Start Failover in the Data Guard configuration;
  • [0026] 8. Configure the Oracle Net for database communications;
  • [0027] 9. Set the Fast-Start Failover Threshold value;
  • [0028] 10. Start the Fast-Start Failover observer process.

[0029] The steps to configure a database configuration for automatic failover are error prone, require a large amount of time, and require manipulation of the databases by hand using SQL*Plus or other programmatic interfaces. What is needed is an easy technique for configuring a database system with replicated data in a plurality of standby databases for automatic failover. It is an object of the invention to provide such a technique.

[0030] The object of the invention is attained by a novel technique for configuring automatic failover from a primary database system to a standby database system. The database systems are coupled together by a network. The technique is characterized by the use of a graphical user interface server that is coupled to the network to provide a graphical user interface to a graphical user interface client that is coupled to the network. A user inputs automatic failover configuration information to the graphical user interface client. The graphical user interface server receives the automatic failover configuration information from the graphical user interface client and stores the automatic failover configuration information in persistent storage accessible via the network. An observer that is coupled to the network and that executes independently of the primary database system and the standby database system obtains the automatic failover configuration information from the persistent storage. The observer initiates the automatic failover in accordance with the automatic failover configuration information.

[0031] Other aspects of the technique include:

• • [0032] using automatic failover status information available to the graphical user interface server in the graphical user interface;
  • [0033] when the automatic failover configuration information so indicates, enabling the observer to initiate automatic failover; and
  • [0034] the observer detects whether an automatic failover is required by pinging the primary database system and receiving an indication in response thereto that the automatic failover is required.

[0035] The automatic failover configuration information may include a designator for the primary database system, a designator for the secondary database system, a location for the observer, a threshold time that determines a minimum period of time for which the observer must be out of contact with the primary database system before the observer initiates the automatic failover, and a restart indication that indicates whether the observer should be restarted when automatic failover status information so indicates.

[0036] In still further aspects of the technique, the observer periodically determines whether the automatic failover configuration information in the persistent storage has changed, and when the observer so determines, the observer obtains the changed automatic failover configuration information. In one version of the invention, the persistent storage belongs to the primary database system.

[0037] Other objects and advantages will be apparent to those skilled in the arts to which the invention pertains upon perusal of the following Detailed Description and drawing, wherein:

[0047] Reference numbers in the drawing have three or more digits: the two right-hand digits are reference numbers in the drawing indicated by the remaining digits. Thus, an item with the reference number 203 first appears as item 203 in FIG. 2.

[0048] The following Detailed Description discloses techniques for automating configuration and monitoring of the Fast-Start Failover (FSFO) feature of an Oracle database system. FSFO itself is an improvement to the Data Guard component of the Oracle 10g Release 2 database system. The disclosed automation techniques are employed in the Data Guard component of the Oracle 10g Release 2 database system.

[0049] FIG. 2 shows a presently preferred embodiment related to the invention failing over from a primary database to a standby database. A normally functioning replicated database system is shown at 203. The replicated database system 203 has a primary database 103 and standby database 211. Additionally, an independently executing and typically remote observer process 209 is in contact with and monitoring both databases 103 and database 211, as shown by arrow 213. Note that observer process 209 may be embodied as a separately executable process on its host machine, or as a thread of execution, if that is more appropriate for its host machine. The only requirement is that the observer be an independently executing entity that is capable of connecting to the databases 103 and 211 over the network 105. Whether it is embodied as a process, a thread, or other separately executing entity, observer 209 will be referred to herein as the ‘Observer process’ or just ‘Observer’.

[0050] When the replicated database system is functioning as shown at 203, primary database 103 is forwarding redo-data 107 to redo log files 109 and the redo-data is being applied to standby database 211 (arrow 215). During normal functioning of the replicated database as shown at 203, primary database 103 fails. At 205 is shown how the replicated system fails over from failed primary database 103 to standby or failover target database 211. Because database 103 has failed such that Observer 209 no longer is in communication with database 103, Observer 209 is in communication only with database 211, as shown by arrow 217 and database 103 has ceased sending database 211 redo-data. Observer 209 automatically initiates a failover such that standby database 211 becomes the new primary database 211. Upon failover, applications that would be attached to failed primary database 103 are re-attached to the new primary database 211 instead. Modifications to the new primary database 211 are stored in redo log files in the usual fashion. At 207 is shown what happens when Observer 209 notes that database 103 has become available again. Observer 209 now has communication with both database systems, as shown by arrow 213(iii). Working together, new primary server 211 and Observer 209 recover failed primary 103 such that it may serve the new primary as its standby server. At this point, database 211 is the primary database and database 103 the standby database. Redo-data 221 flows from database 211 to database 103, as shown by arrow 219. This arrangement may continue indefinitely, or it may continue only until database 103 is synchronized with database 211, at which point database 103 may again become the primary database system and database system 211 again the standby.

Fast-Start Failover System

[0051] FIG. 3 shows the relationship between the components of a database system configured for automatic failover. Fast-Start Failover system 301 includes a primary database system 303, any number of standby database systems 305, Observer process 307, and an Oracle Management Service (OMS) 309, part of Oracle Enterprise Manager (EM). Oracle Enterprise Manager (EM) is a Web-based setup, management, and monitoring framework for Oracle enterprise database system environments, comprising multiple components including OMS 309, repository database 333, and Oracle Agents 335. Each of the components of system 301 preferably runs on its own host machine 311-317. Host machines 311 and 313 are termed in the following database servers. The components are networked together by network 319.

[0052] Each host 311 and 313 contains a primary database system 303 or standby database system 305. These database systems are termed in the following instances of an Oracle database system. The components of the database system include System Global Area (SGA) 321 which is used to share data between processes. Among the processes belonging to an Oracle database instance are database processes 322, including processes devoted to carrying out the Data Guard functionality 323 of the Oracle database. Chief among these with respect to the present invention is Data Guard Monitor process (DMON) 325. DMON 325 maintains a separate persistent store of configuration metadata 329. Metadata 329 stores a profile of the Data Guard configuration, that is, information as to which database system is primary, which database system is standby, whether or not automatic failover is enabled, the value of the FSFO Threshold, and other configuration data. A portion of this profile concerns Fast-Start Failover and is termed in the following the FSFO context.

[0053] DMON process 325 of each database instance 303 or 305 communicates with the DMONs 325 of all the other instances in the Data Guard configuration. These collectively manage and monitor all of the instances of the configuration as a single cohesive unit from the perspective of users of the database instances. DMONs 325 working together in this fashion are collectively referred to as “the Data Guard Broker”.

[0054] Each DMON 325 has its own copy of configuration metadata 329. There is a master copy that is maintained by DMON 325 of primary instance 303. DMON 325 provides a copy of that file to each standby instance's 305 DMON 325, hence all instances 303-305 in the configuration 301 have access to the same information in metadata store 329.

[0055] It is recommended that Observer process 307 be placed on a separate host 315 from other hosts 311, 313, and 317 in the configuration 301. Observer process 307 maintains a small amount of persistent storage 331, into which it records relevant information pertaining to the Data Guard configuration that it will be ‘observing’.

[0056] EM host machine 317 contains Oracle Management Service (OMS) 309 which interfaces with the Data Guard Broker's DMON processes 325. The EM OMS 309 maintains an EM repository database 333 into which is recorded information that is needed by EM to manage the FSFO configuration, including information about all of the databases that it has discovered. Oracle Agent 335 on the respective host machines communicate with the primary and standby database's DMONs and EM OMS 309, as indicated by arrows 341. EM OMS 309 works with Oracle Agent 335, a component that resides on primary server 303, standby servers 305, and Observer Host machine 315. Agent 335 allows the EM OMS to manage and monitor remote systems, and perform remote operations on those remote systems. The combination of the Data Guard Broker infrastructure (DMONs 325), EM OMS 309, and Oracle Agents 335 allow the automation of configuring, maintaining and monitoring the database servers and Observer 307. The Observer Host 315 and EM machines 317 are also connected to network 319 to which primary 311 and standby database 313 servers are connected.

[0057] In conjunction with an HTTP server that is part of EM, EM OMS 309 also functions as the server for the FSFO graphical user interface (GUI), which is the graphical user interface that is used to configure FSFO. This function is indicated by the legend “FSFO GUI server 343”. The graphical user interface utilizes the Internet http protocol. EM OMS 309 and the HTTP server provide the graphical user interface in the form of HTML pages to Web client 337, which is a workstation with a standard Web browser. Web client 337 is connected to the HTTP server via network 336, which may be a component of network 319. EM OMS 309 has a connection via network 319 to one of the machines which host a primary or standby database. In response to inputs from the user of browser 337, FSFO HTML pages made by EM OMS 309 and the HTTP server from information obtained from DMON process 325 are displayed on client 337. The user of client 337 inputs the configuration information to fields of the displayed HTML pages 339.

[0058] In overview, configuration using browser 337 works as follows:

• [0059] 1. The user of browser 337 requests an HTML page belonging to the configuration graphical user interface.
• [0060] 2. EM OMS 309 and the HTTP server construct the HTML page using current FSFO information obtained from DMON process 325 and provides the HTML page to browser 337
• [0061] 3. The user enters values in fields of the HTML page and selects another page.
• [0062] 4. The entered values are retained by EM OMS 309.
• [0063] 5. The above process of making an HTML page, sending it, receiving values from the user, retaining the values, and sending another continues until the user indicates that the configuration process is finished.
• [0064] 6. At that point, EM 309 propagates the updated FSFO information to metadata 329 in servers 303 and 305, where it is persistently stored.
  When the Observer Starts Up:

[0065] Oracle Agent 335 located on Observer Host 315 starts Observer 307 and gives it a network connect string that allows Observer 307 to connect to the primary database server 303. The connect string was provided to Oracle Agent 335 by EM OMS 309 which has this information stored in EM repository database 333.

[0066] Observer 307 uses this connect string to connect to primary database server 303. Observer 307 registers itself with primary server 303's DMON 325, passing information about itself to the server. Note that until this occurs, primary database server 303 has no information about, and is unaware of, Observer 307.

[0067] Registration also includes an authentication step. If Observer 307 is authenticated, its registration information (location of Observer Host 315, etc.) is recorded by primary's 303 DMON process 325 in DG metadata file 329, and a positive acknowledgement is passed back to Observer 307. The acknowledgement includes an observer identifier that identifies the current Observer. Each time an Observer is deregistered and a new Observer registered, the observer identifier is incremented.

[0068] Observer 307 then queries primary database server 303 for FSFO context information by executing a “GET_FSFO” command. Upon receipt of this command, primary database server 303 will pass back to Observer 307 information from DG metadata 329 including:

• • [0069] Whether or not FSFO is enabled at the moment.
  • [0070] Which instance is the primary database system and how to reach it

[0071] If FSFO is enabled, then additionally

• • [0072] Which standby instance 305 is the FSFO target standby database system and how to reach it
  • [0073] The current value of the FSFO Threshold
  • [0074] The Metadata Incarnation Value (MIV), which is a monotonically increasing number that is incremented for every change that is ever made to the DG metadata 329
  • [0075] Other FSFO related information

[0076] Observer 307 stores this information in its local “Observer Storage” file 331.

[0077] If FSFO has not been enabled, Observer 307 will periodically reattempt the gathering of FSFO information from primary database server 303 and will not actually be “observing” until it receives an indication from the primary database server 303 that FSFO has now been enabled.

Observing—Steady State

[0078] Once these initial steps have occurred and FSFO has been enabled by EM at primary database server 303, Observer 307 begins “observing”. This consists of periodic pings to primary database server 303 and FSFO target standby server 305, specifically to the Broker's DMON processes 325 in the primary database server 303 and standby database server 305. Each ping consists of a message sent by Observer 307 to the database server that includes the following:

• • [0079] The Observer ID that Observer 307 received from server 303 upon successful registration;
  • [0080] the Observer's current MIV; and
  • [0081] the current role, either primary or standby, of the server being pinged.
    The current MIV and the current roles were received from server 303 in response to the Observer's most recent “GET_FSFO” command.

[0082] From Observer 307's point of view, the Observer's pinging accomplishes the following purposes:

• • [0083] If the server being pinged fails to respond, the server may be down or connectivity with the Observer may be lost; in the case of primary database server 303, that means that a failover condition may have arisen.
  • [0084] If the server does respond, the response may indicate
    • [0085] that there are no problems;
    • [0086] that the Observer does not have the current FSFO context;
    • [0087] in the case of primary database server 303, that a failover condition has arisen; or
    • [0088] if the current Observer ID on the server is not equal to the Observer's observer ID, that the Observer has been deregistered and is no longer the current Observer. In this case, the Observer process terminates itself.

[0089] From the point of view of servers 303 and 305, the pinging confirms the current role of the server and confirms that Observer 307 is the current observer, is running, and that there is connectivity between the Observer and the server.

[0090] The function that responds to the ping in server 303 or 305 revalidates the authentication code to ensure that this is the one and only Observer that had successfully registered itself with the server. The function then compares the MIV that was supplied by Observer 307 to the current MIV maintained by the server. If the MIV supplied by the Observer is equal to the MIV that is current on the server, the Observer has the current FSFO context. If that is not the case, the server indicates to the Observer that the Observer does not have the current FSFO context and the Observer fetches the current FSFO context from the server. To do this, Observer 307 issues the “GET_FSFO” command to the server in order to refresh the Observer's FSFO context, including a new MIV. The FSFO context is stored in Observer Storage 331, and Observer 307 resumes steady-state pinging. The mechanism for detecting whether Observer 307 has the current FSFO context is also how Observer 307 becomes aware of changes to FSFO context resulting from user inputs at browser 337 that are made while Observer 307 is running.

[0091] Primary server 303 then performs a number of internal checks to determine if there is a problem with the server's database that would warrant an immediate failover. It also checks for any conditions that would specifically prevent an automatic failover, such as the knowledge that primary 303 and standby 305 are not currently synchronized in terms of the redo-data that is currently available to each. After doing all of these internal checks, a primary server 303 sends a response back to Observer 307. The server 303 finishes by taking note of when the ping occurred so that a record exists that Observer 307 appears to be alive. If the response indicates that an immediate failover is necessary, Observer 307 automatically initiates the failover if that is possible.

[0092] The recording of most recent pings from Observer 307 by primary server 303 and standby server 305 collectively serve to reassure the database servers that the Observer is alive and well. Should some calamity occur to the Observer such that the age of most recently recorded pings becomes excessive, primary server 303 makes a warning message indicating an unobserved condition concerning Observer 307 available to EM OMS 309, which may then take steps to restart failed Observer 307. Such restarts in response to unobserved conditions will be discussed in detail later.

FSFO Property Changes and Their Effect on the Observer

[0093] While FSFO is enabled, it is possible to change any FSFO-related property from the EM OMS 309, via browser 337. GUI-driven changes—such as an adjustment of the FSFO Threshold—are automatically picked up on running Observer 307 via the mechanism described in the previous section. Some GUI-driven changes, such as a change of the FSFO target standby database 305, require that FSFO be disabled and re-enabled. However, EM makes all changes transparent from the user's standpoint, in that the “right thing” happens under any circumstance. If FSFO must be disabled and re-enabled as part of a configuration change, EM OMS 309 does it automatically when it applies the changes received from the user at browser 337.

[0094] It is also possible from the GUI to start and stop a running Observer 307, or move the Observer from its current host 315 to a different host. EM OMS 309 accomplishes an Observer 307 move by communicating to the primary server's 303 DMON process 325 that it should convey to existing Observer 307 that its registration has been revoked and that it should shut itself down. This act of de-registering the existing Observer completes immediately. The existing Observer 307 will learn from the primary server 303 as of its next ping that its registration has been revoked, and will then shut itself down. Thus there is no need for EM OMS 309 to await the occurrence of the next ping and the actual Observer shutdown. EM OMS 309 immediately runs a remote operation via Oracle Agent 335 on the new host to start a new Observer process 307. Again, this is a transparent operation from the user's perspective. Any FSFO-related property can be changed from EM from the same GUI used to perform the initial FSFO setup.

Automatic Restart of the Observer in Response to an Unobserved Condition

[0095] As already pointed out, a combination of EM OMS 309, Oracle Agent 335 on primary host 311, and primary database DMON process 325 is used to implement a framework that continuously monitors the status of the FSFO configuration. The Oracle Agent on the primary host connects to the primary database DMON process at regularly scheduled intervals, obtains current status information for the FSFO configuration, and forwards this information to the EM OMS. If the status information indicates that the FSFO configuration is unobserved—which may mean the Observer is no longer running or the Observer machine has been rebooted—the EM OMS raises a user-visible alert on a web page provided by EM for monitoring general status of the primary database. If the user specified the optional Observer restart capability when configuring FSFO (via check-box 515 in the Data Guard Fast-Start Failover GUI, FIG. 5), the EM OMS will automatically dispatch a remote job to Oracle Agent 335 on Observer host 315 to restart the Observer process in an attempt to clear the unobserved condition in the FSFO configuration. If the Observer restart attempt is successful, the EM OMS initiates an immediate reacquisition of the current FSFO status information via the Oracle Agent and primary database DMON process. The current FSFO status information will indicate the unobserved condition is no longer present, and the status displayed by EM for primary database 303 will be updated to reflect normal FSFO operation.

Observer Detects Need for Failover

[0096] There are two basic ways for Observer 307 to decide that a failover should be attempted. The first has already been described. The function on the primary server 303 that responds to the ping will perform certain internal checks to determine if any conditions exist that would require a failover. In a preferred embodiment, one such condition is datafile corruption. In other embodiments, there may be other conditions that require a failover. Conditions in primary database 303 which require a failover are termed in the following failover conditions. The existence of a failover condition is passed back to Observer 307, which immediately begins the failover sequence.

[0097] Another failover condition is lack of a timely response to a ping of primary server 303. If the ping that Observer 307 sends to the server does not receive a timely response, Observer 307 will retry the ping according to a retry algorithm that factors in the value of the FSFO Threshold. If no contact with the primary can be made before that threshold expires, Observer 307 assumes that primary database 303 has failed and begins the failover sequence.

[0098] Note, Observer 307 doesn't actually perform the failover operation. Only standby database 305 can do that with assistance from Observer 307 for certain components of the failover operation that can only be done by a client. Observer 307 will ask standby server 305 if it is “ready to failover”. Standby database 305 evaluates the request and responds to Observer 307. It may respond “No” because, unlike Observer 307, it's possible that standby database 305 is actually being contacted by Primary database 303 on a timely basis. This would be the case where network connectivity between Primary 303 and Observer 307 is down, but is up between Primary 303 and Standby 305.

[0099] After evaluating the relevant conditions, Standby 305 either rejects the request or acknowledges that it can do the failover. If Standby 305 rejects the request, Observer 307 continually retries until either the conditions provoking the failover attempt get resolved, for example, connectivity between Primary 303 and Observer 307 being restored, or when Standby 305 eventually acknowledges that it can do the failover, in which case Observer 307 then tells Standby 305 to proceed with the failover.

[0100] Observer 307 records in its persistent storage 313 that a failover has begun. Observer 307 awaits status from Standby 305 that the failover operation has indeed completed, i.e., Standby 305 has now become Primary 305, and that the Data Guard metadata 329 has been updated accordingly and the MIV incremented. At this point Observer 307 begins pinging the new Primary 305 which, as described above, will respond to the Observer that the Observer no longer has a current MIV. Observer 307 then issues the “GET_FSFO” command to new Primary server 305 to refresh its copy of the FSFO context information.

Recovering the Failed Primary

[0101] Following the failover, Observer 307 continually pings the old Primary 303. Once connectivity is reestablished between old Primary 303 and both Observer 307 and Standby 305 (either by restarting a failed Primary 303 after it crashes or resolution of a network outage that led to the failover), the old Primary 303 learns from the role value in Observer 307's ping that the old Primary is no longer the FSFO primary database and indicates to Observer 307 that it is ready to become the new FSFO target standby.

[0102] Observer 307 will then send a “REINSTATE” command to new Primary 305 to begin the process of recovering the old Primary 303 to the point where it can serve as a standby to the new primary. In response to the “REINSTATE” command, old Primary 303 uses the Oracle flashback feature to roll its redo-data back to the point in time at which the failover took place. From that point, database 303 initiates operation as a standby database for new Primary 305. This completes the full FSFO cycle, in that the FSFO configuration is now enabled for failover in the opposite direction from which it was originally configured, and the new configuration is automatically and seamlessly recognized by Observer 307 and EM OMS 309.

Configuring a Fast-Start Failover System

[0103] A Database Administrator or other suitably privileged user can configure and enable Fast-Start Failover from a Web browser that is connected via the network to EM OMS 309, which in turn is connected to any database 303 or 305 in the Fast-Start Failover system 301. Enabling Fast-Start Failover does not trigger a failover. Instead, it allows Observer process 209 to begin observing primary database 103 and standby database 211 and automatically initiate a failover should a failover condition arise for the primary database.

[0104] Fast-Start Failover is configured for a given Data Guard configuration. When configuring Data Guard, a user can specify a primary database and a set of up to 9 standby databases. The Oracle Enterprise Manager Data Guard Fast-Start Failover GUI configures a preexisting Data Guard configuration for Fast-Start Failover. FIG. 8 shows a page 801 that displays the current status of a Data Guard configuration that has been selected by the user. Included in the current status of the Data Guard configuration is its Fast-Start Failover Status, shown at 804 in FIG. 8. In FIG. 8, FSFO is enabled, as indicated by status field 804. Field 804 also specifies the current FSFO standby database system. When FSFO is not enabled, FSFO field 804 contains a “Disabled” link. The database administrator clicks the “Disabled” link to invoke the FSFO GUI. This action will cause the configuration web page of FIG. 5 to appear.

[0105] FIG. 5 shows a Fast-Start Failover configuration page 501. Page 501 contains a number of data entry fields. The first is Target Database Selection 503, a table that contains rows of databases available for configuration as the FSFO target, i.e., standby databases. The columns of the table are Select, Name, Role, and Log Transport Mode. At 503, database dgps1 is selected in the “select” column and is therefore the prospective or current FSFO target. (There can be up to nine candidate target standby databases in the table, only one of which can be specified as the FSFO target.) The name of the selected database is dgps1. The role of the database is physical standby and the current Log Transport Mode for forwarding redo-data to the database is ASYNC (which will be automatically changed to SYNC by EM during the setup procedure). A FSFO Threshold duration of time representing the amount of time the primary 103 may be safely out of contact with the Observer 209 and the standby 211 is input at 511. The duration of time is further defined by a unit of time 513. A check-box input 515 indicates whether the Observer 209 should automatically be restarted if the Observer process is found to not be running on its host machine. Standby Redo Log Files section 517 provides options for creation of standby redo files that must exist in a standby database system. Because a primary may end up as a standby after a failover, both the primary 103 and standby 211 must have standby redo files. EM automatically calculates the correct complement of standby redo log files—including sizes and number—and displays the result in table 519. The table shows the database name and host where the files will be created, the size of the files and the location of the files. The location may be user configurable, or may be managed automatically by the Oracle-managed files feature of the Oracle database. (The example in FIG. 5 shows the latter case.)

[0106] To specify the location of the Observer process, the user clicks the Set Observer button 509 to request a page to configure the Observer process. FIG. 4 shows a page for the configuration of the Observer 209. Field 403 allows specification of the host machine upon which the Observer process will be run. Field 405 allows the administrator to input the file system location of the Oracle home directory from which the Observer will be invoked. If an Observer has already registered with the primary server 303, fields 403 and 405 contain the names of the existing Observer's host and Oracle home directory.

[0107] FIG. 6 shows a user interface for enabling flashback logging. Flashback is a utility provided by Oracle for reading the history of information in the database system from the redo-data, and is required to enable FSFO. Specifically, flashback logging is a technique that is used to help recover a failed primary database after a failover has occurred such that the failed primary may become a fully functional standby database serving to protect the new primary database. A filesystem directory for a flashback recovery area is shown in input field 603. Input field 605 allows the administrator to enter the size of the Flash Recovery Area in mega-bytes. Field 607 allows the administrator to enter the length of time that flashback data will be retained and field 609 defines the unit of time. A further page will be displayed to ask the administrator if the configuration should proceed.

[0108] FIG. 9 is the HTML page that the user employs to indicate that he or she is finished configuring Fast-Start Failover and wishes to enable Fast-Start Failover with the changes. Page 901 contains panels and buttons. The panel 903 asks if you want to enable Fast-Start Failover with the new configuration. Buttons 905 can input a “yes” or “no” to the question.

[0109] When the user pushes the “yes” button in page 901, EM OMS 309 responds with page 701 in FIG. 7. Page 701 shows how the automated setup of a Fast-Start Failover configuration is progressing. Status bar 703 displays the forward progress of the automated setup. Status fields 705-715 use check mark icons to indicate those portions of the set up which have been completed and an arrow to indicate the portion of the set up which is currently being done. The remaining portions have not been completed. Thus, at this point, standby redo log files have been created on the primary and standby databases (705) and the protection mode of the databases has been upgraded to maximum availability, which is the protection mode that is required for automatic failover to be enabled (707). Currently, the primary and standby databases are being restarted with their new configuration (709). The remaining portions of the set up have not yet been completed. They are enabling Fast Start Failover (711), starting the Observer process (713), and completion of the setup (715).

[0110] Once Fast-Start Failover has been successfully configured, Observer 209 continuously monitors the configuration to ensure that the primary database is available. If Observer 209 infers that a problem exists due to the lack of a response from the primary database, Observer 209 attempts to reconnect to the primary database within the time specified by the FSFO Threshold. If the problem is not remedied within the specified time or the primary database has conveyed directly to the Observer that a failover condition exists, and the target standby database is ready for failover, the Observer immediately invokes a failover at the standby database. Because a Fast-Start Failover is automatic and fast, an administrator may not be aware it has happened until the administrator notices that post-failover recovery at the failed primary database is needed or is already occurring.

[0111] The current state of the Data Guard configuration, including the FSFO target database, is shown in FIG. 8. The Enterprise Manager Data Guard overview page 801 has a section 803 that includes the Data Guard Status. Panel 803 further indicates the protection mode of the Data Guard configuration, the Fast-Start Failover mode (Enabled or Disabled), the name of the Fast-Start Failover target standby database, and the location of the Observer process. Panel 805 contains information about the primary database, including the database name, the host that it is running on, its status, the current log, and a field to allow editing of the database properties. Panel 807 graphically and numerically shows the lag between the application of redo-data to the primary and the standby database. Panel 809 shows the standby database, its name, Data Guard status, role in the Fast-Start Failover configuration, the number of redo log files received by the database, the number of redo log files applied to the database, and the amount of time required to execute an actual failover to the standby database. The Data Guard overview page 801 graphically shows whether a database is ready for failover. As long as redo-data from the primary database still needs to be received by the standby database, the failover operation cannot be performed automatically

[0112] The foregoing Detailed Description has disclosed to those skilled in the relevant technologies how to make and use the techniques of the invention for configuring automatic failover and has further disclosed the best modes presently known to the inventors of so doing. It will be immediately apparent that many variations on the techniques disclosed herein are possible. A graphical user interface like the one disclosed herein may be used with any database backup and recovery system that replicates data across a number of servers and uses a third party (observer) as a broker of the indication of the state of the entire system. The graphical user interface may have any form which permits the user to input the necessary configuration information and will generally have a form which conforms to the conventions for graphical user interfaces that are used in the system the interface is being used with. The information that is necessary to configure automatic failover will also depend on the kind of system that is being configured. Further, many different arrangements may be made to provide the observer with the information obtained via the configuration GUI and interactions between the observer and the standby database system may be different from the ones disclosed herein. For all of the foregoing reasons, the Detailed Description is to be regarded as being in all respects exemplary and not restrictive, and the breadth of the invention disclosed here in is to be determined not from the Detailed Description, but rather from the claims as interpreted with the full breadth permitted by the patent laws.