Method and apparatus for highly-available processing of I/O requests while application processing continues

[0002] 1. Field of the Invention

[0003] The present invention generally relates to operating systems for fault-tolerant distributed computing systems. More particularly, the present invention relates to a system and method that supports asynchronous I/O requests that can switch to a secondary server if a primary server for the I/O request fails.

[0004] 2. Related Art

[0005] As computer networks are increasingly used to link stand-alone computer systems together, distributed operating systems have been developed to control interactions between multiple computer systems on a computer network. Distributed operating systems generally allow client computer systems to access resources or services on server computer systems. For example, a client computer system may access information contained in a database on a server computer system. However, when the server fails, it is desirable for the distributed operating system to automatically recover from this failure without the user client process being aware of the failure. Distributed computer systems possessing the ability to recover from such server failures are referred to as “highly available systems,” and data objects stored on such highly available systems are referred to as “highly available data objects.”

[0006] To function properly, a highly available system must be able to detect a failure of a primary server and reconfigure itself so that accesses to objects on the failed primary server are redirected to backup copies on a secondary server. This process of switching over to a backup copy on the secondary server is referred to as a “failover.”

[0007] Asynchronous I/O requests are particularly hard to implement in highly available systems. Asynchronous I/O requests allow a process to initiate an I/O request and continue processing while the I/O request is in progress. In this way, the process continues doing useful work—instead of blocking—while the I/O request is in progress, thereby increasing system performance. Unfortunately, a process typically has little control over when the I/O request completes. This lack of control over the timing of I/O requests can create problems in highly available systems, which must be able to recover from primary server failures that can occur at any time while an asynchronous I/O request is in progress.

[0008] What is needed is a highly available system that supports asynchronous I/O requests that can switch to a secondary server if a primary server for the I/O request fails.

[0009] One embodiment of the present invention provides a system that allows an I/O request to proceed when a primary server that is processing the I/O request fails, and a secondary server takes over for the primary server. Upon receiving an I/O request from an application running on a client, the system stores parameters for the I/O request on the client, and sends the I/O request to the primary server. Next, the system allows the application on the client to continue executing while the I/O request is being processed. If the primary server fails after the I/O request is sent to the primary server, but before an I/O request completion indicator returns from the primary server, the system retries the I/O request to the secondary server using the parameters stored on the client. The I/O request may originate from a number of different sources, including a file system access, an I/O request from a database system, and a paging request from a virtual memory system. In a variation on the above embodiment, the act of storing the parameters for the I/O request on the client includes creating a distributed object defined within a distributed object-oriented programming environment, and sending a reference to the distributed object to the primary server to be stored on the primary server. This causes a distributed operating system to keep track of the reference so that if the primary server fails, the reference count on the distributed object drops to zero and the distributed operating system notifies the client that the distributed object is unreferenced. This allows the client to deduce that the primary server has failed.

[0010] One aspect of the above embodiment involves locking a page of memory associated with the I/O request before sending the I/O request to the primary server. This ensures that the page remains unmodified in case the I/O request needs to be retried. The page is ultimately unlocked when the I/O request completes and the primary server informs the client of the completion of the I/O request.

[0011] Still other embodiments of the present invention will become readily apparent to those skilled in the art from the following detailed description, wherein is shown and described only the embodiments for the invention by way of illustration of the best modes contemplated for carrying out the invention. As will be realized, the invention is capable of other and different embodiments and several of its details are capable of modifications in various obvious respects, all without departing from the spirit and scope of the present invention. Accordingly, the drawings and detailed description are to be regarded as illustrative in nature and not as restrictive.

[0012] FIG. 1 illustrates a distributed computing system in accordance with an embodiment of the present invention.

[0013] FIG. 2 illustrates functional components within a client and a primary server involved in implementing highly available asynchronous I/O operations in accordance with an embodiment of the present invention.

[0014] FIG. 3 illustrates part of the internal structure of a callback object in accordance with an embodiment of the present invention.

[0015] FIG. 4 is a flow chart illustrating some of the operations involved in performing an asynchronous I/O operation in accordance with an embodiment of the present invention.

[0016] FIG. 5 is a flow chart illustrating some of the operations involved in recovering from a failure during an asynchronous I/O operation in accordance with an embodiment of the present invention.

[0017] Description of Distributed System

[0018] FIG. 1 illustrates a distributed computing system in accordance with an embodiment of the present invention. This distributed computing system includes network 110, which is coupled to client 102, primary server 106 and secondary server 108. Network 110 generally refers to any type of wire or wireless link between computers, including, but not limited to, a local area network, a wide area network, or a combination of networks. In general, a client, such as client 102, refers to an entity that requests a resource or a service. Correspondingly, a server, such as primary server 106 or secondary server 108, services requests for resources and services. In certain cases, the client and server for an object may exist on the same computing node. In other cases, the client and server exist for an object on different computing nodes. In the example illustrated in FIG. 1, client 102, primary server 106 and secondary server 108 exist on separate computing nodes.

[0019] Primary server 106 and secondary server 108 are coupled to storage device 112. Storage device 112 includes non-volatile storage for data to be accessed by primary server 106 and secondary server 108. Although FIG. 1 illustrates direct communication links from storage device 112 to primary server 106 and to secondary server 108, these communication links may actually be implemented by messages across network 110, or another independent network.

[0020] In one embodiment of the present invention, the operating system for the distributed computing system illustrated in FIG. 1 is the Solaris MC operating system, which is a product of Sun Microsystems, Inc. of Palo Alto, Calif. The Solaris MC operating system is a UNIX-based operating system. Hence, in describing the present technology, UNIX terminology and concepts are frequently used. However, this usage is for purposes of illustration and is not to be construed as limiting the invention to this particular operating system.

[0021] In the illustrated embodiment, client 102 includes highly-available asynchronous I/O system 104, which may be implemented as a library and which facilitates asynchronous I/O operations that can be directed to a secondary server 108 when a primary server 106 that is processing the I/O request fails. For example, assume client 102 has an outstanding I/O request to primary server 106. If primary server 106 fails while this I/O request is outstanding, client 102 is eventually notified by the distributed operating system that primary server 106 failed. This causes client 102 to retry the failed I/O request on secondary server 108. Note that secondary server 108 is capable of processing the same I/O request because it also can access storage device 112.

[0022] Functional Components Involved in Asynchronous I/O

[0023] FIG. 2 illustrates functional components within client 102 and primary server 106, which are involved in implementing highly available asynchronous I/O operations in accordance with an embodiment of the present invention. In the illustrated embodiment, client 102 from FIG. 1 contains user process 204. User process 204 makes an I/O request by calling an I/O function from system library 208. System library 208 includes functions that communicate with proxy file system (PXFS) 212.

[0024] User process 204 may include any process in client 102 that is capable of generating an I/O request to primary server 106. This includes, but is not limited to a user process that generates a file system reference, a database process that generates a database access, and a paging system that generates a page reference. Although FIG. 2 presents a “user” process for illustrative purposes, in general, any “user” or “system” process may generate the I/O request.

[0025] System library 208 includes a collection of functions that implement various system calls, including functions that carry out I/O requests. An I/O routine within system library 208 typically converts a user-level system call from user process 204 into a kernel-level system call to perform the I/O operation.

[0026] Proxy file system (PXFS) 212 is part of a highly available distributed file system that supports failovers from a primary server 106 to a secondary server 108 when primary server 106 fails. In the illustrated embodiment, PXFS 212 includes callback object 214, which contains information related to an associated asynchronous I/O operation.

[0027] PXFS 212 within client 102 communicates with PXFS 222 within primary server 106. Note that PXFS 212 within client 102 and PXFS 222 within primary server 106 are different parts of the same distributed file system. PXFS 222 includes distributed object pointer 223, which is a reference to a callback object 214 within client 102. In fact, distributed object pointer 223 creates a distributed reference 225 to callback object 214. If primary server 106 fails, distributed reference 225 disappears. This causes the count of active references to callback object 214 to drop to zero, which causes the distributed operating system to notify PXFS 212 on client 102 that callback object 214 is unreferenced. Since primary server 106 is the only entity holding a reference to callback object 214, client 102 can conclude that primary server 106 has failed.

[0028] Primary server 106 additionally contains storage device driver 224, which receives commands from PXFS 222 and, in response to these commands, accesses storage device 112.

[0029] Storage device driver 224 communicates with storage device 112 in order to perform specified I/O operations to storage device 112.

[0030] FIG. 3 illustrates part of the internal structure of a callback object 214 in accordance with an embodiment of the present invention. The embodiment illustrated in FIG. 3 includes pointer to I/O request 302 and I/O request status 304. Pointer to I/O request 302 is a pointer to an I/O request object containing information related to the pending I/O request. This pointer allows the pending I/O request to be retried on secondary server 108 when primary server 106 fails. Although the embodiment of callback object 214 illustrated in FIG. 3 takes the form of an object defined within an object-oriented programming system, in general, any type of data structure that stores equivalent data items may be used. Note that client 102 creates and maintains a separate callback object 214 for each pending I/O request.

[0031] Asynchronous I/O Process

[0032] FIG. 4 is a flow chart illustrating some of the operations involved in performing an asynchronous I/O operation in accordance with an embodiment of the present invention. These operations are described with reference to the functional components illustrated in FIG. 2. First, user process 204 (from FIG. 2) makes an I/O request (step 402). As mentioned previously, this I/O request may include any type of I/O request to storage device 112 (from FIG. 2), including a file system access, an I/O request from a database system, or a paging request from a virtual memory system. Next, I/O request 206 executes a system call from within system library 208 (step 404). This system call generates a kernel system call 210, which accesses proxy file system (PXFS) 212 on client 102.

[0033] In processing the I/O request, PXFS 212 creates and stores callback object 214 (step 406). In doing so, PXFS 212 initializes pointer to I/O request 302 from FIG. 3 to point to an associated I/O request object (in order to allow the I/O request to be retried), as well as setting I/O request status 304 to “in progress.” (step 408)

[0034] Next, PXFS 212 makes an invocation to primary server 106 (step 410). This invocation includes a reference to callback object 214. In response to the invocation, PXFS 222 in primary server 106 stores the reference to callback object 214 as distributed object pointer 223. This causes a distributed operating system to keep track of a distributed reference 225 to callback object 214. If primary server 106 fails, the distributed operating system notifies client 102 that callback object 214 is unreferenced. Since primary server 106 is the only entity holding a reference to callback object 214, client 102 can conclude that primary server 106 has failed; client 102 can then take appropriate action.

[0035] Next, PXFS 222 on primary server 106 calls storage device driver 224 to start an I/O operation (step 412). Storage device driver 224 initiates the I/O operation by sending a command to storage device 112. At this point the invocation returns to client 102 (step 414). Next, PXFS 212 on client 102 then forwards the return to user process 204, which allows user process 204 to continue processing (step 416). User process 204 can thereby perform useful work instead of waiting for the I/O operation to complete. In the mean time, the I/O operation continues processing and completes at some undetermined time in the future.

[0036] When the I/O request is completed by storage device 112, storage device 122 sends an interrupt to storage device driver 224. In response to the interrupt, storage device driver 224 calls I/O done function (step 420). This function causes primary server 106 to notify client 102 that the I/O request is complete (step 422). Furthermore, if the I/O request was for a read operation, data read from storage device 112 is passed back from client 102 to primary server 106 at this time. Next, PXFS 212 on client 102 sets I/O request status 304 to “done,” and unlocks any pages that were locked during the I/O request (step 424). During an I/O operation, the pages remain locked until the I/O operation completes to prevent the pages from being swapped out or deleted when the I/O is in progress. Note that the pages are locked at some time before the I/O operation is sent to primary server 106 in step 410. Also note that the interrupt may complete (in step 420) at any time after the I/O request starts in step 412, because the I/O request is executing on a separate thread. Hence, step 420 may follow any of states 412, 414 and 416 as is indicated by the dashed lines.

[0037] Next, primary server 106 releases distributed reference 225 to callback object 214 (step 426). Since there is only one reference to callback object 214, when distributed reference 225 is released, callback object 214 is unreferenced. The distributed operating system will eventually detect this fact and client 102 will receive an “unreferenced” message on callback object 214 (step 428). In response to this unreferenced message, client 102 examines I/O request status 304 within callback object 214. If I/O request status indicates the I/O request is complete, client 102 finally deletes callback object 214 (step 430). At this point, the I/O operation is complete.

[0038] Failure Recovery

[0039] FIG. 5 is a flow chart illustrating some of the operations involved in recovering from a failure during an asynchronous I/O operation in accordance with an embodiment of the present invention. If primary server 106 fails before the I/O invocation is made to primary server 106 (in step 410 of FIG. 4), the invocation is automatically retried to secondary server 108 from FIG. 1 by the distributed operating system (step 502).

[0040] Next, if primary server 106 fails after the I/O invocation to primary server 106 is made in step 410, but before the invocation returns to client 102 in step 414, the replica framework of the distributed operating system performs a retry of the I/O request to secondary server 108 (step 504). In the illustrated embodiment, the replica framework is part of the Solaris MC operating system produced by SUN Microsystems, Inc. of Palo Alto, Calif. However, other operating systems can use analogous mechanisms to perform the retry. Note that before the invocation reaches primary server 106, distributed reference 225 to callback object 214 may not exist. Hence, client 102 cannot rely on receiving an unreferenced notification in case primary server 106 dies.

[0041] Next, if primary server 106 fails after the invocation returns to client 102 (in step 414), but before primary server 106 notifies client 102 of completion of the I/O operation in step 422, a series of events take place (step 506). When primary server 106 dies, the reference count maintained by the distributed operating system for callback object 214 drops to zero. This is detected by the distributed operating system, and client 102 receives an unreferenced message on callback object 214 from the distributed operating system. In response to this unreferenced message, PXFS 212 in client 102 checks to make certain that I/O request status 304 within callback object 214 is set to “in progress.” If so, PXFS 212 retries the I/O request to secondary server 108 using the original request structure indexed by pointer to I/O request 302 within callback object 214. Note that user process 204 is not aware of this failure.

[0042] If primary server 106 fails after primary server 106 notifies client 102 of completion of the I/O request, client 102 sets I/O request status 304 to “done.” (step 508). Primary server 106 additionally cleans up the data structures associated with the I/O request. This may include unlocking pages involved in the I/O request and deleting callback object 214.

[0043] Thus, the present invention supports highly-available asynchronous I/O requests that can switch to a secondary server if a primary server for the I/O request fails.

[0044] While the invention has been particularly shown and described with reference to embodiments thereof, those skilled in the art will understand that the foregoing and other changes in form and detail may be made therein without departing from the spirit and scope of the present invention.