Distribution of physical file systems

[0002] The present invention generally relates to distributed file systems, and more particularly to distribution of physical file systems using a shared data coherency and lock control service.

[0003] Conceptually, a physical file system architecture includes system meta-data, M_sys, object meta-data, M_object, and object data. The M_sysdata describes the entire file system. For example, M_sysincludes system parameters, an i-node bitmap, a block bitmap and other file system-dependent information.

[0004] The M_objectdata describes a file or directory. In an example physical file systems, a particular M_objectsegment includes the file i-node and direct and indirect blocks. The i-node includes information such as the name of the file, access rights, update times, pointers to data blocks, and pointers to indirect, double indirect, and triple indirect blocks. The indirect blocks also contain pointers to data blocks. The file data is the data referenced and manipulated by one or more client application programs.

[0005] A distributed physical file system is generally characterized by multiple storage servers that store and serve data to client applications. The storage servers and systems that host the client applications are connected to a network. The physical file system (PFS) generally provides a locking mechanism to enforce exclusive access to some of the storage space. Specifically, the PFS coordinates locking the M_sysand M_objectdata. For example, when a client is writing to a file, the i-node is locked to prevent another client from manipulating the same file. Any operations that result in allocating or de-allocating i-node or data blocks cause the associated bitmap areas to be locked.

[0006] In order to provide acceptable performance, distributed PFSs also support local caching of file data at the systems that host the client applications. However, in order to support caching, the distributed PFS includes coherence control logic to ensure that the client applications share a consistent view of the file data.

[0007] Many distributed PFSs are proprietary, making it difficult to spread data in present file systems to distributed file systems. In addition, present distributed PFSs, for example, Frangipani, GFS, and FPFS offer a distributed lock manager and service at the kernel, which may not scale well beyond a certain number of storage servers.

[0008] A system and method that address the aforementioned problems, as well as other related problems, are therefore desirable.

[0009] In various embodiments, the invention provides client applications with access to file data in a distributed file system. Each of the client applications is coupled to a virtual file system that that provides an interface to one or more physical file systems. The virtual file system interfaces with a lock agent that runs locally with the virtual file system. The virtual file system is invoked when client applications request file access, and validation requests are submitted to the lock agent by the virtual file system in response to the file access requests. The lock agent in turn sends lock requests to the lock server. The lock server manages locks and data coherency for the physical file systems. When the lock server grants a lease, the lease and a validation code are returned to the lock agent. The “lease” is a lock that is granted for a specified period of time, and the validation code indicates whether the data in the buffer cache of the virtual file system are valid. The lock agent then returns the lease to the virtual file system, which then submits the file access request to a selected one of the physical file systems. Having the data coherency and lock service at the level of the virtual file system allows distribution of virtually any physical file system in an efficient and straightforward manner.

[0010] It will be appreciated that various other embodiments are set forth in the Detailed Description and Claims which follow.

[0011] Various aspects and advantages of the invention will become apparent upon review of the following detailed description and upon reference to the drawings in which:

[0012] FIG. 1 is a functional block diagram of a computing arrangement in which data in a physical file system is distributed, and at the virtual file system level a lock service is used for locking and coherency functions;

[0013] FIG. 2 is a flowchart of a process implemented by the virtual file system for processing read and write requests;

[0014] FIG. 3 is a flowchart of a process performed by the CHECK_GLOBAL_INODE routine in processing read/write requests from the virtual file system; and

[0015] FIG. 4 is a flowchart of a process performed by the virtual file system in processing non-idempotent file operations.

[0016] In various embodiments of the invention, conventional and/or standards-based lock service components are used in conjunction with a virtual file system. The virtual 10 file system provides a plurality of client applications with access to data in a distributed file system. The data in the distributed file system is managed by one or more physical file systems. The data concurrency and coherency is managed by one or more lock servers, and the virtual file system interfaces with the lock server(s) via a lock agent. The present invention uses locking protocols at the virtual file system level. Implementing the locking at the virtual file system level and above the physical file system level allows the same lock components to be used in a system where there are multiple, different physical file systems below the physical file system.

[0017] FIG. 1 is a functional block diagram of a computing arrangement 100 in which data in a physical file system is distributed, and at the virtual file system level a lock service is used for concurrency and coherency functions. Computing arrangement 100 includes multiple client systems 102, 104, 106, a file system control server 108, multiple lock servers 110, 112, 114, and a data server arrangement 116. The various systems and components are inter-coupled via a conventional network 118. The particular hardware and software computing arrangements on which the client and server systems are deployed are implementation dependent, and the present invention is suitable for a variety of hardware and software architectures.

[0018] Each of the client systems 102, 104, 106 hosts one or more client applications. For example, client system 102 hosts client application 122. The client applications read and/or write data that are stored in data server arrangement 116. Client application 122 invokes file access routines of virtual file system 124, which in turn invokes routines of the physical file system 126 for file access.

[0019] The virtual file system 124 is interposed between the client application 122 and the physical file system 126 to provide an abstract file interface to the client application. Thus, the client application includes generic file interface calls to the virtual file system instead of file interface calls to the specific routines of the physical file system. This allows the virtual file system to support any physical file system that adheres to the virtual file system interface. Thus, multiple physical file systems 126 can be implemented on a single client system and access for the client applications provided via a single interface. The virtual file system also provides common services, such as file data caching, for different physical file systems. An example virtual file system is that provided with the Linux operating system.

[0020] In accordance with one embodiment of the present invention, implementation of the lock and coherency protocol is left to the virtual file system. The physical file system manages the file meta-data (M_object) and the file data (from the user's perspective) and is adapted to allow the virtual file system to manage selected system meta-data (M_sys) via the control server. Example physical file systems include the ReiserFS and EXT2 systems.

[0021] In one embodiment, data server arrangement 116 is implemented with a storage area network (SAN). A SAN can be implemented with an Internet Protocol (IP) arrangement such as iSCSI, or with a fiber channel arrangement. Software such as McData's SAN Management and IBM's Tivoli is required to manage and configure a SAN.

[0022] Cache storage of object data and meta-data is provided and managed by buffer cache 132. For example, the Linux operating system includes a buffer cache.

[0023] Virtual file system 124 reads an i-node 142 of a file into its program memory when processing a file access request directed to the file. Before the i-node is used, the virtual file system calls the lock agent 144 to validate the i-node as between other client systems 104, 106 and validate the range of data referenced in the file access request. The validation request is implemented as a conventional lease request that is sent from the lock agent to the lock server.

[0024] The present invention includes a lock agent 144 and one or more lock servers 110, 112, 114, which provide object locking and coherency services to the virtual file system. Using standards-based lock logic, for example, NFS locking, eliminates re-inventing a locking mechanism. Interfacing the lock logic with the virtual file system 124 does not require the lock logic to be compatible with the physical file system(s) 126.

[0025] The lock agent 144 maps the i-node in a validation request to a lock object. Since the lock server 110 needs unique identifiers for the objects to which locks are applied, the lock agent needs to provide the identifier for an i-node for which a lock is needed. The lock agent uses information in the i-node to uniquely identify the lock object. If the lock agent 144 has a valid lease on the lock object, control is returned to the virtual file system. Otherwise, an NFS read lease or write lease request is generated and sent to one of the lock servers 110, 112, 114. The lock server is selected based on the i-node number. Each i-node has a number that is assigned by the physical file system 126 when the i-node is created. In one embodiment, the lock service for the i-nodes are distributed evenly among the lock servers, and a modulo function is used to determine the appropriate lock server.

[0026] The terms “lock” and “lease” are both used in this application. An object is said to be locked if a client has a lease on the object. A lease is defined by an object being exclusively held by a client for a certain duration.

[0027] The lock server provides a response to the lock agent that indicates whether a lease was granted. In addition, the lock server indicates whether the data in buffer cache 132 is invalid. When the lock server grants a write lease for selected blocks, leases granted for the selected blocks subsequent to expiration of the write lease will indicate to the requesters that the data are invalid. In one embodiment, the lock servers are implemented using conventional, standards-based software such as that available in NFS. It will be appreciated that in other embodiments the lock servers can be implemented using other conventional software such as that conforming to the CIFS protocol from Microsoft.

[0028] Control server 108 processes non-idempotent operations (some operations on M_sysdata) for the physical file system(s) 126. “Idempotent” operations are those for which the side-effects of N>0 identical requests are the same as for a single request, aside from error or expiration issues. Non-idempotent operations include, for example, delete, rename, truncate, etc. Since the M_sysdata is modified on only one system, concurrency control of the M_sysdata is simplified with implementation on a single system. Furthermore, the details of the underlying physical file system do not need to be addressed in constructing the control server.

[0029] FIG. 2 is a flowchart of a process implemented by the virtual file system for processing read and write requests. At step 302, the process verifies that the requested operation is permitted. For example, the process checks whether the client application has permission to perform the specified operation according to file permission tags. If the operation is permitted, the process continues at step 304, where the CHECK_GLOBAL_INODE routine is called. The CHECK_GLOBAL_INODE routine validates the i-node 142 across other client systems having access to the file and validates the requested file data. The virtual file system obtains the i-node for the referenced file from the physical file system when a file is opened.

[0030] Upon return from the CHECK_GLOBAL_INODE routine, at step 306 the virtual file system calls the physical file system with the read/write operation specified by the client application. At step 308, the response or data are returned to the calling client application to complete the process.

[0031] FIG. 3 is a flowchart of a process performed by the CHECK_GLOBAL_INODE routine in processing read/write requests from the virtual file system. At step 352, the process determines the lock object that is associated with the i-node using information in the i-node. Decision step 354 tests whether the client system has a valid lease for the lock object. The validity of a lease is determined by the duration of the lease, the time at which the lease was granted, and the current time. The lock agent 144 tracks this information for the objects for which lock services are provided. If the lease is valid, control is returned to the virtual file system. Otherwise, the process is directed to step 356.

[0032] At step 356, a read lease or write lease request is generated. At step 358, the process determines which lock server is responsible for the i-node using the i-node number. The request is then transmitted to the selected lock server at step 360. The lock server determines whether a lease request is in conflict with the current status of the referenced blocks. For example, multiple clients can concurrently hold read leases. However, if a write lease is valid, the lock server will not grant another lease until the write lease expires. When a write lease expires and a lease is subsequently granted to a client, the lock server indicates that the data in the client's cache are invalid. When the lock server receives a write lease request and there are valid read leases to other clients or a valid write lease to another client, the write lease is granted after all the valid leases expire.

[0033] At step 362, the CHECK_GLOBAL_INODE routine receives a response from the lock server indicating whether the lock was granted. At step 364, selected information in the response from the lock server is recast to an i-node structure, which is returned to the virtual file system.

[0034] The response from the lock server indicates whether the requested range of data is invalid. If so, the lock agent invokes the physical file system at step 366 to read the most recent file data into the buffer cache 132. The process then returns control to the virtual file system.

[0035] FIG. 4 is a flowchart of a process performed by the virtual file system in processing non-idempotent file operations. As described above, the non-idempotent file operations include, for example, delete, rename, truncate, etc. When a non-idempotent file operation is detected in a file access request, the request is forwarded to the control server as shown by step 402. The control server interacts with the physical file system as necessary to perform the file operation. When a response is received from the control server, the response is returned to the client application at step 404.

[0036] Embodiments of the present invention, beyond those described herein, will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. It is intended that the specification and illustrated embodiments be considered as examples only, with a true scope and spirit of the invention being indicated by the following claims.