Control of concurrent access to a partitioned data file

[0001] The present invention generally relates to controlling concurrent access to file data.

[0002] When a software application is first developed, the data it references may be organized in separate files. From the application developer's perspective, this organization may be convenient when the application is in its infancy. However, as an application evolves and requirements change, many more files may be introduced into the application data set. At some point, the number of files may prove to be a burden from the developer's perspective. Thus, over time the notion of having separate files may progress from being desirable to being problematic.

[0003] While potentially problematic in some regards, maintaining a data set in separate files may be advantageous for supporting requirements for concurrent access to the data set. If the data are distributed across many files, management of concurrent access may be of lesser concern if concurrent access is rarely needed to the data in a single file.

[0004] Given various complexities associated with programmatically tracking a large number of files in a data set, certain circumstances may call for consolidating the data set into a single file or just a few files. However, some applications may have multiple processes which require concurrent access to the data set. If the data set is concentrated in just a few files and concurrent access is frequent, then efficient management of concurrent access may be of greater concern.

[0005] Data management software, for example, a database management system, may be used to control concurrent access to a data set. However, some applications may be unsuitable for implementation using data management software because this software may be cumbersome to install and maintain and may introduce significant overhead.

[0006] A system and method that address the aforementioned problems, as well as other related problems, are therefore desirable.

[0007] The various embodiments of the invention provide for controlling concurrent access to a data file. The data file is organized into a plurality of partitions, and a lock file that is accessible to the client applications is persistently stored. The lock file includes lock objects that correspond to the plurality of partitions, and each lock object indicates a lock status of the corresponding partition. The client applications perform an atomic file system operation on a persistent file system object when exclusive access is sought to the lock file. Successful completion of the atomic file system operation indicates that a client application has exclusive control over the lock file, and the lock file may thereafter be updated, the atomic operation undone, and access made to a selected partition.

[0008] It will be appreciated that various other embodiments are set forth in the Detailed Description and Claims which follow.

[0015] The various embodiments of the invention support concurrent access to a persistent, partitioned data file. A lock file is used to manage the status of locks associated with the partitions of the data file, and an atomic operation on a persistent file system object is used as the semaphore to control access to the lock file. Each application seeking access to the lock file must first obtain the semaphore. Use of an atomic file system operation permits the file system to be used as the mechanism that limits access to the semaphore, thereby limiting access to the lock file and the various partitions.

[0016] FIG. 1 illustrates a computing arrangement 100 in which various embodiments of the invention may be used. The arrangement includes multiple client systems 102, 104, and 106 that host respective client applications 108, 110, and 112. The client applications share access to a data set that is stored in data file 120.

[0017] The data file may be hosted by a single file server system 122 or within a distributed file system, depending on implementation requirements. In either scenario, a file management system 124 is used to manage a collection of data files 126, which includes the data file 120 of interest to the client applications. Each client system also hosts a file system interface component 130, 132, or 134 that provides the interface to the file management system 124 for access to the data file 120. The client systems and server system(s) are coupled via network 136, the characteristics of which depend on implementation requirements.

[0018] To support concurrent access by the client applications, data file 120 is partitioned according to implementation requirements. That is, while one client application is accessing one partition, another client application may access another partition. The partitions may be of equal sizes or the sizes may vary from one partition to another.

[0019] In the example embodiment, the lock status of each partition may be in one of three states: unlocked, exclusive lock, or read-only lock. When a partition is unlocked, the partition is available to be locked by a client application, either exclusively or for read-only access. The client applications are expected to obtain a lock on a partition before performing any read or write operation on the partition.

[0020] A client application obtains an exclusive lock on a partition when the client application seeks to write to the partition. When one client application has an exclusive lock on a partition, the other client applications are expected to refrain from writing to the partition, and should another client application access data in the partition the integrity of any data read from the partition cannot be guaranteed. The embodiments described herein assume only a single writer is allowed in each partition. However, it will be appreciated that the embodiments could be extended to manage multiple writers per partition.

[0021] When a partition has a read-only lock, multiple client applications may read from the partition, and client applications are expected to refrain from writing to the partition.

[0022] Client applications are expected to respect whether a lock is granted. For example, even though a lock may be denied to a client application, there is no central gate keeper that would prevent a client application from accessing the data file. This lock mechanism relies on the integrity of each client application. This simplifies and supports efficient management of concurrent access.

[0023] The lock status information for the partitions is stored as a lock file(s) 138, which is accessible to the client applications. The lock file 138 and data file 120 may be subject to the same file system management 124, or may be under separate file systems depending on implementation requirements. Storing the partitions' lock status information as a file allows the client applications to determine the status of the partitions without requiring a specially adapted, or commercially available, centralized lock manager. The file system together with integrity of the client applications is used to manage concurrent access.

[0024] Each of the client applications reads from and writes to the lock file 138, and therefore, access to the lock file must be controlled. Access to the lock file is controlled and serialized via semaphore 140. Conceptually, the semaphore may be viewed as a lock that can be obtained by only one client application. The semaphore must be obtained before a client application reads from or writes to the lock file. The semaphore is released when the operation on the lock file is complete.

[0025] In an example embodiment, the semaphore is implemented as a status code returned from an operation on a file system object. For example, creating a directory in the file system is used as the semaphore. This operation is atomic. That is, the file system management 124 ensures that this file system operation is complete before another file system operation is allowed. Each client application that needs access to a partition is configured to attempt to create the same directory whenever the partition is to be locked. Thus, if a client application attempts a create-directory operation and receives a status code from the file system indicating that the operation was successful, then the semaphore belongs to that client application. If a client application attempts a create-directory operation and receives a status code from the file system indicating that the operation failed, then the semaphore belongs to another client application and the requesting client application must wait and retry. It will be appreciated that other file system operations may be used to implement the semaphore function depending on implementation requirements and file system capabilities. In other embodiments the semaphore mechanism may be implemented by any other technique known to one of ordinary skill in the arts, as long as the semaphore is accessible to all client applications that need access to the lock file.

[0026] Because there is no centralized lock manager, each client application is required to implement a deadlock detection and/or avoidance mechanism. Those techniques that are commonly used in distributed lock management may be implemented by the client applications.

[0027] Even though a network arrangement of client systems is used to illustrate hosting of multiple client applications, it will be appreciated that one, more, or all of the client applications could be hosted on a single system. Control over concurrent access via the file system may be desirable in arrangements with one or just a few client systems for the reasons set forth above. The various embodiments of the invention supports scalability from a single system to multiple systems.

[0028] FIG. 2 is a block diagram that illustrates a partitioned data file 202 and an associated lock file 204. The data file is divided into multiple partitions 206, 208, 210, 212, and 214, and the lock file includes lock information objects 222, 224, 226, 228, and 230 that are associated with the partitions.

[0029] The partitions of data file 202 are illustrated in a logical view rather than a physical view. That is, even though the partitions are shown as being contiguous in the logical view, it will be appreciated that there is no requirement that the partitions be physically contiguous. However, to preserve data integrity the partitions should not physically overlap.

[0030] Each lock information object describes the lock status of an associated partition and includes information that defines the partition with which the lock information is associated. For example, lock information object 222 describes partition 206.

[0031] The lock information includes the type of lock held on the partition, a timestamp indicating the time at which the partition was locked, an identifier of the system hosting the client application, an identifier of the user using the client application, the name of the client application, an identifier of the process executing the client application, and the partition number. At system initialization, the lock file starts as being empty. As lock requests are made and locks are obtained, the lock file is updated accordingly.

[0032] FIG. 3 is a flowchart of an example process implemented by a client application for accessing a file in accordance with various embodiments of the invention. The process generally entails obtaining the semaphore, procuring the needed lock on a selected partition, performing a file operation, and releasing the lock.

[0033] If a client application needs to access a shared data file during its application-specific processing (step 302), the client application attempts to create a directory (step 304). Each client application attempts to create the same directory when the semaphore is needed. Thus, if a first client application has already created the directory and thereby obtained the semaphore, a second client application will receive an error status code if an attempt is made to create the same directory. If the directory was not successfully created (decision step 306), the process returns to perform further application-specific processing. The client application may choose to repeat the attempt to create the directory or abandon the operation.

[0034] If the directory is successfully created, the lock information object that defines the partition to be accessed is obtained from the lock file (step 308). The client application then attempts to update the lock status in the lock information object according to the file operation to be performed (step 310). Because different operations require different types of locks, the client application may be unable to obtain the desired lock.

[0035] After attempting to obtain the desired lock, the directory is deleted (step 312), which permits other client applications to access the lock file. If the client application obtained the desired lock (decision step 314), the process performs application-specific file operations on the selected partition (step 315). Otherwise, the process returns to step 302 to perform other application-specific processing, such as waiting and retrying to obtain the desired lock.

[0036] When access to the selected partition is no longer needed, the client application performs the create-directory operation in an attempt to obtain the semaphore (step 316). The client application repeats the create-directory operation until the directory is successfully created (decision step 318).

[0037] The lock information object is once again read from the lock file (step 320), and the lock status is updated to indicate that the lock held by the client application has been released (step 322). Once the lock information object has been written to persistent storage, the directory is deleted (step 324) and the process returns to perform further application-specific processing (step 302).

[0038] FIG. 4 is a flowchart of an example process implemented by a client application for obtaining a lock on a partition in support of a read operation. The illustrated process steps assume that the semaphore has been obtained and the appropriate lock information object has been read.

[0039] If the desired partition is presently exclusively locked (decision step 452), a status code is returned to indicate that the needed lock cannot be obtained (step 454) and control is returned to step 310 (FIG. 3) of the client application. Otherwise, the process checks whether there is read lock on the partition (decision step 456). If so, a client application identifier of the client application seeking the lock is added to the lock information object (step 458). The identifiers of all client applications currently holding read locks are stored in the lock information object.

[0040] If the desired partition is presently unlocked (decision step 460), the lock status is set to read lock and the client application identifier is added to the lock information object (step 462). An error status is returned if the lock status has an illegal value (step 464).

[0041] It will be appreciated that when a read lock is released by a client application, the application's identifier is removed from the lock information object. If no other client application identifiers remain, then the status is changed to unlocked.

[0042] FIG. 5 is a flowchart of an example process implemented by a client application for obtaining a lock on a partition in support of a write operation. The illustrated process steps assume that the semaphore has been obtained and the appropriate lock information object has been read.

[0043] If the desired partition is presently exclusively locked (decision step 502), a status code is returned to indicate that the needed lock cannot be obtained (step 504) and control is returned to step 310 (FIG. 3) of the client application. Otherwise, the process checks whether there is read lock on the partition (decision step 506). If so, an exclusive lock cannot be obtained, and the appropriate status code is return.

[0044] If the desired partition is presently unlocked (decision step 508), the lock status is set to exclusive lock. An error status is returned if the lock status has an illegal value (step 464).

[0045] It will be appreciated that in releasing an exclusive lock, the client application changes the lock status of the lock information object to unlocked.

[0046] Those skilled in the art will appreciate that various alternative computing arrangements would be suitable for hosting the processes of the different embodiments of the present invention. In addition, the processes may be provided via a variety of computer-readable media or delivery channels such as magnetic or optical disks or tapes, electronic storage devices, or as application services over a network.

[0047] The present invention is believed to be applicable to a variety of systems for controlling concurrent access to a data set and has been found to be particularly applicable and beneficial in controlling concurrent access a persistently stored data set. Other aspects and embodiments of the present invention will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. It is intended that the specification and illustrated embodiments be considered as examples only, with a true scope and spirit of the invention being indicated by the following claims.