Method for Reporting File Write Status in a Shared File System

The present invention relates to apparatus and method for performing transfer, between a computer operating system and a peripheral device, of data in the form of files, and the management of such files. In particular, the apparatus and method apply to such transfers and management using shared buffers.

Data is transferred between the main memory of a computer and an external or peripheral physical file, such as a hard disk or other storage device, through use of file information blocks (FIBs), also referred to as logical files. Each logical file has a file description which is basically a control point for logical operations against the file, and has its own copy of where in the file such an operation is taking place at any given time.

In reading data from or writing data to physical media, one approach has been to perform direct input/output between a computer and such media. In other words, no intermediate or intermediary buffers are used, but instead the user performs physical input/output operations directly from his arrays to the media; the user is then responsible for managing its own buffering. A second approach is where intermediary buffers were used that were private to each logical file. In other words, each buffer is permanently assigned to a physical location in memory. In this approach, if each of two programs has a buffer, then there is no coordination of physical reads and writes, so that until the content of one buffer is written onto physical media, other programs could not be aware of the change. Furthermore, even if the content of a buffer has been forced out to disk, if the content of any corresponding buffer in another program is then not discarded, then that program would see bad data at the buffer instead of the proper data at the disk. For this reason, a private buffer scheme is not a desirable approach where multiple programs or program modules want to share the same physical file. The direct I/O approach could be useable for that purpose, but required some other mechanism for synchronizing or coordinating writes to physical media so that each program or module would know when it had to discard its data and read same again from the media. Accordingly, there exist applications in which there is a need to have two or more separate programs or modules to share one physical file. For such purposes, sharing buffers is preferred. With shared buffers, each buffer services one location or set of locations in the storage device, but is shared with, and can be utilized by any of, multiple programs, users, modules, etc. However, problems have been encountered in that one user might write into a buffer, and then a second user might take some action requiring the buffer to be transferred to physical media. If that transfer failed, the data loss would not be reported to the user or program that originally provided to the shared buffer the data thereby lost. Such a user would accordingly assume that the content of that buffer had been written into the desired storage location, and the future operation of that user would be affected thereby. The present invention avoids this problem.

Accordingly, an object of the present invention is to provide apparatus and method for transfer of data between one or more storage locations and a plurality of users.

Another object of the present invention is to provide apparatus and method for sharing otherwise dedicated buffers among a plurality of users.

A further object of the present invention is to provide apparatus and method for controlling access to shared buffers.

Still another object of the present invention is to provide apparatus and method for notifying a user as to whether a data transfer operation requested by that user has been accomplished.

A still further object of the present invention is to provide apparatus and method for providing error notification to a logical file so that the job that issued a write command knows that this operation was not accomplished.

Yet another object of the present invention is to provide apparatus and method for providing parallelism in the handling of multiple I/O commands, in that multiple input/output operations can take place between multiple stacks and multiple logical files at the same time.

Still another object of the present invention is to provide apparatus and method for efficiently implementing shared buffers that also efficiently informs a user providing data to a buffer if that data is not successfully written from that buffer.

Briefly, these and other objects of the present invention are accomplished by a pool of shared buffers wherein each buffer is dedicated to one or more storage location(s) but is shared among a plurality of logical files or file information blocks. A separate descriptor is provided for each file which identifies each buffer that that file is currently using. This linkage is not severed until any writes necessitated by modifications to the buffer via that logical file have been issued and completed, and any errors have been reported. If an attempt to write data via a shared buffer to a storage location such as in magnetic media incurs an error, then that error is reported back to the originator of that data (e.g. a computer program) to provide notification that the data did not get written to the desired storage location. In a shared buffer space, many logical files could write into that space before their data is written physically to the respective desired storage locations; a single physical write step will suffice to so transfer that data to the desired locations. The invention is thereby able to do a minimal number of physical writes while still reporting the writes and any errors to the originators of the respective data.

Other objects, advantages and novel features of the invention will become apparent from the following detailed description of the invention when considered in conjunction with the accompanying drawings.

In the drawings,

FIG. 1 is a diagrammatic representation of an arrangement of a digital computer main memory according to one embodiment of the present invention, and of a storage device such as a disk drive and media, for transfer of data therebetween;

FIG. 1A is a flowchart showing a process, according to one embodiment of the present invention, for controlling operation of the apparatus of FIG. 1;

FIG. 1B is a flowchart showing a portion of another process, according to another embodiment of the present invention, for controlling operation of the apparatus of FIG. 1;

FIG. 1C is a flowchart showing another portion of the process partially shown in FIG. 1B for controlling operation of the apparatus of FIG. 1;

FIG. 2 is a flowchart showing a portion of a process, according to one embodiment of the present invention, for controlling operation of the apparatus of FIG. 1;

FIG. 3 is a flowchart showing another portion of a process, according to one embodiment of the present invention, for controlling operation of the apparatus of FIG. 1;

FIG. 4 is a flowchart showing another portion of a process, according to one embodiment of the present invention, for controlling operation of the apparatus of FIG. 1;

FIG. 5 is a flowchart showing another portion of a process, according to one embodiment of the present invention, for controlling operation of the apparatus of FIG. 1;

FIG. 6 is a flowchart showing another portion of a process, according to one embodiment of the present invention, for controlling operation of the apparatus of FIG. 1;

FIG. 7 is a flowchart showing another portion of a process, according to one embodiment of the present invention, for controlling operation of the apparatus of FIG. 1;

FIG. 8 is a flowchart showing another portion of a process, according to one embodiment of the present invention, for controlling operation of the apparatus of FIG. 1; and

FIG. 9 is a flowchart showing another portion of a process, according to one embodiment of the present invention, for controlling operation of the apparatus of FIG. 1.

Referring now to the drawings, wherein like reference characters designate like or corresponding parts throughout the several views, there is shown in FIG. 1 a digital data processing system 10 including a main memory portion 12 and a disk drive 14. The disk drive 14 includes a disk or other suitable physical media 16 that includes at least one physical file 18 and a directory 20 for physical file(s) 18. However, it should be understood that the practice of the present invention is not limited to utilization of a disk drive as a storage device; any other suitable storage device such as a random access memory (RAM) could be so utilized instead. Also, instead of using a single such storage device, multiple such storage devices, such as concatenated disks, could instead be used, with a single directory being provided for that entire group.

As shown in FIG. 1, main memory portion 12 includes a portion containing a first program 22 and a portion containing a second program 24. As used herein, the terms program, job, user, and module are used interchangeably. Although only two programs 22 and 24 are shown in FIG. 1, it should be understood that the present invention can be practiced with any number of such programs. However, some particular advantages of the present invention are utilized when main memory portion 12 includes more than one program. Main memory portion 12 further includes a plurality of buffers 26, 28, 30, 32 and 34. Again, although five buffers are shown in FIG. 1, the present invention can be practiced with any number of buffers. Each such buffer is dedicated to a particular portion of physical file 18 (defined by disk file header or directory 20), but can each be utilized by (i.e. shared by) programs 22 and 24 and any other such program(s). Thus, a single shared buffer may have no user pointing to it, or one user pointing to it, or more than one user pointing to it. FIG. 1 illustrates one example of buffer sharing at a single point in time, in that logical file 50 points to buffers 26 and 30 while logical file 56 points to buffers 30 and 34 and neither logical file 50 and 56 points to any of buffers 28 and 32. For that example, logical file 50 has a link to buffer 26 and buffer 30. At that same instant, logical file 56 has a link to buffer 30 and buffer 34. Also, buffers 28 and 32 do not have any logical files linked to them at that instant. Nonetheless, buffers 28 and 32 may still have a valid copy of some physical space in the file, but they do not then have any unwritten changes in them, because a link from a logical file to a buffer is not removed until the physical write from that buffer to media 16 is accomplished. Otherwise, if the connection from such a buffer to the logical file is lost, and the data from that logical file is not written from that buffer to the media 16 or other storage location, then it would no longer be known who to inform that the data could not be written. Accordingly, a linkage between a logical file and a buffer is not broken if that logical file has done a write to that buffer and that write has not yet been forced out such as to physical media 16. However, it should be understood that a single logical file could be simultaneously linked to many more than two shared buffers. Buffers 26, 28, 30, 32 and 34 together constitute a buffer pool 36 and are accessed by a two-dimensional shared buffer pool array 38. Array 38 includes a row index 40 and a plurality of rows of buffer pool information container identified thereby, of which only one row 42 (pointed to by slot 97 of index or array 38 in FIG. 1) is shown in FIG. 1 for simplicity of illustration. Although in FIG. 1 row index 40 is indicated as indexing zero to 2000 rows, it should be understood that this number is merely illustrative and that the present invention can be practiced with any number of rows. An operand resizes array 38 and array 44 by changing the number of rows pointed to by each such array. Each row has a container or set of entries M, one for each shared buffer. For each buffer, there is an entry under that buffer's container or portion of the shared buffer pool row to indicate whether a read of the content of that buffer is being performed, whether there is a write to that buffer outstanding and whether the commanded input/output (I/O) operation for that shared buffer has been completed. Parallel to array 38 there is also a two-dimensional header index array 44 including a row index 46 and a plurality of header index rows for that array. For simplicity of illustration, only one such header index row 48 (pointed to by slot 97 of array 40 in FIG. 1) is shown in FIG. 1. Although an index 46 for zero to 2000 header index rows is shown in FIG. 1, it should be understood that this number is merely illustrative and that the present invention can be practiced with any number of rows. However, a buffer pool row and a header index row will be needed for each set of shared buffers corresponding to a physical disk file. For this embodiment, the buffers, logical files, containers and indexes of FIG. 1 are pieces of main memory 12.

First program 22 includes one or more logical files 50 and an index 52 of MOM or prime descriptors for those logical files. Each logical file has a set of one or more buffer information containers 54. Likewise, second program 24 includes one or more logical files 56 and an index 58 of MOM descriptors for same. Each such logical file 56 includes a set of one or more buffer information containers 60. Index 52 contains a MOM descriptor for each logical file 50 of program 22. Index 58 contains a MOM descriptor for each logical file 56 of second program 24. As shown in FIG. 1, each logical file has such a container for each physical file, and each such container contains an entry for each buffer in the pool dedicated to that physical file.

Thus, each open physical disk file has its own private set of buffers. This set of buffers is shared among all of the logical files which have the physical disk file open, and are using buffer sharing. It is possible, but not required, to force all logical files using a given physical disk file to use buffer sharing.

Regarding logical file data structures, each open physical disk file has an associated integer value, known as its disk file header index, which uniquely identifies that physical disk file within the context of the entire system 10. In the attached code listings, the variable UNITNUMBER is an element of the logical file structure being operated upon, and contains the disk file header index of the physical disk file which is being used by the logical file.

Each logical file 50, 56, etc. has an associated set 54, 60, etc. of one or more buffer information containers, each of which may contain information about a buffer in use by the logical file. In the code listings below, IOAREA is an element of the logical file structure being operated upon, and contains a deindexed reference to the currently in-use buffer information container. The buffer information container includes elements IOADW, BUFFPOOLINDEX, IOSCOMPLETEDTARGET, IOAW.IOOUTSTANDINGF and IOCBDESC. When the buffer information container is linked to a shared buffer, element IOADW holds the file relative address of the beginning of the file space mapped to that buffer. When the buffer information container is not linked to a shared buffer, then element IOADW holds a value of -1. Element IOADW is checked against the value of -1 whenever it is necessary to determine whether or not a buffer information container is linked to a shared buffer.

When the buffer information container is linked to a shared buffer, then element BUFFPOOLINDEX holds the index of that buffer in the structure holding all the shared buffers for the physical disk file. Thus, the variable elements UNITNUMBER and IOAREA[BUFFPOOLINDEX] together uniquely identify the correct buffer within the context of all buffers in the system 10.

When the buffer information container is linked to a shared buffer, and the logical file then accessing that buffer has not modified the data in that buffer since the link between the two was created, then element IOSCOMPLETEDTARGET holds a value of 1. When the buffer information container is linked to a shared buffer, and this logical file has modified the data in that buffer since the link was created, then element IOSCOMPLETEDTARGET holds a value of one plus the value which variable element SHAREDBUFFIOSCOMPLETED held at the time that this logical file most recently modified the data in that buffer.

When the buffer information container is linked to a shared buffer, then element IOAW.IOOUTSTANDINGF holds its value true when the logical file manipulation routines must invoke procedure or routine SHAREDBUFFWAITP before either using that buffer or delinking from that buffer.

When the buffer information container is linked to a shared buffer, then element IOCBDESC contains the logical result descriptor most recently reported to this container (as a result of an invocation of routine SHAREDBUFFWAITP, described below). A logical result descriptor contains a Boolean flag reflecting overall success/failure of an I/O operation, and other fields containing more detailed information concerning the outcome of the I/O operation.

Regarding locking, all routines in the code listings below are executed from logical file manipulation routines which have already obtained the lock which guards access to the logical file's data structures.

Regarding buffer pool data structures, the set of shared buffers corresponding to a physical disk file is stored in a row of the system-wide global data structure SHAREDBUFFPOOL. The appropriate row in that structure is selected by indexing by the physical disk file's disk file header index, described above. Most of the routines in the code listings below declare a local reference structure named POOL and use it to hold a de-indexed reference to the appropriate row of global data structure SHAREDBUFFPOOL. Each row of structure SHAREDBUFFPOOL contains a number of elements reflecting on or indicative of the overall status of the shared buffer pool. Among those elements are element SHAREDBUFFTOTALRECS and element SHAREDBUFFHEADERTOTALRECS. Element SHAREBUFFTOTALRECS is the bound on records which have been transferred to buffers. Element SHAREDBUFFHEADERTOTALRECS is the bound on records spanned by the largest end-of-file (EOF) value which has been successfully recorded in the disk file header on the physical disk media.

Each row of structure SHAREDBUFFPOOL also contains a set of one or more buffers. Each such buffer includes the nine elements described immediately below. The value of element SHAREDBUFFBUFFER points to the data space for that buffer. The value of element SHAREDBUFFFILEADDRESS shows the file relative address of the beginning of the file space mapped to the buffer. The value of element SHAREDBUFFSECTORS indicates the length of the file space mapped to the buffer. The value of element SHAREDBUFFIOCB indicates the physical I/O control structure for the buffer. Element SHAREDBUFFEVENT is the synchronization variable used by the physical I/O routines invoked at several points in the code printout given below to report physical I/O completion. Element SHAREDBUFFLRD is the logical result descriptor of the physical I/O most recently processed by routine SHAREDBUFFWAITP. While the buffer is initially being read (and therefore does not yet have valid data), element SHAREDBUFFIOSCOMPLETED contains a value of zero. When the buffer has valid data, but no writes have been processed by routine SHAREDBUFFWAITP, element SHAREDBUFFIOSCOMPLETED contains a value of one. Otherwise, element SHAREDBUFFIOSCOMPLETED contains a value of one plus the number of writes which have been processed by routine SHAREDBUFFWAITP. While a buffer is being written (from the time that the write command for same is issued to the time that the write completion is processed by routine SHAREDBUFFWAITP), element SHAREDBUFFWRITEOUTSTANDING contains a value of true. The value of element SHAREDBUFFUSERS indicates the number of logical file buffer information containers linked to the shared buffer. Such linkage is established by routine ASSOCIATESHAREDBUFFERP and is broken by routine DISASSOCIATESHAREDBUFFERBODY, both described below. In other words, routine ASSOCIATESHAREDBUFFERP initiates the reading process.

The major routines shown in the flowcharts of FIGS. 2-9 and shown in the following code printouts include ASSOCIATESHAREDBUFFERP, INITIATESHAREDWRITEP, SHAREDBUFFWAITP, DISASSOCIATESHAREDBUFFERBODY, MODIFYSHAREDBUFFPREFACE, MODIFYSHAREDBUFFEPILOG, MODIFYSHAREDBUFFPAUSE, MODIFYSHAREDBUFFRESUME, CHECKSHAREDEOF and SHAREDBUFFSYNCDSKEOFP. These routines are invoked from the logical file manipulation routines in response to read and write requests from programs. The particular sequence of invocations will depend on the operation requested by the program and the current state of the logical file. Two example flows are given here for illustration purposes.

In response to a read request, at step 1002 logical I/O first invokes routine CHECKSHAREDEOF to ensure that the request is for data at a currently valid file address. Then, at step 1004 all buffer information containers are checked for a container currently linked to a shared buffer mapped to the required file space. If such a buffer is not found, then at step 1006 routine ASSOCIATESHAREDBUFFERP is invoked to establish a shared buffer in that state. (In order to free a buffer information container for this purpose, invocations of routine INITIATESHAREDWRITEP, routine SHAREDBUFFWAITP, and/or routine DISASSOCIATESHAREDBUFFERBODY may be needed. Any error detected on write at this point must be reported to the program.) Next, if variable IOAW.IOOUTSTANDINGF is true (step 1008), then routine SHAREDBUFFWAITP is invoked (step 1010). Next, if IOCBDESC is true (step 1012), then an error is returned to the program (step 1014). Otherwise, data is returned to the program from the buffer (step 1016). This sequence is illustrated in FIG. 1A.

Note that if the read request had spanned file space mapped to multiple buffers, the data transfer would have been performed in multiple segments, with buffer switches possibly involving invocations of routine ASSOCIATESHAREDBUFFERP, routine SHAREDBUFFWAITP, routine INITIATESHAREDWRITEP, and/or routine DISASSOCIATESHAREDBUFFERBODY.

In response to a write request, as shown in FIG. 1B all buffer information containers are checked for a container currently linked to a shared buffer mapped to the required file space (step 1102). If such a buffer is not found, then routine ASSOCIATESHAREDBUFFERP is invoked (step 1104) to establish a shared buffer in that state. (In order to free a buffer information container for this purpose, invocations of routine INITIATESHAREDWRITEP, routine SHAREDBUFFWAITP, and/or routine DISASSOCIATESHAREDBUFFERBODY may be needed. Any error detected on write at this point must be reported to the program.) Next, if at step 1106 variable IOAW.IOOUTSTANDINGF is true, then at step 1108 routine SHAREDBUFFWAITP is invoked. Next, if at step 1110 variable IOCBDESC is true, then presence of an error is reported or returned to the program (step 1112). Otherwise, in FIG. 1C step 1114 data from the user program is stored in the buffer. This process for selecting a shared buffer is illustrated in FIG. 1B. FIG. 1C shows what happens after the shared buffer is selected.

This data transfer operation is bracketed by invocations of routine MODIFYSHAREDBUFFPREFACE and routine MODIFYSHAREDBUFFEPILOG. Immediately (if the program has requested that a physical write be issued and its result reported immediately), or at some later point (when the buffer information container is to be reused or when the logical file is being closed), routine INITIATESHAREDWRITEP is invoked by the logical file (step 1116); if at step 1118 variable IOAW.IOOUTSTANDINGF is then true, then routine SHAREDBUFFWAITP is invoked (step 1120); if variable IOCBDESC is then true (step 1122), then presence of an error is reported or returned to the program (step 1124). This sequence, which follows that of FIG. 1B, is illustrated in FIG. 1C.

Note that if the write request had spanned file space mapped to multiple buffers, then the data transfer would have been performed in multiple segments, with invocations of routine MODIFYSHAREDBUFFPAUSE and routine MODIFYSHAREDBUFFRESUME bracketing the buffer switch(es), and the buffer switch(es) possibly involving invocations of routine ASSOCIATESHAREDBUFFERP, routine SHAREDBUFFWAITP, routine INITIATESHAREDWRITEP, and/or routine DISASSOCIATESHAREDBUFFERBODY.

In one possible sequence of invocations of the routines in this embodiment of the present invention, once the shared buffer to be used for a data transfer has been identified (step 1113; also see FIG. 1B), the data in the buffer is then modified (step 1114) from the logical file (hereinafter referred to as the subject logical file) seeking to perform a write, e.g. to physical media 16. Routine INITIATESHAREDWRITEP is then invoked (step 1116; also see FIG. 3) to perform the write from the shared buffer to the media 16. During processing of routine INITIATESHAREDWRITEP, if routine INITIATESHAREDWRITEP places a value of true in element IOAW.IOOUTSTANDINGF, which is detected at step 1118, then routine SHAREDBUFFWAITP is invoked (step 1120; also see FIG. 4) to permit the physical I/O to occur. If data modified from the subject logical file fails to be successfully written to the physical disk media 16 (e.g. due to a path, drive or media error) (step 1122), then the logical result descriptor reported to that logical file (via element IOCBDESC) will reflect an error (step 1124). After the physical I/O to media 16 or elsewhere has been completed, routine DISASSOCIATESHAREDBUFFERBODY is invoked or called (step 1126) to delink the buffer information container from the currently linked shared buffer. This sequence, which (as discussed above) follows that of FIG. 1B, is illustrated in FIG. 1C.

In FIGS. 2-9, the eight-digit numbers in parentheses refer to the lines of code, in the following printout, where these operations are performed.

Routine ASSOCIATESHAREDBUFFERP is invoked by the logical file manipulation routines whenever a buffer spanning a given file relative address is needed. If routine ASSOCIATESHAREDBUFFERP places a value of true in element IOAW.IOOUTSTANDINGF, then routine SHAREDBUFFWAITP must be invoked before the data in that buffer is accessed. A flowchart for routine ASSOCIATESHAREDBUFFERP is shown in FIG. 2. In FIG. 2, once this routine has been called or initiated, the routine first looks for a valid buffer spanning the desired file address (step 102). If such a buffer is found (step 104), then that buffer is linked to the logical file buffer information container for the logical file containing the information to be transferred (step 106). In that container, element IOSCOMPLETEDTARGET is set equal to 1 (step 108). It is then determined whether the present value of buffer element SHAREDBUFFIOSCOMPLETED is greater than zero (step 110), indicating that the buffer has valid data. If so, then the value of element IOAW.IOOUTSTANDINGF is set to false, and the value of element IOCBDESC is set equal to the value of element SHAREDBUFFLRD (step 112). Otherwise, if the value of element SHAREDBUFFIOSCOMPLETED is equal to zero, then the value of element IOAW.IOOUTSTANDINGF is set equal to true (step 114), indicating that routine SHAREDBUFFWAITP must be invoked before using or delinking from the buffer. At step 116 directly following step 112 and step 114, it is determined whether the buffer linked at step 106 must be read; this determination is based on the result of the test at step 126. If the buffer must be read, then a read command is issued at step 118, after which routine ASSOCIATESHAREDBUFFERP is exited (step 120). If step 116 instead does not find that the buffer linked at step 106 must now be read, then the routine is exited at step 120 without a read command being issued at step 118. However, if step 104 determines that a valid buffer spanning the desired file address has not been found by step 102, then instead of proceeding directly to step 106, a buffer not then in use is found at step 122. The region of file space to span and set up the buffer of step 122 is then determined at step 124. If the file space of step 124 has been previously written to (step 126), then the buffer must be read, which is performed at subsequent step 118. (step 128 records this requirement in a local variable.) After step 128, the value of element SHAREDBUFFIOSCOMPLETED is set equal to zero at step 130, after which the process links that buffer to the logical file buffer information container at step 106. Otherwise, if the file space identified at step 124 is previously unwritten file space (step 126), then the content or value of that file space is set to all zeros (step 132). After the buffer of steps 122 and 124 has been zeroed at step 132, then at step 134 the value of element SHAREDBUFFIOSCOMPLETED is set equal to 1, and element SHAREDBUFFLRD is provided with a value indicating a non error result. Step 106 directly follows each of steps 130 and 134.

Routine INITIATESHAREDWRITEP is invoked by the logical file manipulation routines after the point where the data in the buffer being accessed has been modified from the subject logical file providing that data, and before the point where that buffer is delinked from that logical file. This routine must be invoked at the point where the user (e.g. the program) of the logical file requires that the data reach the physical disk media 16, even if that buffer is not to be delinked from the subject logical file at that point. For example, in a POSIX (Portable Operating System Interface) compliant operating system, the fsync function would invoke this routine. A flowchart for routine INITIATESHAREDWRITEP is shown in FIG. 3. After this routine is invoked (step 200), it is then determined at step 202 whether element SHAREDBUFFISOCOMPLETED is greater than or equal to the present value of element IOSCOMPLETEDTARGET. If not, then the routine sets the value of element IOAW.IOOUTSTANDINGF to be true at step 204. Otherwise, from step 202 the routine proceeds to step 206 at which it is determined whether element SHAREDBUFFLRD contains a non error result. If not, then at step 208 the value of element IOSCOMPLETEDTARGET is set equal to the value of element SHAREDBUFFIOSCOMPLETED (the number of writes which have been processed by routine SHAREDBUFFWAITP) plus 1. Immediately after step 208, at step 204 the value of element IOAW.IOOUTSTANDINGF is set to true, which will cause an invocation of routine SHAREDBUFFWAITP. If at step 206 the value of element SHAREDBUFFLRD is found to contain a non error result, then at step 210 the value of element IOAW.IOOUTSTANDINGF is set to false, and the value of element IOCBDESC is set equal to the value of element SHAREDBUFFLRD. After step 210, routine INITIATESHAREDWRITEP is exited (step 212). After step 204, it is determined at step 214 whether the value of element SHAREDBUFFWRITEOUTSTANDING is true (indicating that a buffer is then being written to). If so, then routine INITIATESHAREDWRITEP is exited (step 212). Otherwise, after step 214 a write to the designated buffer is issued at step 216. After step 216, the value of element SHAREDBUFFWRITEOUTSTANDING is set to true (step 218), after which routine INITIATESHAREDWRITEP is exited (step 212).

Routine SHAREDBUFFWAITP is invoked by the logical file manipulation routines after an invocation of either routine ASSOCIATESHAREDBUFFERP or routine INITIATESHAREDWRITEP leaves the value true in element IOAW.IOOUTSTANDINGF. Routine SHAREDBUFFWAITP waits for the physical I/O to complete (if it is not already completed), and reports the result via element IOCBDESC. Routine SHAREDBUFFWAITP must be invoked (after routine INITIATESHAREDWRITEP is exited) at the point where the user of the subject logical file requires that the data reach the physical disk media 16, even if the buffer is not to be delinked from the subject logical file at that point. For example, in a POSIX compliant operating system, the fsync function would invoke this routine. Routine INITIATESHAREDWRITEP and routine SHAREDBUFFWAITP are responsible for maintaining the following invariant: If data modified from the subject logical file fails to be successfully written to the physical disk media 16 (e.g. due to a path, drive, or media error), then the logical result descriptor reported to that logical file (via element IOCBDESC) must reflect an error. Logical file manipulation routines are responsible for insuring that errors reported via element IOCBDESC are properly reported to the client or user of the subject logical file. A flowchart for routine SHAREDBUFFWAITP is shown in FIG. 4. After this routine is invoked (step 300), it is determined at step 302 whether the value of element SHAREDBUFFIOSCOMPLETED is less than the value of element IOSCOMPLETEDTARGET. If not, then at step 304 the value of element SHAREDBUFFLRD is placed in element IOCBDESC (step 304), after which the routine is exited (step 306). Otherwise, if at step 302 the value of element SHAREDBUFFIOSCOMPLETED is found to be less than the value of element IOSCOMPLETEDTARGET, then at step 308 if the physical I/O is not yet completed, then the routine waits for that completion. After step 308, it is determined at step 310 whether the value of element SHAREDBUFFIOSCOMPLETED is equal to zero, indicating that the buffer has just been read. If so, then since this read is now completed, at step 312 the routine then increments by one the value of element SHAREDBUFFIOSCOMPLETED, and sets the value of element SHAREDBUFFLRD to show the result of that physical I/O. step 312 is immediately followed by step 304 discussed above, after which the routine is exited (step 306). Otherwise, if at step 310 the value of element SHAREDBUFFIOSCOMPLETED is not equal to zero (indicating that the buffer has just been written to physical media), the routine next determines whether the value of element SHAREDBUFFWRITEOUTSTANDING is true (indicating that the write completion has not yet been processed by another invocation of routine SHAREDBUFFWAITP) at step 314. If so, then at step 316 the value of element SHAREDBUFFWRITEOUTSTANDING is set false to indicate that the write from that buffer has been completed, the value of element SHAREDBUFFIOSCOMPLETED is incremented by one, and the value of element SHAREDBUFFLRD is set to indicate the result of that physical I/O. Step 316 is immediately followed by step 304 discussed above, after which the routine is exited (step 306). Otherwise, if at step 314 the value of element SHAREDBUFFWRITEOUTSTANDING is found to be false, then the routine proceeds directly to step 304, after which the routine is exited (step 306). After routine SHAREDBUFFWAITP is exited, the user invoking that routine sets the value of element IOAW.IOOUTSTANDINGF to be false, so that routine SHAREDBUFFWAITP is not unnecessarily reinvoked (step 318).

Routine DISASSOCIATESHAREDBUFFERBODY is invoked by the logical file manipulation routines to delink a buffer information container from its currently linked buffer. It is the responsibility of the logical file manipulation routines to have previously invoked routine INITIATESHAREDWRITEP since (and if) the buffer data was last modified from the subject logical file. It is the responsibility of the logical file manipulation routines to have already invoked routine SHAREDBUFFWAITP if necessary, i.e. element IOAW.IOOUTSTANDINGF must hold the value of false when routine DISASSOCIATESHAREDBUFFERBODY is invoked.

Invocations of routines MODIFYSHAREDBUFFPREFACE and MODIFYSHAREDBUFFEPILOG bracket the code in the logical file write routines which transfer data to buffers. By ensuring that the shared buffer pool 36 lock is held and that the value of element SHAREDBUFFWRITEOUTSTANDING is false at the start of data transfer, routine MODIFYSHAREDBUFFPREFACE (and routine MODIFYSHAREDBUFFRESUME) ensure that no physical write is in process, and that no physical write can begin, during data transfer to a buffer 26, 28, 30, 32 or 34.

A flowchart for routine MODIFYSHAREDBUFFPREFACE is shown in FIG. 5. After the routine is invoked (step 400), the shared buffer pool is locked (step 402). After step 402, at step 404 it is determined whether element SHAREDBUFFWRITEOUTSTANDING has a value of true, indicating that a buffer is then being written to physical media. If so, then the routine waits for that write to be completed (step 406); the shared buffer pool lock of step 402 may be relinquished during this wait, but must be reinstated thereafter. If at step 404 element SHAREDBUFFWRITEOUTSTANDING has a value of false, then at step 408 it is determined whether an APPEND write is then in progress. If not, then the routine is exited (step 410). Otherwise, if at step 408 an APPEND write is detected, then at step 412 it is determined whether the next record to be so written is element SHAREDBUFFTOTALRECS. If so, then the routine is exited (step 410). Otherwise, at step 414 the next record is repositioned at the bound defined by the value of element SHAREDBUFFTOTALRECS. This may cause a change in which shared buffer is pointed to by element IOAREA. The routine at step 416 then again determines whether element SHAREDBUFFWRITEOUTSTANDING has a value of true. If so, then at step 418 the routine waits for the write thereby indicated to be completed; the shared buffer pool lock may be relinquished during this wait. If at step 416 element SHAREDBUFFWRITEOUTSTANDING has a value of false, then the routine returns to step 412, from which it may exit (step 410) or may again proceed to step 414 described above.

A flowchart for routine MODIFYSHAREDBUFFEPILOG is shown in FIG. 6. After that routine is invoked (step 450), the routine determines whether the location or address value of the record after the last record just transferred is greater than the value of element or variable SHAREDBUFFTOTALRECS, the bound on records which have been transferred to buffers. If not, then at step 454 element or variable IOSCOMPLETEDTARGET is given a value equal to that of element SHAREDBUFFIOSCOMPLETED plus one, indicating that the buffer has been modified to contain valid data that must be written to physical media 16 at some point in the future. Otherwise, between step 452 and step 454, element SHAREDBUFFTOTALRECS is given a value indicative of the location or address of the record after the last record that had just been transferred. After step 454, the shared buffer pool 36 is unlocked (step 456), and the routine is exited (step 458).

Invocations of routine MODIFYSHAREDBUFFPAUSE and of routine MODIFYSHAREDBUFFRESUME bracket the point in the logical file write routines where data transfer is interrupted to switch buffers. These routines are to be invoked only when logical file write operations span buffer boundaries. For non-APPEND writes, the shared buffer pool 36 lock is relinquished during this switch of buffers. Relinquishing the lock allows other write operations from other logical files to proceed concurrently with any physical I/O's which may be necessary to accomplish the switch of buffers. For APPEND writes, the shared buffer pool 36 lock is not relinquished during the switch of buffers. This provides atomicity of APPEND writes with respect to each other. The variable SHAREDBUFFLOCKHELD is set to true during this buffer switch. Element SHAREDBUFFLOCKHELD is an element of the logical file structure. Doing so prevents invocations of routine ASSOCIATESHAREDBUFFERP, routine INITIATESHAREDWRITEP, and routine SHAREDBUFFWAITP which occur during buffer switch from attempting to lock the shared buffer pool 36 (and thereby deadlocking). In this regard, see the listings below of routine LOCKSHAREDBUFFERPOOL and routine UNLOCKSHAREDBUFFERPOOL.

A flowchart for routine MODIFYSHAREDBUFFPAUSE is shown in FIG. 7. After that routine is invoked (step 500), at step 502 the value of element IOSCOMPLETEDTARGET is set equal to the value of element SHAREDBUFFIOSCOMPLETED plus one, indicating that the container is linked to a shared buffer and the subject logical file has modified the data in that buffer since this link was created. At step 504, it is determined whether an APPEND write is then being performed on that buffer. If so, then at step 506 element or variable SHAREDBUFFLOCKHELD is given a value of true. Otherwise, if no APPEND write is present at step 504, then at step 508 the shared buffer pool is unlocked. POSIX is described in International Standard ISO/IEC 9945-1 (IEEE Std 1003.1) Information Technology--Portable Operating System Interface (POSIX)--Part 1: System Application Program Interface (API) [C Language] which is hereby incorporated by reference herein. As discussed in International Standard ISO/IEC 9945-1: 1990, supra, at Part 1, page 119, lines 201-203: "If the O.sub.-- APPEND flag of the file status flags is set, the file offset shall be set to the end of the file prior to each write, and no intervening file modification operation shall be allowed between changing the file offset and the write operation." After either of step 506 or step 508, this routine is exited (step 510).

A flowchart for routine MODIFYSHAREDBUFFRESUME is shown in FIG. 8. After this routine is invoked or initiated (step 550), the routine determines whether an APPEND write is being performed (step 552). If so, then at step 554 element SHAREDBUFFLOCKHELD is given a value of false, after which the routine is exited (step 556). Otherwise, after step 552 the routine at step 558 determines whether element SHAREDBUFFWRITEOUTSTANDING has a value of true, indicating that the content of a buffer is then being written to physical media 16. If so, then at step 560 the routine waits for this write to be completed; the shared buffer pool 36 lock may be temporarily relinquished during this wait. If at step 558 element SHAREDBUFFWRITEOUTSTANDING is found to have a value of false, then the routine is exited (step 556).

Routine CHECKSHAREDEOF is invoked by the logical file read routines just before data transfer to buffers. Routine CHECKSHAREDEOF prevents reading beyond the current value of element SHAREDBUFFTOTALRECS. A flowchart for routine CHECKSHAREDEOF is shown in FIG. 9. After this routine is invoked or initiated (step 600), this routine first determines whether the location of the next record plus the number of records to be read is greater than the value of element SHAREDBUFFTOTALRECS (step 602). If not, then retransfer can continue (step 604). Otherwise, after step 602 the routine next determines at step 604 whether the location of the next record plus one is greater than the present value of element SHAREDBUFFTOTALRECS. If so, then the routine reports that the end of file has been reached (step 606), and any further retransfer is bypassed (step 608). Otherwise, after step 604 the routine prepares to report a short read (step 610), shortens the transfer then in progress so that it is bounded by the value of element SHAREDBUFFTOTALRECS (step 612), and then retransfer is permitted to continue (step 604).

Routine SHAREDBUFFSYNCDSKEOFP is invoked by the logical file manipulation routines after the end of file has been extended via write operations. If the value of element SHAREDBUFFTOTALRECS reflects an end of file that is greater than the end of file which has already been successfully recorded in the disk file header or directory 20 on the physical disk media 16, then routine SHAREDBUFFSYNCDSKEOFP updates the disk file header on the physical disk media, waits for the write to complete, and reports an error if anything in this process was not completely successful. This routine must be invoked at the point where the user of the logical file requires that data which it has written beyond the old end of file reach the physical disk media 16, even if the logical file is not to be delinked from the shared buffer pool 36 at that point. For example, in a POSIX compliant operating system, the fsync function would invoke routine SHAREDBUFFSYNCDSKEOFP if any writes have been beyond the old end of file. Routine SHAREDBUFFSYNCDSKEOFP is responsible for maintaining the following invariant: If data modified from the subject logical file fails to become part of the file due to failure to successfully write the physical disk file's disk file header 20, then an error must be reported to that logical file. Logical file manipulation routines are responsible for ensuring that any errors reported by routine SHAREDBUFFSYNCDSKEOFP are properly reported to the client or user of the logical file.

The present invention can be implemented in software. An example of such software, written in the NEWP language, is given in the following pages. However, other programming languages can be used for this purpose. FIGS. 2-9 show flowcharts for above-identified portions of this software. ##SPC1##

Some of the many advantages of the present invention should now be readily apparent. For example, in error reporting, when data written via a given logical file fails to be successfully written to the physical disk media 16, or fails to become part of the file due to failure to successfully write the physical file's disk file header 20 (containing an increased end-of-file value) to the physical disk media 16, an error is reported to that logical file (and then to the client or user of that logical file). Also, physical writes are minimized; if several logical files modify the data in a given buffer before any requests that the buffer be written, then only one physical write need be performed.

Overlap of buffer modification and physical write is prevented. While a physical write is in process for a given buffer, all modifications to that buffer are blocked. Thus, those logical file write operations which do not span buffer boundaries will appear indivisibly (all or none) on the physical disk media 16 after a system 10 interruption.

More than one logical file can be linked to a single shared buffer as long as the shared buffer contain any changes intended for physical media 16 from the logical files that have not yet been written from such shared buffer to that media. In other words, no link from a logical file to a shared buffer is removed until the physical write to physical media required by that logical buffer is accomplished. In other words, the linkage between a logical file and a shared buffer is not broken if that logical file has done a write to that buffer and that write has not yet been forced out to the desired location in the physical media. Furthermore, where two such logical files have accomplished such a write to the same shared buffer that has not yet been forced out to physical media, the respective links between that shared buffer and those two (or more) logical files not only are maintained until the data from those logical files has been forced out from the shared buffer to the media, but furthermore this is accomplished using a single physical write operation to the media. When a logical file has finished providing its data to a shared buffer, instead of issuing a physical write, it instead invokes a procedure called INITIATESHAREDWRITEP which decides whether a write from the buffer to physical media needs to be issued. This procedure is invoked by the program 22, 24, etc. for that logical file when it is done with that shared buffer and a physical write would normally be issued. Procedure INITIATESHAREDWRITEP coordinates decisions about writes among all of the logical files.

The first step is to make sure that the user of a--logical file calls routine INITIATESHAREDWRITEP before giving up its connection to a buffer which that logical file has written to. That logical file is the only logical file that would remember or keep a record that it had written to that shared buffer. Alternatively, if a logical file did not remember or keep a record of that information, then a list could have been added off to the side which remembers which logical files had modified a particular buffer. Now, instead of requiring a logical file to finish all physical writes from a buffer to media, instead before a logical file severs its link to a shared buffer that it has written to, it calls routine INITIATESHAREDWRITEP. That routine will report to that logical file all errors (if any) that relate to the data from that logical file to be so written. Thus, all such errors are reported to the logical file before the link is broken.

If routine INITIATESHAREDWRITEP just did the physical write from the shared buffer to the physical media rather than looking to see whether it needed to do that write, all errors related to that data would still be reported back to the appropriate logical file when they need to be. Thus, any errors in writing something to media in this case would get back to the user to inform that user that the data did not get so physically written. For example, if the value 5 is to be written to a certain slot on the file, and that 5 never gets forced out to the physical media, then the logical file providing that 5 should be informed of that error. If routine INTIATESHAREDWRITEP just did a write whether it was necessary or not, and waited for that write synchronously and informed the user software program of the result, then it would be thereby ensured that no user's data did not get written out to physical media without that user finding out about that problem.

However, as written, routine INITIATESHAREDWRITEP not only avoids forcing a user to wait for the physical write to be accomplished, but furthermore that routine is not forced to always wait for the end of the physical I/O to occur in cases where the user is going to use another buffer and can leave sit the buffer which that user has just issued the write on and find out about it later. Also, where two or more logical files have changed the same shared buffer, a separate physical I/O step is not needed for each such user. For example, if ten logical files have accessed or changed the same shared buffer, then it is obviously advantagous to do a single physical input/output step instead of ten. Furthermore, overlapping is avoided. For example, if one program wrote to some place within a shared buffer and another program wrote to a different place in that same shared buffer, then after those writes have been accomplished, both programs can decide that they are done with that buffer and call routine INITIATESHAREDWRITEP. As discussed above, routine INITIATESHAREDWRITEP accomplishes the physical write for each of the two users using one I/O from that shared buffer to the physical media. Where two programs have both made changes to a single shared buffer, and both programs therefore think that that single shared buffer needs to be written to physical media, in practice the content of that buffer only needs to be written to physical media once because one write would push out both such changes. A single shared buffer would only write to one area of the physical media. For example, a shared buffer may have a size of 2000 bytes, and one program may have changed byte 7 and another program may have changed byte 43. A single write would put both changes out to the appropriate location in physical media at one time, especially since it is already determined from the particular bytes modified and the area of physical media which that buffer would operate on. In such a situation, two physical I/Os for the two changes would be redundant. For that reason, the structure of FIG. 1 includes a mechanism that would track which locations in physical media have to be written to, and that would coordinate the write operation. This is accomplished by two-dimensional shared buffer pool array 38 (particularly in that it includes the buffer information containers of shared buffer pool rows 42, etc.), two-dimensional header index array 44, and logical file dope vectors 54, 60, etc. Each row 42 contains a number of variables for each shared buffer; FIG. 1 shows a separate column of row 42 for each shared buffer 26-34 of pool 36. One place in row 42 shows where the buffer pool 36 believes the end of file is. For each shared buffer 26-34, there is a variable in row 42 called SHAREDBUFFIOSCOMPLETED. Variable SHAREDBUFFIOSCOMPLETED is an integer counter, and in each row 42, the entry for each shared buffer has a SHAREDBUFFIOSCOMPLETED counter associated with it. Each column of row 42 is a dope vector, one for each shared buffer. Each such dope vector has a set of words, some of which are descriptors and some of which are integers. Each logical file dope vector 54, 60, etc. has a buffer information container called IOSCOMPLETEDTARGET. Variable SHAREDBUFFIOSCOMPLETED and variable IOSCOMPLETEDTARGET together ensure that each user program is informed of any errors in performing physical writes it has requested to physical media, while ensuring that only one physical write is required for each shared buffer no matter how many programs have written to that buffer. Of these functions, the former is associated with the copy descriptor of row 42, and the other is associated with the MOM descriptor. Each variable SHAREDBUFFIOSCOMPLETED is associated with a single shared buffer while each variable IOSCOMPLETEDTARGET is associated with a program's use of that buffer. For any one buffer, if variable SHAREDBUFFIOSCOMPLETED has a value of zero in it, then the data in that buffer is not yet valid; in other words, a read must still be performed to fill or provide that buffer with the desired data. If variable SHAREDBUFFIOSCOMPLETED has a value of zero, that identifies a buffer which points to a place in the physical media which is already allocated but for which the data has not yet been read from the physical file into the buffer. Before that read can be accomplished, if the user wants to write a byte into a particular location in the physical file, and if the shared buffer is for example 2000 bytes long, that buffer cannot simply be filled with zeros except for the one byte to be so written, because then the value of the other 1999 bytes would be thereby erased. There are two ways in which a buffer could be filled: for accomplishing a read from physical media or by issuing a write to that buffer with data to be provided to physical media. However, if the location in the shared buffer to be so modified corresponds to a location beyond the end of file for the physical media, then that data value is simply zeroed and is not read. If routine ALLOCATESHAREDBUFFERP is asked to find a buffer, and such a buffer is not already available so that a new buffer has to be allocated for space that is beyond the present end of file, then that routine ALLOCATESHAREDBUFFERP simply zeros the value of that data rather than doing a read. However, if that space is within the file (is not beyond the current end of file), then the requested read has to be performed within a currently valid location. Thus, when variable SHAREDBUFFIOSCOMPLETED has a value of one, that value indicates that the data in the corresponding shared buffer is valid, which means that either a read to that buffer has been accomplished or the value of that buffer has been zeroed depending on whether the media storage space represented by that buffer is within or beyond the current end of file and has never been written. A value of two or more for variable SHAREDBUFFIOSCOMPLETED means that the data in that shared buffer is valid. A value of two for that variable means that shared buffer has been written to physical media once. A value of three for variable SHAREDBUFFIOSCOMPLETED means that the data in the corresponding shared buffer is valid, and that that buffer has been written twice. Another variable stored in row 42 is variable SHAREDBUFFWRITEOUTSTANDING; a value of false for that variable means that there is no write outstanding, while a value of true indicates that a physical write to the media has been issued but completion of that write has not yet been processed. If that variable SHAREDBUFFWRITEOUTSTANDING has a value of true, then the corresponding physical write may have been finished, but the data that the I/O processor has put into memory has not been checked to see whether it finished that write successfully or not; the result of that write has not been checked and so completion of that write has not yet been processed. Thus, if the value of variable SHAREDBUFFIOSCOMPLETED is one, then either the corresponding buffer has never been written to, or else a write from that buffer to physical media has been issued but whose completion has not yet been processed. The value of variable SHAREDBUFFIOSCOMPLETED has a maximum that is conceptually unboundedly large. The value of variable IOSCOMPLETEDTARGET at any given moment is meaningful only if a logical file has modified the corresponding shared buffer. So, at the point where that logical file asks to be hooked to that buffer and this variable is still zero, at the point where this logical file changes a byte (for example, byte 17) in that buffer, a value is then stored in that variable IOSMPLETEDTARGET recording that that byte of the shared buffer has been changed. The value of variable IOSCOMPLETEDTARGET is therefore set to the value of variable SHAREDBUFFIOSCOMPLETED plus one. If the value of variable SHAREDBUFFIOSCOMPLETED is two, then variable IOSCOMPLETEDTARGET has a value of three. This records that until the value of variable SHAREDBUFFIOSCOMPLETED goes to three, the data that has been written to the corresponding shared buffer has not yet been physically written from that buffer to the appropriate location in disk media 16 or the like.

When the program(s) 22, 24, etc. have finished writing data from logical file(s) 50, 56, etc. to a particular shared buffer (such as buffer 30), and have finished doing so, then routine INITIATESHAREDWRITEP is called. In that routine, the value of variable IOSCOMPLETEDTARGET in buffer information container (BIC) set 54, 60, etc. is compared with the current value of variable SHAREDBUFFIOSCOMPLETED held in the column associated with that shared buffer of the appropriate row of array 38 associated with a particular location (e.g. 97) of the logical file 50, 56, etc. being written to that buffer. When that comparison is made, if the value of variable SHAREDBUFFIOSCOMPLETED is greater than or equal to the value of variable IOSCOMPLETEDTARGET, that indicates that the write to physical media 16 has been successfully done since that shared buffer was modified by the program. If so, then nothing further need be done with regard to that write, and normal processing by the programs in memory 12 can continue; a new write is not needed. Although this comparison indicates whether the physical write has been accomplished, it is necessary to check variable SHAREDBUFFLRD to see if that write has been done successfully. The value of variable SHAREDBUFFLRD is the result of the last I/O that has been done. The value of variable SHAREDBUFFLRD is then checked to determine whether that write was successful.

When routine INITIATESHAREDWRITEP is called, it is first determined whether there is a need to do a physical I/O operation. This is determined by checking whether a write has been done since the shared buffer has last been modified and if that write was successful. If both of those things are true, then there is no need to do a further write and routine INITIATESHAREDWRITEP can be exited. So, if the value of variable SHAREDBUFFIOSCOMPLETED is greater than or equal to the value of variable IOSCOMPLETEDTARGET, then it is known that a write command has been issued. A write command can be issued because another program other than that (or those) modifying that shared buffer has issued a write, or it could happen because it has been incorrectly attempted to twice write from the shared buffer so modified.

For APPEND extensions, APPEND writes must be atomic with respect to each other. A program must be able to issue an APPEND write without using any locking protocols and be able to rely on the file space used being disjoint from that used by any other concurrent APPEND writes issued by other programs. This is accomplished by the present invention.

The entire shared buffer pool 36 need not be serialized. By holding the shared buffer pool's lock over the entire span of an APPEND write, but giving it up when switching buffers on a non-APPEND write, atomicity of APPEND writes is ensured while permitting concurrency for non-APPEND writes. In this regard, it may be necessary to issue and wait for physical I/O's when buffers are switched or exchanged.

Furthermore, reads are protected from seeing invalid file space. To ensure atomicity of APPEND writes which cross buffer boundaries, the value or location of the end-of-file can be incremented before transferring data to the buffers. That way, if control is lost while switching buffers, any other APPEND write which occurs while control is lost will not overlap. Unfortunately, if control is lost while switching buffers, any read operation which occurs while control is lost will see an end-of-file spanning file space which is not valid. In the present invention, this problem is avoided by postponing the end-of-file update until the entire APPEND data transfer completes, so that reads are protected from seeing invalid file space.

Obviously, many modifications and variations of the present invention are possible in light of the above teachings. It is therefore to be understood that the foregoing embodiments are presented by way of example only and that, within the scope of the appended claims and equivalents thereto, the invention may be practiced otherwise than as specifically described.