Method and system for archiving and compacting data in a data storage array

[0001] Embodiments of the present invention relate to managing data in data storage arrays. More specifically, the present invention pertains to a method and system for archiving and compacting data in a data storage array.

[0002] For many organizations, managing data storage resources is a time consuming process which is frequently prone to error. Data storage systems can quickly fill with redundant or outdated files which are occasionally purged from the system. Frequently, it is difficult to determine the ownership of the files being purged or whether those files are actually still needed. This results in mistakenly removing needed files or allowing outdated or redundant files to remain on the data storage array. Furthermore, it is a time consuming process for a person to examine these files and try to determine which files should be purged and which files should remain on the storage array.

[0003] For example, software development is frequently performed by teams of developers working in geographically separated locations. These development teams will use a centralized data storage area for storing the particular software component they are working on. Typically, these components resemble directories having hundreds of subdirectories and files. Other software development teams can access this data to ensure their components work together, to import dependencies, as well as in the final integration of the software components.

[0004] During the development cycle, there are frequent updates to the components, as well as patches of previous component versions, and/or multiple component versions to allow for the dependencies of other software components. Thus, in a short amount of time the centralized data storage area can fill with redundant or outdated files which must be purged. Occasionally, older files may be extremely stable components and are still considered the most current version. This complicates manually selecting and purging files because the age of the file is not an accurate indicator of whether the file should be purged. Additionally, this complicates identifying files which may be candidates for archival.

[0005] Often a list is distributed of the files which are to be purged. This allows developers to identify files which should remain in the centralized data storage area. However, it is a time consuming task for the developers to search the data component hierarchy to determine which files should be purged. This is increasingly true as directory structures become larger and more complex.

[0006] Thus, prior art methods for managing centralized data storage systems are time consuming and prone to error. Specifically, there is no method for automatically identifying files which are candidates for archival or removal from the data storage array.

[0007] Additionally, there is no method for consolidating files across multiple storage devices (e.g., multiple disk drives in a data storage array). Currently, compacting data in data storage arrays involves compacting each data storage device in the data storage array as a separate resulting in haphazard distribution of available free space among the separate devices. This results in less than optimal utilization of the free space in the data storage array. For example, collectively the free space in the data storage array may be large enough to accommodate large files and directories. However, prior art methods for compacting data result in multiple smaller areas of free space which, individually, may not be large enough to accommodate the larger files.

[0008] Accordingly, a need exists for a method and system for automatically identifying files which can be archived. While meeting the above stated need, it would be advantageous to provide a system which can consolidate free in data storage arrays comprising multiple data storage devices.

[0009] Embodiments of the present invention provide a method and system for archiving and compacting data in a data storage array. Embodiments of the present invention facilitate automatically identifying files which are candidates for archiving. According to one embodiment of the present invention, files are assigned a time period for maintaining the files in a data storage array. When the time period expires, users can be notified of an impending archival process in order to identify files which should be exempted from the list of archival candidate files. Files which are not identified as exemptions are automatically archived and removed from the data storage array.

[0010] Additionally, embodiments of the present invention facilitate compacting the remaining files across multiple data storage devices in the data storage array in order to consolidate free space into fewer, larger areas. In one embodiment, the free space on each of the storage devices in the data storage array is determined. A drive list is created in which the storage devices are sorted in descending order based upon their amount of free space. The storage device having the largest amount of free space is set aside to support the compacting process. The files on the largest remaining storage device are then sorted in descending order based upon their size. The largest file is compared to a minimum size threshold and, if it exceeds the minimum size threshold, is moved to a second storage device in the data storage array. This process continues until there are no more files remaining on the storage device larger than the minimum size threshold. The process is then repeated upon the next storage device in the drive list, etc.

[0011] Embodiments of the present invention are directed to a method and system for archiving and compacting data in a data storage array. In one embodiment, a plurality of archivable files are identified, archived, and removed from a data storage array. A first configuration of the data storage array is then determined in which the free space on each of a plurality of share storage devices in the data storage array is calculated. A process is performed for moving a plurality of file entities from a first of the shared data storage devices to other shared data storage devices in the data storage array. The moving process is repeated to create a second configuration of the data storage array. In the second configuration, the first shared data storage device has a greater amount of free space than in the first configuration.

[0023] Reference will now be made in detail to the preferred embodiments of the present invention, examples of which are illustrated in the accompanying drawings. While the present invention will be described in conjunction with the following embodiments, it will be understood that they are not intended to limit the present invention to these embodiments alone. On the contrary, the present invention is intended to cover alternatives, modifications, and equivalents which may be included within the spirit and scope of the present invention as defined by the appended claims. Furthermore, in the following detailed description of the present invention, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, the present invention may be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the present invention.

[0024] Notation and Nomenclature

[0025] Some portions of the detailed descriptions which follow are presented in terms of procedures, logic blocks, processing and other symbolic representations of operations on data bits within a computer memory. These descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. In the present application, a procedure, logic block, process, or the like, is conceived to be a self-consistent sequence of steps or instructions leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, although not necessarily, these quantities take the form of electrical or magnetic signal capable of being stored, transferred, combined, compared, and otherwise manipulated in a computer system.

[0026] It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the following discussions, it is appreciated that throughout the present invention, discussions utilizing terms such as “archiving,” “determining,” “calculating,” “performing,” “recognizing,” “creating,” “identifying,” “removing,” “comparing,” “adding,” “selecting,” “moving,” “sorting,” “repeating,” “excluding,” “designating” or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.

[0027] With reference to FIG. 1, portions of the present invention are comprised of computer-readable and computer-executable instructions that reside, for example, in computer system 100 which is used as a part of a general purpose computer network (not shown). It is appreciated that computer system 100 of FIG. 1 is exemplary only and that the present invention can operate within a number of different computer systems including general-purpose computer systems, embedded computer systems, and stand-alone computer systems.

[0028] In the present embodiment, computer system 100 includes an address/data bus 101 for conveying digital information between the various components, a central processor unit (CPU) 102 for processing the digital information and instructions, a volatile main memory 103 comprised of volatile random access memory (RAM) for storing the digital information and instructions, and a non-volatile read only memory (ROM) 104 for storing information and instructions of a more permanent nature. In addition, computer system 100 also includes a data storage array 105 for storing vast amounts of data. There are a variety of implementations of data storage array 105 upon which embodiments of the present invention may be utilized including: Storage Area Network (SAN) systems, Network Attached Storage (SAN) systems, Redundant Arrays of Multiple Disks (RAID), etc. It should be noted that the software program for archiving and compacting data of the present invention can be stored either in volatile memory 103, data storage array 105, or in an external storage device (not shown).

[0029] Devices which are optionally coupled to computer system 100 include a display device 106 for displaying information to a computer user, an alpha-numeric input device 107 (e.g., a keyboard), and a cursor control device 108 (e.g., mouse, trackball, light pen, etc.) for inputting data, selections, updates, etc. Computer system 100 can also include a mechanism for emitting an audible signal (not shown).

[0030] Furthermore, computer system 100 can include an input/output (I/O) signal unit (e.g., interface) 109 for interfacing with a peripheral device 110 (e.g., a computer network, modem, mass storage device, etc.). Accordingly, computer system 100 may be coupled in a network, such as a client/server environment, whereby a number of clients (e.g., personal computers, workstations, portable computers, minicomputers, terminals, etc.) are used to run processes for performing desired tasks. In particular, computer system 100 can be coupled in a system for archiving and compacting data in a data storage array.

[0031] FIG. 2 is a data flow and block diagram showing components used in the compacting of data in accordance with embodiments of the present invention. FIG. 2 will be referenced in conjunction with the following discussions to clearly describe embodiments of the present invention.

[0032] FIG. 3 is a flowchart of a computer implemented method 300 for archiving and compacting data in a centralized data storage array in accordance with embodiments of the present invention. In step 310 of FIG. 3, a plurality of files are archived. In embodiments of the present invention, all entities on data storage array 105, whether referring to a single file or group of related files (e.g., a directory), are generally managed in a similar manner. In one embodiment of the present invention, when a file is initially stored in data storage array 105, a time period for maintaining the file in the data storage array is assigned to the file. In one embodiment of the present invention, the archival process is initiated automatically at a predefined time interval.

[0033] Files which are candidates for archival are automatically identified when the time period for maintaining the files in the data storage array has expired. In one embodiment, a list of archival candidates is distributed to allow users to create ad-hoc exemptions to the archival rules. For example, if access to a file is still needed beyond the defined time period, the time period for maintaining the file on data storage array 105 can be extended. In one embodiment of the present invention, archiving comprises creating a copy of the selected files for storage at a remote location. The archived files, as well as files which are no longer needed, can then be removed from data storage array 105. The method for automatically archiving data is described in greater detail in FIG. 4.

[0034] In step 320 of FIG. 3, a first configuration of data storage array 105 is determined. After removing archived files and/or other unneeded files, it is likely that the individual data storage devices in data storage array 105 are fragmented. In embodiments of the present invention, free space in data storage array 105 is consolidated by moving files between the data storage devices of data storage array 105 to create larger areas of free space on some of the data storage devices. This allows users to store larger files on data storage array 105 after the compacting process has been performed. A first step in moving the files is to determine the first configuration of data storage array 105. In one embodiment of the present invention, the amount of free space and the total capacity of each of the data storage devices is determined.

[0035] In step 330 of FIG. 3, a process is performed for moving a plurality of files from a first data storage device in data storage array 105 to a second data storage device in data storage array to create a second configuration of the data storage array. Once the configuration of data storage array 105 has been determined, a process is then performed which moves files between the data storage devices. Again, in one embodiment of the present invention, all file entities (e.g., a single file or a group of related files such as a directory) are moved in a similar manner. Files are moved from a first data storage device to a second data storage device to create second configuration of data storage array 105 in which free space consolidated in fewer, larger areas. In one embodiment of the present invention, the data storage devices are sorted into a list (e.g., drive list 210 of FIG. 2) and the process for moving files is performed sequentially upon the list of data storage devices. For example, all of the files (e.g., file list 230 of FIG. 2) on the first data storage device which are within assigned parameters are moved from the first data storage device to a second data storage device within data storage array 105. The process for moving files between data storage devices is discussed in greater detail in FIG. 5.

[0036] In one embodiment of the present invention, the process for moving files is performed upon the half of the data storage devices having the greatest amount of free space. For example, if there are 30 data storage devices in the data storage array, the process for moving files is repeated until files have been moved from the 15 data storage devices having the greatest amount of free space. This expedites the moving process because at some point the benefit derived from moving files between the various data storage devices is out-weighed by the amount of time used to perform the process.

[0037] In so doing, the present invention provides a method and system for archiving and compacting data in a data storage array. Files are automatically identified which are candidates for consolidation. This reduces the workload associated with manually identifying files which can be archived. Additionally, the possibility of error in identifying files which are candidates for archival is greatly reduced. The present invention also provides a method and system for compacting files in a data storage array which consolidates available free space in the data storage array as a whole rather than a collection of separate data storage devices.

[0038] FIG. 4 is a flowchart of a computer implemented method 310 for archiving data in accordance with one embodiment of the present invention. Process 310 of FIG. 4 corresponds to step 310 of FIG. 3.

[0039] In step 410 of FIG. 4, an assigned logical date associated with a file is recognized. In accordance with embodiments of the present invention, the logical date defines the beginning of a time period for maintaining a file in data storage array 105. For example, in a software development environment the logical date can be a milestone in the development cycle defining when a software component should be available in data storage array 105 for access by other development teams. In accordance with embodiments of the present invention, the logical date can be assigned by a user when they initially store the file in data storage array 105. Alternatively, a logical date (e.g., a system timestamp) can be automatically assigned to the file if a user does not assign a logical date to the file.

[0040] In step 420 of FIG. 4, an assigned category associated with a file is recognized. In accordance with embodiments of the present invention, the assigned category defines a length of time for maintaining the file in data storage array 105. For example, a “category A” file is kept in data storage array 105 for 3 weeks after the logical date of step 410, while a “category B” file is kept for 2 weeks after the logical date, etc. Thus, the logical date in combination with the assigned category define a specific time period for maintaining a given file in data storage array 105. Additionally, the category can be used for implementing other rules or policies on specific files. This is discussed in greater detail in the discussion of step 430.

[0041] In step 430 of FIG. 4, the file is added to a list of candidate files. In one embodiment, a query is performed in which the length of time defined by the assigned category of step 420 is subtracted from the current date to define an archival date. If the logical date of step 410 precedes the archival date, the file is added to the list of candidate files. In other words, if the time window for maintaining the file in data storage array 105 has elapsed, the file is automatically added to a list of candidate files. The list of candidate files identifies files which are eligible for archival but have not yet been selected for archival.

[0042] For example, if the current date is January 31, and a file has a predetermined time period of 3 weeks, the archival date is January 10. In one embodiment, any file with a logical date which precedes the archival date is automatically added to the candidate list. Therefore, if a file has a logical date of January 9 or before, it is automatically added to the list of candidate files.

[0043] Additionally, in one embodiment of the present invention, files lacking either an assigned logical date or an assigned category are automatically added to the list of candidate files. This facilitates archival of files which were, for example, stored in data storage array 105 prior to the implementation of the present invention.

[0044] In accordance with embodiments of the present invention, files can be identified as exceptions to the archival process according to the category which has been assigned to them. Files preserved by policies or business rules are queried out of data storage array 105 and filtered out of the candidate list. For example, a policy can be defined as:

• • [0045] if a file has a logical date D earlier than the archival date, and a real timestamp later than the archival date, then all the files in the same category with logical timestamps later than D will also remain in data storage array 105.

[0046] In step 440 of FIG. 4, an archivable file is identified from the list of candidate files and a copy of the archivable file is created. In one embodiment, a list of the candidate files is distributed to allow users to see what files are on the archivable file list. Additionally, users can then identify ad-hoc exceptions to the archival process. For example, if a particular file is needed for a longer period than was originally anticipated, a user can identify that file as an ad-hoc exception to the archival process to extend the time period for maintaining that particular file in data storage array 105. Files which are identified by users as ad-hoc exceptions are removed from the candidate entities list. A copy of each of the files on the archivable file list is made for storage at a remote location.

[0047] In step 450 of FIG. 4, the archivable file is removed from data storage array 105. After a copy has been made of the archivable files, the archivable entities are removed from data storage array 105. Additionally, files which are no longer needed (e.g., files not archived and not identified as exceptions to the archival process) are removed from data storage array 105. In one embodiment of the present invention, information is maintained to facilitate future recovery of the file if necessary. For example, a repository file number, symbolic link, real path, archive date etc., is maintained to facilitate retrieving the archived file becomes necessary.

[0048] Thus, embodiments of the present invention provide a method for automatically identifying files which are candidates for archival. This reduces the possibility of human error in identifying archivable files while reducing the workload for users. Additionally, the method of the present can be adapted to the unique requirements and policies of the user.

[0049] FIG. 5 is a flowchart of a computer implemented process 330 for moving a plurality of files between a plurality of data storage devices in a data storage array in accordance with embodiments of the present invention. Process 330 of FIG. 5 corresponds to step 330 of FIG. 3. In step 510 of FIG. 5, a data storage device in data storage array 105 is selected. FIG. 6 is a flowchart showing in greater detail one embodiment of process 510 for selecting a data storage device.

[0050] Referring now to step 610 of FIG. 6, the plurality of data storage devices is sorted in descending order. In one embodiment of the present invention, the descending order is based upon the amount of free space remaining on each of the data storage devices as determined in step 320 of FIG. 3. Thus, a list of the data storage devices (e.g., drive list 210 of FIG. 2) is created wherein the device with the greatest amount of free space (e.g., device 211 of FIG. 2) is at the top of the list and the device with the least amount of free space (e.g., device 215 of FIG. 2) is at the bottom of the list.

[0051] In step 620, of FIG. 6, a data storage device in data storage array 105 is selected as a support device (e.g., support device 240 of FIG. 2). In one embodiment of the present invention, the data storage with the greatest amount of free space device (e.g., device 211 of FIG. 2) is set aside for use as a support device 240. In accordance with embodiments of the present invention, support device 240 can be used as temporary storage space when moving files between the various data storage devices in data storage array 105. Support device 240 is removed from drive list 210 to prevent including it in subsequent steps for compacting data.

[0052] In step 630 of FIG. 6, a data storage device in the shared storage array is selected as the first data storage device to be compacted. In one embodiment of the present invention, the remaining data storage device having the greatest amount of free space (e.g., device 212 of FIG. 2) is selected as the first data storage device to be compacted. Therefore, the first data storage device has less free space than support device 240, but has a greater amount of free space than any other data storage device in data storage array 105. Thus, a data storage device is selected for compacting from data storage array 105 in accordance with embodiments of the present invention.

[0053] Referring now to step 520 of FIG. 5, a file from the first data storage device is selected. FIG. 7 is a flowchart showing in greater detail process 520 for selecting a file in accordance with embodiments of the present invention. In step 710 of FIG. 7, the size of each file on data storage device 212 is calculated.

[0054] In step 720 of FIG. 7, the files are sorted. In one embodiment of the present invention, a file list 220 is created which lists the files on device 212 sorted in descending order based upon the size of the file. For example, the largest file on file list 220 (e.g., file 221 of FIG. 2) is at the top of the list 220 followed by successively smaller files with the smallest file (e.g., file 224 of FIG. 2) at the bottom of the list 220.

[0055] In step 730 of FIG. 7, a file is selected and compared to a size threshold. In one embodiment of the present invention, the largest file (e.g., file 221 of FIG. 2) is selected as the first file for moving. In accordance with embodiments of the present invention, the each file being moved is compared with a size threshold. In one embodiment, the size threshold defines the minimum size of a file being moved. While the present embodiment recites the size threshold defining the minimum size of a file being moved, the present invention is well suited utilizing the size threshold to define a variety of parameters.

[0056] Referring now to step 530 of FIG. 5, the selected file (e.g., file 221 of FIG. 2) is then moved from the first data storage device (e.g., device 212 of FIG. 2) to a second data storage device in data storage array 105 (e.g., device 214 of FIG. 2). FIG. 8 is a flowchart of a process 530 for moving a file from a first data storage device to a second data storage device in accordance with one embodiment of the present invention. Process 530 of FIG. 8 corresponds to step 530 of FIG. 5.

[0057] In step 810 of FIG. 8, each data storage is compared with a predetermined receiving threshold. For example, each of the data storage devices which is not the support device or the first data storage device being compacted can be added to a list of candidate receiving devices (e.g., candidate receiving list 230 of FIG. 2). In one embodiment of the present invention, the predetermined receiving threshold is a percentage of the amount of free space compared to the total capacity of the data storage device and is unique to each of the candidate receiving devices. This receiving threshold can be used to identify storage devices which may not have sufficient capacity for receiving additional files. For example, the receiving threshold can be set so that a data storage device with free space less than 20% of its total capacity is not eligible for receiving additional files. In one embodiment of the present invention, devices not having enough free space are removed from the list of candidate receiving devices 230.

[0058] In step 820 of FIG. 8, one of the data storage devices (e.g., device 214 of FIG. 2) is selected to receive the file from the first data storage device. There are a variety of rules for selecting the receiving data storage device. For example, in one embodiment, the first data storage device on the list of candidate receiving devices having enough free space to accommodate the file is as the receiving data storage device. In another embodiment, the data storage device having an amount of free space which most closely matches the size of the file being moved is selected as the receiving device.

[0059] In step 830 of FIG. 8, the file is moved from the first data storage device to the receiving data storage device. The selected file is moved from the data storage device upon which it resides to a second data storage device in data storage array 105. For example, file 221 is moved from data storage device 212 to data storage device 224 if device 224 exceeds its receiving threshold and has sufficient space for receiving file 221.

[0060] In step 840 of FIG. 8, the amount of free space on the receiving data storage device is recalculated. After receiving the file from the first data storage device, the amount of free space on the receiving data storage device 214 is recalculated and compared with its total capacity to determine whether device 224 is eligible for receiving additional files. If receiving data storage device 214 is still within the receiving threshold, it will remain on the list of candidate receiving devices 230, otherwise, it will be removed from the list of candidate receiving devices 230. In accordance with embodiments of the present invention, step 530 is repeated until there are no longer any files remaining on the data storage device that are larger than the minimum size threshold. When this occurs, the data storage device being compacted is removed from the list of data storage devices and the compacting process is repeated. For example, the amount of free space on the remaining data storage devices is determined, the devices are sorted according to the amount of free space on each device, and the process for moving files is initiated upon the device (e.g., device 213 of FIG. 2) having the greatest amount of free space.

[0061] An example of the high-level logic describing process 530 can be exemplified in the following pseudo-code where “driveList” describes a list of data storage devices in data storage array 105 and fileList describes a list of the files on a given data storage device:

• [0062] for a file in fileList {
  • [0063] if (size (file)>FILE_SIZE_THRESHOLD) {
    • [0064] s<- find slot on a D^r from (D³ . . . D^x) using first fit rule:
    • [0065] (or best-fit rule, or other rule defined for the specific
    • [0066] situation)
    • [0067] if (slot s found) {
      • [0068] move file to s:
      • [0069] decrease free space on D^r by size(file);
      • [0070] if (free space (D^r)/space(D^r)<threshold) {
        • [0071] driveList <_driveList-D^r:
      • [0072] }
    • [0073] }
  • [0074] }
• [0075] }

[0076] In accordance with embodiments of the present invention, the number of times this process is performed can be set as a parameter. For example, if the parameter is set to only perform one “loop” of process 330, files will only be moved from the first data storage device in data storage array 105 (e.g., device 211 of FIG. 2). If the parameter is set to perform two loops, only two data storage devices (e.g., devices 211 and 212 of FIG. 2) will be compacted.

[0077] In one embodiment of the present invention, another parameter allows a user to determine which devices in data storage array 105 will be compacted. For example, in a data storage array having 36 data storage devices, the data compacting process could be limited to devices D4–D14.

[0078] In so doing, a process for archiving and compacting data in a data storage array is performed in accordance with embodiments of the present invention. The process for compacting data in the present invention consolidates free space in the data storage array as a whole rather than consolidating free space in each data storage device separately. This results in fewer, larger areas of free space in the data storage array which can accommodate larger files.

[0079] FIGS. 9A and 9B are a flowchart of a computer implemented method 900 for archiving data stored in a centralized data storage array (e.g., data storage array 105 of FIG. 1) in accordance with embodiments of the present invention. Method 900 is used to automatically identify files which are candidates for archiving and for archiving and removing the files from a data storage array.

[0080] In step 905 of FIG. 9A, a logic operation is performed to determine whether a logical date has been assigned to a selected file (e.g., file 221 of FIG. 2). If a logical date is recognized, the selected file proceeds to step 910 of FIG. 9A, otherwise, the selected file is automatically added to a list of candidate files (e.g., candidate list 935 of FIG. 9B). The logical date defines a beginning of a time period for maintaining the selected file in data storage array 105.

[0081] In step 910 of FIG. 9A, a logic operation is performed to determine whether a category has been assigned to a selected file. If a category has been assigned to the selected file, the selected file proceeds to step 920 of FIG. 9A, otherwise, the selected file is automatically added to a list of candidate files (e.g., candidate list 935 of FIG. 9B). The category is used to define the length of the time period for maintaining the selected file in data storage array 105.

[0082] In step 915 of FIG. 9A, the current date is determined.

[0083] In step 920 of FIG. 9A, the time period defined by the category of step 910 is subtracted from the current date. For example, if the time period for maintaining the selected file in data storage array 105 is 3 weeks, and the current date is January 31, an archival date (e.g., archival date 925 of FIG. 9A) of January 10 is defined.

[0084] In step 930 of FIG. 9A, a logic operation 930 is to determine whether the time period for maintaining the selected file in data storage array has elapsed. For example, in one embodiment of the present invention, if the logical date precedes the archival date, the time period for maintaining the selected file has elapsed and the selected file will automatically be added to candidate list 935 of FIG. 9B. In the example cited above, it the logical date is January 4, and the archival date is January 10, the selected file will automatically be added to candidate list 935. If the time period for maintaining the selected file on data storage array has not elapsed, the file will remain on data storage array 105 (e.g., step 931 of FIG. 9A).

[0085] Referring now to step 940 of FIG. 9B, a logic operation is performed to determine whether any of the files on candidate list 935 are exceptions to the archival process. In one embodiment, category 910 of FIG. 9A can also include policies or rules which will identify a selected file as an exception to the archival process. Files preserved by policies or business rules are queried out of data storage array 105 and filtered out of the candidate list. For example, a policy can be defined as:

• • [0086] if a file has a logical date D earlier than the archival date, and a real timestamp later than the archival date, then all the files in the same category with logical timestamps later than D will also remain in data storage array 105.

[0087] In accordance with embodiments of the present invention, a list can be distributed to allow users to identify files which are ad-hoc exceptions to the archival process. Ad-hoc exceptions to the archival process are files which are not exempted from archival based upon their category, but which should be maintained in data storage array 105. For example, if access to a file is still necessary after the time period for maintaining the file on data storage array 105 has expired, a user can identify that file as an ad-hoc exception to the archival process. The file will then be removed from candidate list 935 and the selected file will remain on data storage array 105 (e.g., step 941 of FIG. 9B).

[0088] In step 945 of FIG. 9B, an archivable file list is created. In one embodiment of the present invention, archivable file list 945 comprises the files from candidate list 935 which were not identified as exceptions to the archival process. In one embodiment of the present invention, users can identify files which are no longer needed on data storage array 105 and which can be deleted. A copy of each of the files on archivable file list 945 is made for storage at a remote location. In one embodiment of the present invention, information is kept to facilitate future recovery of the file if necessary. For example, a repository file number, symbolic link, real path, archive date etc., is kept to facilitate retrieving the archived file becomes necessary.

[0089] In step 950 of FIG. 9B, files on the archivable file list are copied for storage at a remote location. In step 955 of FIG. 9B, the archived files are removed from data storage array 105. Once a copy of one of the archivable files is created, the file is removed from data storage array 105.

[0090] FIGS. 10A, 10B, and 10C are a flowchart of a computer implemented method 1000 for compacting data stored in a data storage array in accordance with embodiments of the present invention.

[0091] In step 1001 of FIG. 10A, the free space on each of the data storage devices comprising data storage array 105 is determined. In embodiments of the present invention, the total capacity of each of the data storage devices is determined as well.

[0092] In step 1002 of FIG. 10A, the data storage devices are sorted in descending order. In embodiments of the present invention, the data storage devices are sorted in descending order based upon the amount of free space as determined in step 1001. Referring to FIG. 2, drive list 210 lists the data storage devices comprising data storage array 105. The data storage devices are listed in descending order (e.g., device 211–device 215) based upon the amount of free space remaining on each drive.

[0093] In step 1003 of FIG. 10A, a support device is selected. In one embodiment, the data storage device with the largest amount of free space is selected as the support device. The support device can be used to facilitate moving files between the data storage devices during the compacting process. Referring to FIG. 2, device 211 is selected as support device 240 because it has a greatest amount of free space than any other data storage device in data storage array 105.

[0094] In step 1004 of FIG. 10A, a data storage device is selected for compacting. In one embodiment of the present invention, the data storage device with the largest amount of free space of the remaining data storage devices on drive list 210 (e.g., device 212 of FIG. 2) is selected as the first data storage device to be compacted.

[0095] In step 1005 of FIG. 10A, the file sizes on the selected data storage device are calculated. In embodiments of the present invention, the files residing on the selected data storage device are sorted in descending order based upon their size. Referring to FIG. 2, file list 220 lists files 212–224 in descending order based upon the size of each file.

[0096] In step 1006 of FIG. 10A, the largest file on the selected data storage device (e.g., file 221 of FIG. 2) is selected. In accordance with embodiments of the present invention, the largest file on a data storage device selected for compacting is the first file to be moved.

[0097] Referring now to FIG. 10B, in step 1007 a logic operation is performed in which the file from step 1006 is compared with a predetermined size threshold. In embodiments of the present invention, the size threshold defines the minimum size a file must be in order to be moved to a different data storage device. In one embodiment, the size threshold is set per system (e.g., computer system 100 of FIG. 1).

[0098] If the selected file (e.g., file 221 of FIG. 2) is larger than the predetermined size threshold, process 1000 proceeds to 1011 of FIG. 10C. If the selected file is smaller than the predetermined size threshold, there are no files on the selected data storage device which are large enough to move. Process 1000 then proceeds to step 1008.

[0099] In step 1008 of FIG. 10B, a logic operation is performed to determine whether another data storage device of data storage array 105 should be compacted.

[0100] In embodiments of the present invention, parameters can be set to limits the number of data storage devices being compacted. For example, a parameter can be set limiting the compacting process to a specific number of devices such as the first five devices in data storage array 105. In another embodiment, a parameter can be set limiting the compacting process to a specific set of data storage devices in data storage array 105. For example, if data storage array 105 is comprised of 30 data storage devices, the parameter can be set to only compact drives 5–12.

[0101] In another embodiment of the present invention, the compacting process is performed upon no more than one half of the total number of data storage devices in data storage array 105. Thus, if data storage array is comprised of 36 data storage devices, the compacting process is performed upon the first 18 data storage devices on drive list 210.

[0102] Referring again to step 1008 of FIG. 10B, if there are no more data storage devices in data storage array 105 to be compacted (e.g., no more data storage devices on drive list 210 of FIG. 2), process 1000 ends. If there are other data storage devices remaining on drive list 210, process 1000 continues at step 1018.

[0103] In step 1018 of FIG. 10B, the previously selected data storage device is excluded from process 1000. For example, at some point there are no files remaining on device 212 for moving to other data storage devices. Data storage device 212 is then removed from drive list 212 as it will not be included in process 1000.

[0104] In step 1020 of FIG. 10B, the free space on the data storage devices remaining on drive list 210 is recalculated. At this point, process 1000 returns to step 1004 of FIG. 10A.

[0105] Referring now to step 1011 of FIG. 10C, a receiving data storage device is (e.g., device 215 of FIG. 2) selected. A second data storage device (e.g., device 215 of FIG. 2) is selected from drive list 210 to receive the selected file from data storage device 212.

[0106] In step 1012 of FIG. 10C, a logic operation is performed in which the receiving threshold of the selected receiving device is compared with the amount of available free space. In embodiments of the present invention, the amount of free space is compared to the total capacity of the selected data storage device, and if the amount of free space exceeds the receiving threshold, the selected device can receive files from other data storage devices. For example, if the receiving threshold is set at 20%, at least 20% of the total capacity of the selected data storage device should be free space.

[0107] If the amount of free space is larger than the receiving threshold, process 1000 continues to step 1013. If the amount of free space on the selected receiving data storage device is less than the receiving threshold, process 1000 continues to step 1016.

[0108] In step 1016 of FIG. 10C, the selected receiving device is removed from the list of data storage devices. Because the amount of free space does not exceed the receiving threshold, the selected data storage device is not a candidate for receiving files during process 1000 and is removed from drive list 210. At this point, a new receiving data storage device is selected and process 1000 proceeds from step 1011.

[0109] Referring now to step 1013 of FIG. 10C, a logic operation is performed in which the size of the selected file is compared with the available free space on the receiving data storage device. If the selected file is larger than the amount of free space on the selected data storage device, process 1000 continues to step 1017.

[0110] In step 1017 of FIG. 10C, a different receiving device is selected from drive list 210. If the selected file is smaller than the amount of available free space on the selected receiving device, process 1000 proceeds to step 1014. In step 1014 of FIG. 10C, the selected file is moved to the selected receiving data storage device. In step 1015 of FIG. 10C, the amount of free space remaining on the receiving device is recalculated. This is to determine whether the data storage device has enough available free space after receiving the file to exceed the receiving threshold and receive additional files.

[0111] At this point, process 1000 returns to step 1006 and selects the largest file remaining on file list 220 for moving to another data storage device.

[0112] In so doing, a method for compacting data in a data storage array is performed in accordance with one embodiment of the present invention. The present invention consolidates free space in data storage arrays by compacting the data storage array as a whole rather than a collection of separate data storage devices.

[0113] The preferred embodiment of the present invention, a method and system for archiving and compacting data in a data storage array, is thus described. While the present invention has been described in particular embodiments, it should be appreciated that the present invention should not be construed as limited by such embodiments, but rather construed according to the following claims.