System and method for providing property histories of objects and collections for determining device capacity based thereon

[0002] The invention relates generally to the field of computer system storage and more particularly to a system and method for providing property histories of objects for more accurate forecasting of computer system storage capacity.

[0003] In the capacity planning process for computer system storage, data is collected on the use of storage in the environment, the data is analyzed to predict the future utilization of the container, and system states are defined (e.g., thresholds) that trigger actions to prevent containers from filling to their limits. The current practice of storage capacity planning collects information from the storage environment only about the utilization of containers without regard for their content. That is, when observations of the containers are made, the content of the container (i.e. objects residing in the container) is ignored. The size of each object in the storage environment is also not recorded or retained as the size history of that object. Likewise, predictions about future utilization are based on containers rather than individual objects; even though additions and deletions of individual objects and changes to objects can affect the utilization of the container.

[0004] Current storage capacity planning practices use a percent utilization threshold to initiate action. For example, if the container is projected to reach 90% utilization, then action must be taken. The growth rates of utilization of containers, however, may vary significantly. If the combined growth of the size of objects now in a container is rapid, then the utilization threshold may not allow enough time to act. On the other hand, if the growth is very slight, then a static threshold may be very aggressive and cause undue action.

[0005] It is, therefore, an object of the present invention to provide a system and method for maintaining a historical record of individual objects residing on computer storage.

[0006] It is another object of the invention to provide an improved method for forecasting computer storage capacity by utilizing historical records of individual objects residing on the computer storage.

[0007] These and other objects are realized by the present invention comprising a system and method for managing storage resources in a computer system. The invention periodically tracks and stores the history of each object stored in the storage resources. For each storage container, the life expectancy is calculated by utilizing the object histories for all objects currently stored in that container.

[0008] The invention will now be described in greater detail with specific reference to the appended figures wherein:

[0009] FIG. 1 provides a schematic drawings of a system in accordance with the present invention;

[0010] FIG. 2 provides a representative process flow for collecting object histories;

[0011] FIG. 3 provides a representative process flow for constructing a container utilization history; and

[0012] FIG. 4 provides a representative process flow for calculating the life expectancy for a container.

[0013] This invention is directed towards managing storage resources in a computer system. In an environment in which the use of storage grows rapidly, administrators and capacity planners require information on growth to prevent service outages caused by storage filled to capacity, and to balance resource usage across multiple storage devices. For purposes of the ensuing description, the following glossary of terms may assist in a complete understanding of the invention:

[0014] A processor is a computer system with the capability to execute computer software and programs. Here the term processor refers to central processing unit along with the memory and input/output interfaces for transferring digital data between the inside of the system and the outside world. It does not include any devices for the permanent storage of data. Also included is the operating and support software, i.e. operating system and support subsystems that allow the hardware devices to be used.

[0015] Storage generically means those devices used to permanently store and retrieve digitized data where the data can be overwritten and the storage can be reused.

[0016] A storage system is the physical storage device, related devices and methods, connection hardware between the processor and storage, any intermediate mechanisms that function in the transfer of data between processors and storage, and software that is used to identifying and managing stored data.

[0017] A storage environment of a processor is the set of one or more storage systems where the processor can read and/or write data permanently into storage.

[0018] A storage object is an identified collection of digital data which is kept together within the storage subsystem and has a name (identifier) unique to the storage subsystem. In most computer systems, this is called a file or a table in relational database systems. A storage object has an external property which is the amount of storage necessary to contain the object and is called the size of the objects, usually measure in bytes of data. The storage subsystem maintains the association of the objects name with the area of storage where the object resides. The processor access and updates the object by reference to the object's identifier. Over time, the object can be stored in different locations of the storage subsystem without impacting the processor's ability to access the object. Storage objects can be also be a collection of smaller storage entities managed by the storage subsystem. For example, in many UNIX systems, users files are collected together into volumes rooted in the file system at names like /home/user. This collection is always, in practice, kept together and the collection would be moved as an single object. Therefore the collection can be considered an object and the aggregate size of the collection is the size of the object.

[0019] A container is a generalized term that represents an identifiable and limited part of storage subsystem. The limit is measured in the same unit as the storage objects. In some systems a container is a physical direct access storage device (DASD). However, a physical drive can be divided into two or more logical containers called partitions (as in UNIX, among other systems). Other storage systems can combine several physical disk drives to for a file system (as in the AIX logical volume group). In either case, there is a limit which cannot be exceeded without external intervention in the operation of the processor interacting with the storage subsystem. A storage subsystem has one or more storage containers and each container can hold zero or more storage objects up to but not exceeding its limit. The processor has access to the storage subsystem(s) in its storage environment. Through the storage subsystems, it can identify all the containers in each subsystem and the limits of those containers and it can identify all the objects in each container and the size of each object.

[0020] A repository is a means for storing structured data external to the processor. Data in the repository is saved and accessed in a storage subsystem but is also supported by software, such as relational database software, that provides access to the structure of the data. Database software is not essential for a repository as the content of the repository can be stored in simple storage objects, frequently called a flat file.

[0021] Free space represents the amount of storage in a container that does not hold objects and is available to store new objects or allow for the expansion of existing objects.

[0022] The capacity of a container is its storage limit. The capacity of a storage subsystem is the sum of the limits of each container within the subsystem. The capacity of a storage environment is the sum total of the capacity of all storage subsystems within the environment.

[0023] Utilization of a container is the sum total of all the sizes of the objects in the container plus any storage used to manage and identify the storage objects (like directory space in UNIX systems). Utilization of a subsystem is the sum of the utilization of each container in the subsystem and utilization of the environment is the sum of the utilization of all subsystems within an storage environment. Utilization is usually expressed as a percentage of the capacity of the container.

[0024] Capacity planning is the process of projecting and managing utilization of a storage environment by projecting and managing the subsystems and containers within the environment. Capacity planning is depicted by three high level steps: (1) collecting information about the storage environment, (2) projecting the state of the storage environment at some time in the future, and (3) defining actions to prevent the utilization of any container reaching the container's limit. Although a single snapshot of the storage environment can be used for projecting, the projections are usually done using historical information. This invention addresses the methods used in steps (1) and (2).

[0025] A threshold is an artificial limit on utilization that is used by the capacity planning process to prioritize containers that need action. When the projected utilization of a container exceeds the threshold, the container is selected by the process to be managed and actions will be defined (in step 3).

[0026] The life expectancy of a container is the period of time that the utilization of the container is expected to be less than the capacity of the container. If the utilization of the container is projected to always be less than the capacity, that is the projected change in utilization is constant or diminishing, then the life expectancy of the container is expected to be infinite. Under other conditions, where the life expectancy is limited, the capacity planning process must identify actions that prevent the container from filling up.

[0027] Action horizon is the period of time needed to take action to change a storage subsystem.

[0028] Two aspects of this invention extend and improve the current state of the art in storage capacity planning. First, this invention calls for the recording of the size of individual storage objects in the environment and saving the size information in a location-independent way. Storage objects can move between containers. Therefore, the life span of an object may exceed the time period that it resided in its current container. Recording the history of an object, regardless of where it resided, provides accurate information of the size of that object in the past. Second, this invention projects container utilization based on the histories of the individual objects currently residing in the container. These histories are combined to create a history of the utilization of the container as if it had, in the past, only held the current inventory of objects. In this way, the errors inherent from the previous addition and deletion of objects is eliminated. Since the history of the objects can be longer than their residency in their current container, there are more observations with which to make a projection. The combination of the combining of histories and the added length of the histories may result in more accurate projections of future utilization of the container. More accurate projections allow for the more accurate management of storage environments.

[0029] The figure of merit used by this invention is life expectancy. Life expectancy takes into account not only the current utilization of the container but also the growth in utilization, if the utilization in increasing over time. Life expectancy normalizes the projection to the same dimension for all containers and it projects, in the same unit, as the lead time needed to take preventive action. The threshold in the process is set to the time (e.g., days) needed to take preventive action (the action horizon). The life expectancy of each container is compared with the time horizon threshold and those containers that fall below the threshold are singled out for action. This benefits the storage capacity planning process in two ways. First, effort on containers with slow growth can be deferred, saving effort and interruption to the storage system. Second, containers with rapid growth are identified and singled out for action, thus preventing emergency action and/or system failure. Using life expectancy makes the comparison of the action horizon and the change in the container utilization more accurate and less prone to error than conventional methods.

[0030] FIG. 1 provides an illustration of a system where this invention would apply. Processor 1 includes a computer process in accordance with the invention. The processor would have access to a data repository 2, such as a relational database, wherein data are saved and retrieved when needed. The processor also has access to data about properties of the storage environment 3 of the computing installation and about objects that are part of the storage environment. The objects of interest for this invention are containers 4 that exist in the storage system, such as a file system. A container stores data objects 5, such as computer files.

[0031] Each container has a unique identifier called the container name. A container j also has a limit (L_j), called the container size, on the number of bytes of information that the container can hold. Objects 5 stored in the container also each have an identifier that is unique within the environment 3. Each storage object i uses some amount of storage space (S_i), measured in bytes, in the container where they are currently stored. The size of storage objects can change over time (S_i(t)) with the object taking more or less space in the container, where t represents some point in time. Objects can be moved from one container to another, but they retain their unique identifier. Storage in the container that is not occupied by objects is called free space (F_j). Free space of a container j is calculated by subtracting the sum of the sizes of all the objects in j from the size limit of the container L_j:

[0032] FIG. 2 details how object histories are collected. An object history is the record of the size of a specific object over time, S_i(t). It is preferred that object history information be collected periodically; therefore, at the end of a time period shown at 100, the collection process begins. At 101, a list of all the containers in the environment is developed dynamically by searching the storage environment for storage objects. The process then loops through the list of containers until all objects in all containers have been processed via steps 102-109. One at a time, each container name is selected from the container list, at 103, for processing to update the history of objects within that container. A list of objects in the container is created at 104. Unless the object list is empty, as determined at step 105, the inventive process loops through the list of objects in the selected container until all the objects have been processed by steps 106-109. First, at 106, the name of the object is selected from the list. Next, the size of the object is determined at 107. Thereafter, the names and the size are checked for accuracy (for example, the size cannot be negative) at step 108. Finally, the data (time of the observation, identifier of the object, and its size) for this object are recorded in the data repository 2 at step 109. Once all of the objects in the list for the selected container are processed, as indicated by a “yes” answer at step 105, then the process returns to step 102. If, as determined at step 102, the container list is empty, such that the histories for all objects in all containers have been updated, then the process returns to the beginning, i.e., step 100, to wait until the end of the next time period.

[0033] FIG. 3 details how the container utilization history is constructed in steps 200 through 211. First, a list of containers in the environment is created at step 200. First, a name of a container is selected from the container list at 203. Next, at step 205, the utilization history for the container is initialized with the identifier of the container and an empty vector for the utilization history. The empty vector will later be replaced with the combined histories of the objects currently held in the container. A container utilization history is a matrix with two columns, with the first column representing a time period and the second column representing the utilization of the container. The length of the container history is a parameter, which need not coincide with the history of any particular object in the container. Utilization is defined as the sum of the size of the objects in the container at time t, U_j(t)=Σ_viejS_i(t)) After the matrix has been initialized, a list of the names of the objects currently stored in the container is obtained at step 206, for example by referencing a directory or inspecting the container, as discussed above with reference to steep 101 of FIG. 2. The process then loops through the list of objects until all have been processed by steps 207-210. Once it has been determined, at step 207, that the object list is not empty, then one item is selected from the object list at step 208. The identifier of the object is used to retrieve the history of that object from the repository in step 209. That retrieved object history is combined with the utilization history of the container at step 210. This is done by adding the size of the object at each point in time represented in the utilization matrix from 205. It may be necessary to interpolate the size if the history of the object and the utilization history time periods do not coincide. For any time periods in the container utilization history for which there are no object history observations, no updates are made to the utilization history. Any time periods that extend back before the beginning of the container utilization history are ignored. When the object list is exhausted, as determined at step 207, the utilization history along with the identifier of the container is written back to the repository at step 211. The utilization history of the container now represents a combined history of all the objects that are currently stored in the container, without regard to where those objects were stored at previous times.

[0034] The final part of the invention is the calculation of the life expectancy for each container. The life expectancy is defined as the projected amount of time remaining before the container is full, if the change in the cumulative size of the objects in the container over time is increasing. This is calculated as illustrated in step 300-309. The list of containers in the environment is formed at step 300. The loop comprising steps 302-309 processes all the containers in the list. First, once it is determined that there is an entry in the list, at step 302. At step 303, a container identifier is selected from the list. Next, the current size limit of the container is determined (L_j) at step 304. Then, the utilization history for the container is retrieved from the repository at 305. A model of the change in utilization is constructed and parameters of the model are calculated at 306. Two values are projected from the model at step 307: the utilization of the container at time t as projected by the model, U_j(t), and the rate of change in utilization, C_j(t). Note that a variety of modeling techniques could be used to forecast storage usage within the scope of this invention. If the change in utilization over time is positive, as determined at step 308, then the utilization is growing and the container will eventually fill. The calculation of life expectancy can be represented by the general formula E_j(t)={L_jU_j(t),C_j(t)}. Under the conditions of linear growth, the life expectancy for container j at time t is calculated, at step 309, by the formula E_j(t)=(L_j−U_j(t)/C_j(t)). The life expectancy metric combines resource utilization, growth in utilization, and capacity constraints into a single, intuitive, device- and system-independent measure. Computing life expectancy requires historical data on resource usage and a means to forecast future usage, but the metric is independent of the forecasting method. Traditionally resource usage is monitored in aggregate, at the container level. If the contents of the container (i.e., objects) can change over time, keeping historical resource usage data on objects is required to accurately compute life expectancy. When employing the current invention, the life expectancy of a container is a more accurate measure due to the fact that the life expectancy reflects the objects currently residing within it.

[0035] The description of this invention is expressed entirely for a storage environment, but the methods and system have broader application. The application of this invention is easier to express for a storage environment because the storage limits (capacity) and the object sizes are easily identified. However, other environments have similar limits and utilization with the same three step capacity planning process.

[0036] To apply this invention to other environments, the identification of containers, objects, limits and sizes must be translated into the new environment. For example, to apply this invention to the execution capacity of a computer system, we would use a measure of CPU power, such as instructions per second. Each computer system (a container) would be assigned a limit (the capacity of the system) and the CPU usage of each user or computer job would be recorded. As users or jobs can be assigned to different computer systems, the history of their usage would be combined as with storage objects, to provide a projects of usage. This projection, along with the CPU capacity is used to define a life expectancy for the CPU capacity of the system. The same concept can be applied to network bandwidth where the capacity is the total number of bits that can pass through a network link, which is the analog of the container in this model. History of each user traffic on the link is recorded and the capacity of the link can be managed in a fashion similar to system storage or CPU time. Other applications of the model can be identified whenever the essential elements of container(s), their capacity, the identifiable objects in the containers and their resource usage can be recorded. The invention has been described with reference to preferred embodiments. One having skill in the relevant art will recognize that modifications can be made without departing from the spirit and scope of the invention.