Tracking memory space in a storage system

[0001] This application relates to the following patent applications which are filed concurrently herewith and incorporated herein by reference: Ser. No. 12/261,064 entitled ALLOCATING PRIORITIES TO PREVENT DEADLOCKS IN A STORAGE SYSTEM; PCT/US08/81661 entitled CREATING SNAPSHOTS OF A FILE SYSTEM; Ser. No. 12/261,068 entitled MANAGING COUNTERS IN A DISTRIBUTED FILE SYSTEM; PCT/US08/81664 entitled ONLINE CHECKING OF DATA STRUCTURES OF A FILE SYSTEM; and Ser. No. 12/261,071 entitled ENUMERATING METADATA IN FILE SYSTEM DIRECTORIES.

[0002] Filesystems track where data is stored in memory and where free or available space exists in memory. Bitmaps are one way to track this free space.

[0003] A bitmap is an array of bits in which each bit corresponds to a block of memory. A logical 1 or 0 indicates whether the block of memory is available for storage or full. Bitmaps generally require low overhead since each bit represents a relatively large block of memory, for example 1 bit for each 4K block size.

[0004] The size of the bitmap is easily managed for smaller filesystems but can become inefficient or unmanageable for much larger filesystems. For example, a 32 KB bitmap could be used for a 1 GB filesystem, or a 23 MB bitmap used for a 1 TB filesystem. These bitmaps are small and easily managed. For a much larger filesystem, such as a 1 PB filesystem, 32 GB bitmap would be needed. Managing and scanning a bitmap of this size is time consuming and costly.

[0005] Furthermore, traditional bitmaps do not perform well when directly applied to a symmetric clustered file system. Performance is degraded due to cross-server contention over the shared allocation data structures as the servers in the cluster concurrently allocate and free space from the file system. An extension to this approach is a dynamic partition scheme, where the cluster nodes agree to allocate from separate regions of the file system to avoid cross-server contention over the allocation data structures. This approach, however, degenerates as the file system becomes full. As the nodes concurrently search for the remaining free space on the file system, lock contention occurs which slows the performance of the file system.

[0010] Exemplary embodiments relate to tracking and managing free or available memory space in a storage system. A hierarchical bitmap system or free space map includes multiple bitmap layers that track free space available in a cluster file system.

[0011] In one embodiment, a free space map or memory allocation map tracks available (i.e., free) and unavailable (i.e., used) memory space in a storage system. The filesystem uses bitmaps or tables to track which data storage blocks are free and which data storage blocks are used. A hierarchical storage scheme is used so these bitmaps or tables can be quickly searched to determine a location of available memory.

[0012] In one embodiment, a lowest layer of the hierarchical bitmap system includes multiple bitmap blocks with each bit corresponding to a block of memory. A second or higher layer in the hierarchy is located above the lowest layer. This second layer includes a bitmap array with each bit corresponding to one of the bitmap blocks in the lowest layer. A “1” or “0” in this second layer indicates whether its corresponding bitmap block in the lowest layer has available memory or is full.

[0013] The hierarchical bitmap system enables the filesystem to quickly determine a location of free memory without searching through the entire sequence of bitmap arrays. Bitmap blocks located at the top of the hierarchical bitmap tree represent multiple different blocks of memory or multiple different bitmap blocks. As such, large sections of the bitmap blocks are not required to be searched since a bit higher in the tree corresponds to numerous bitmap blocks. For instance, a logical “1” in a higher branch of the bitmap tree indicates that plural bitmap blocks in a lower branch all have no available memory. Hence, these lower bitmap blocks are not required to be searched for available memory since an indication (i.e., logical “1”) in the higher branch indicates these memory blocks in lower branches are all full.

[0014] FIG. 1 shows a distributed or cluster storage system 100 in accordance with an exemplary embodiment of the present invention. By way of example, the system is a cluster storage network and/or a storage area network (SAN) that includes a plurality of client computers, nodes, or host computers 102 and one or more storage devices or arrays 103A, 103N that include one or more storage controllers 104 (shown by way of example as an array controller), a plurality of storage devices 106 (shown by way of example as disk array 1 to disk array N), and a file system manager 108 in communication with the storage controllers and devices.

[0015] The host computers 102 (shown as host 1 to host N) and file system manager 108 are coupled to the array controllers 104 through one or more fabrics or networks 110, and the storage devices or arrays 103 are coupled to the storage devices 106 through one or more fabrics or networks 111. For instance, the hosts communicate with an array controller using a Small Computer System Interface (SCSI) or other interface/commands over a fiber channel (FC). By way of example, networks 110 and 111 include one or more of the Ethernet, fibre channel (FC), serial attached SCSI (SAS), iSCSI, internet, local area network (LAN), wide area network (WAN), public and/or private networks, etc. Communications links 112 are shown in the figure to represent communication paths or couplings between the hosts, controllers, and storage devices.

[0016] The filesystem manager 108 stores and organizes computer files so the files and corresponding data can be managed and discovered for the host or client computers 102. By way of example, in one embodiment the file system manager is included in a server (such as being a file system driver in a server computer). As discussed in more detail below, the filesystem manager 108 uses a hierarchical storage scheme (such as a bitmap tree) to track and/or determine used and free storage space in the storage system.

[0017] In one exemplary embodiment, the storage devices (such as array controller 104 and disk arrays 106) are network attached devices providing random access memory (RAM) and/or disk space (for storage and as virtual RAM) and/or some other form of storage or storage device, such as magnetic memory (example, tapes), micromechanical systems (MEMS), or optical disks, to name a few examples. Typically, storage devices include larger amounts of RAM and/or disk space and one or more specialized devices, such as network disk drives or disk drive arrays, (example, redundant array of independent disks (RAID)), high speed tape, magnetic random access memory (MRAM) systems or other devices, and combinations thereof. In one exemplary embodiment, the storage devices include one or more servers.

[0018] The storage controller 104 manages various data storage and retrieval operations. Storage controller 104 receives I/O requests or commands from the host computers 102, such as data read requests, data write requests, maintenance requests, etc. Storage controller 104 handles the storage and retrieval of data on the multiple disk arrays 106 and disk groups. In one exemplary embodiment, storage controller 104 is a separate device or may be part of a computer system, such as a server. Additionally, the storage controller 104 may be located with, proximate, or a great geographical distance from the disk arrays 106 or from each other.

[0019] The array controller 104 includes numerous electronic devices, circuit boards, electronic components, etc. By way of example, the array controller 104 includes firmware 120, an input/output (I/O) scheduler 122, a queue 124, one or more interfaces 126, one or more processors 128 (shown by way of example as a CPU, central processing unit), and memory 130 (including read and write cache). CPU 128 performs operations and tasks necessary to manage the various data storage and data retrieval requests received from host computers 102. For instance, processor 128 is coupled to a host interface 126A that provides bidirectional data communications to one or more host computers 102. Processor 128 is also coupled to an array interface 126B that provides bidirectional data communications to the disk arrays 106.

[0020] Memory 130 is also coupled to processor 128 and stores various information used by processor when carrying out its tasks. By way of example, memory 130 includes one or more of volatile memory, non-volatile memory, or a combination of volatile and non-volatile memory. The memory 130, for example, stores applications, data, control programs, algorithms (including software to implement or assist in implementing embodiments in accordance with the present invention), and other data associated with the storage device (example, state data such as mapping metadata, configuration metadata, and cached user data). The processor 128 communicates with memory 130, interfaces 126, and the other components via one or more buses 132.

[0021] FIG. 2 shows a free space map, memory allocation map, or hierarchical bitmap system 200 in accordance with an exemplary embodiment of the present invention. The system includes multiple bitmap blocks arranged in a hierarchical structure with multiple layers, branches, or levels. A bottom or lowest layer or branch 210 includes plural bitmap blocks 220A, 220B, . . . 220N. A second, top, or higher layer or branch 230 includes a bitmap block 240.

[0022] Each bitmap block includes an array of bits having a logical “1” or “0.” A logical “1” or “0” indicates whether the block of memory is available for storage or full. For example, a “1” indicates that a block of memory is available or free; and a “0” indicates that the block of memory is unavailable or used. Alternatively, a “1” indicates that a block of memory is unavailable or used; and a “0” indicates that the block of memory is available or free.

[0023] In the lower layer or branch 210, each bit in each block of the bitmap corresponds to one data storage block. For instance, bitmap block 220A includes a logical “1” in a first block 260 which indicates that the data storage block corresponding to block 260 is full. By contrast, a second block 262 includes a logical “0” which indicates that the data storage block corresponding to block 262 is empty or available for storage.

[0024] In the second layer or branch 230, each bit in each block of the bitmap corresponds to one of the bitmap blocks in the lower layer 210. A logical “1” indicates that the corresponding lower bitmap block is full; and a logical “0” indicates that the corresponding lower bitmap block has one or more empty data storage blocks. For instance, block 270 includes a logical “0” which indicates that bitmap block 220A has one or more free data storage blocks (i.e., the second, fourth, and fifth bits being “0” in bitmap block 220A). Block 272 includes a logical “1” which indicate that bitmap block 220B is full (i.e., all bits are marked with a “1” which indicates no available or free space in bitmap block 220B).

[0025] A hierarchical bitmap tree describes the allocated/free state for each block on the file system and is managed as a highly concurrent tree data structure. The leaf nodes (i.e., bitmap blocks) of the branches contain bitmap-style allocated/free state for the space-managed regions of the filesystem. Along with this detailed state in the tree leaves is summary state that includes a count of free blocks of each available size. This summary information is propagated up to the higher levels of the space manager tree data structure by having each internal node of the tree provide summary information for the level(s) of the tree below it. With this data structure in place, clients of the file system space manager can immediately obtain a summary view of what lies beneath any point of the tree and determine whether the desired item can be found anywhere in a branch of a tree. This eliminates I/O and associated cross-node lock contention that would have been required to process the lower levels of the tree. A lazy update scheme for higher level summary state is also used to avoid needing to update all parent summary state up to the root with each individual allocate or free operation.

[0026] FIG. 3 shows a hierarchical bitmap tree 300 in accordance with an exemplary embodiment of the present invention. The tree includes a top or root bitmap block (BMP) 310; a second layer with bitmap blocks 320A, 320B, . . . 320N; and a third layer with bitmap multiple bitmap blocks 340A . . . 340N, 350A . . . 350N, and 360A . . . 360N.

[0027] For convenience of illustration, only a few bitmap blocks and layers are shown in the hierarchical bitmap tree 300. The tree, however, can include many more bitmap blocks and layers.

[0028] With exemplary embodiments, available space is quickly discovered even in clustered or distributed filesystems having large amounts of shared memory. For instance, if bitmap block 320A includes all logical “1s” then the memory associated with bitmap blocks 340A to 340N is full. Thus, the filesystem would not spend processing time to search bitmap blocks 340A to 340N for available memory since a bit in bitmap block 310 would be set to “1” indicating no available memory.

[0029] In one embodiment, the size of the tree and/or bitmap blocks is dynamic and changes as the size of memory in the file system increases or decreases. For example, when blocks of memory (such as hard disks or other storage medium) are added to the file system, bits corresponding to these blocks of memory are added to a bitmap block in the tree. With larger expansions of memory, entire bitmap blocks or branches are added to the tree. As such, the tree grows (i.e., increases and decreases) in size as the amount of memory in the file system changes.

[0030] In one embodiment, updates to higher level branches or layers are prioritized over updates to lower level branches or layers. Looking to FIG. 3 for example, updates to bitmap blocks 320A, 320B, and 320C occur before and are prioritized over updates to bitmap blocks 340A . . . 340N, 350A . . . 350N, and 360A . . . 360N.

[0031] One exemplary embodiment is a method for free space management in a distributed or clustered file system that enables highly efficient allocation and tracking of available memory. This method allows cluster nodes to, in the common case, allocate and free space from the file system without actively negotiating with other nodes at the time of the allocation or free. As the filesystem contents nears capacity and the remaining free space becomes more highly contended, communication among the cluster nodes becomes required to negotiate the use of this remaining free space. But even in this case, this negotiation and corresponding cluster-wide lock contention is minimized using the allocation summary state information kept in the multi-level free space tracking metadata structures of the file system.

[0032] FIG. 4 shows a flow diagram for tracking and updating memory space using a hierarchical bitmap system in a file system in accordance with an exemplary embodiment of the present invention. For discussion, the filesystem is a cluster filesystem in accordance with one exemplary embodiment.

[0033] According to block 400, a first node in a cluster file system retrieves a second level bitmap block from memory. For example, a computer or server retrieves second layer bitmap blocks from a hard disk (such as retrieving block 230 in FIG. 2 of blocks 320A to 320C in FIG. 3).

[0034] According to block 410, the first node updates one or more of the second level bitmap blocks. This update includes changing one or more bits from a “1” to a “0” and/or from a “0” to a “1.” For example, the first node changes block 270 in FIG. 2 from a “0” to a “1” to indicate that the corresponding bitmap block 220A is full.

[0035] According to block 420, the first node sends the updated second level bitmap blocks back to memory.

[0036] According to block 430, a second node in a cluster file system retrieves the second level bitmap block from memory. For example, another computer or server retrieves second layer bitmap blocks from a hard disk (such as retrieving block 230 in FIG. 2 of blocks 320A to 320C in FIG. 3).

[0037] According to block 440, the second node updates one or more of the second level bitmap blocks. This update includes changing one or more bits from a “1” to a “0” and/or from a “0” to a “1.” For example, the second node changes block 272 in FIG. 2 from a “1” to a “0” to indicate that the corresponding bitmap block 220B is available.

[0038] According to block 450, the second node sends the updated second level bitmap blocks back to memory.

[0039] According to block 460, the first node in a cluster file system retrieves the first level bitmap block from memory. This first level reports to or forms part of the hierarchical tree for the previously retrieved second level. For example, the first computer or server retrieves a first layer bitmap block from a hard disk (such as retrieving block 220A in FIG. 2).

[0040] According to block 470, the first node updates first level bitmap blocks to correspond with previous update to second level bitmap blocks. The first computer or server updates the first layer bitmap block to indicate the change previously made to the second level bitmap block. For example, all bits in bitmap block 220A are changed to logical “1” to indicate that the corresponding data storage blocks are full. This change corresponds to the change previously made in bitmap block 230 (i.e., the change of bit 270 from a logical “0” to a logical “1”).

[0041] According to block 480, the first node sends the updated first level bitmap blocks back to memory.

[0042] With the method of FIG. 4, updates to the hierarchical bitmap tree or table occur less frequently and in less time. In the example provided above, during the first update, the first node only updated the second level bitmap blocks; the first level bitmap block was not updated. This reduced the overall number of updates required and hence reduced the time needed to update the bitmap tree. The second node retrieved the bitmap tree with a current or updated second level.

[0043] In one exemplary embodiment, the nodes in the cluster file system do not immediately update the entire bitmap tree. Instead, updates are provided to a higher level (such as roots or branches) and not to lower levels (such as leaves corresponding to the roots or branches). This partial update enables other nodes waiting for the bitmap tree to access the bitmap tree more quickly. For example, in one embodiment, only one node can update the second bitmap tree at one time (i.e., two different nodes cannot simultaneously update the bitmap blocks).

[0044] As used herein and in the claims, the following words are defined as follows:

[0045] A “bitmap” means a type of memory organization or image file format that stores digital images as a map of bits or an array of bits.

[0046] A “data storage block” or “storage block” means specific areas in memory, such as a hard disk. For example, one data storage block is a collection of eight sectors or 4,096 bytes, referred to as 4K bytes.

[0047] A “filesystem” or “file system” is a collection of file data, maintained by a filesystem implementation which is a set of data types, methods, and algorithms (typically implemented within an operating system instance) that store, organize, and maintain file data, frequently in some kind of file and/or directory hierarchy (albeit various alternatives and choices exist in the exact organizational structure made manifest by the filesystem implementation to the consumers of the file data). The actual file data and associated filesystem meta-data which describe the location, layout, directory organization, etc. of all file data within the filesystem is in turned stored on a data storage device (e.g., single hard disk, CD-ROM, disk storage array, network attached storage (NAS), etc.).

[0048] The term “storage device” means any data storage device capable of storing data including, but not limited to, one or more of a disk array, a disk drive, a tape drive, optical drive, a SCSI device, or a fiber channel device. Further, a “disk array” or “array” is a storage system that includes plural disk drives, a cache, and controller. Arrays include, but are not limited to, networked attached storage (NAS) arrays, modular SAN arrays, monolithic SAN arrays, utility SAN arrays, and storage virtualization.

[0049] In one exemplary embodiment, one or more blocks or steps discussed herein are automated. In other words, apparatus, systems, and methods occur automatically. The terms “automated” or “automatically” (and like variations thereof) mean controlled operation of an apparatus, system, and/or process using computers and/or mechanical/electrical devices without the necessity of human intervention, observation, effort and/or decision.

[0050] The methods in accordance with exemplary embodiments of the present invention are provided as examples and should not be construed to limit other embodiments within the scope of the invention. Further, methods or steps discussed within different figures can be added to or exchanged with methods of steps in other figures. Further yet, specific numerical data values (such as specific quantities, numbers, categories, etc.) or other specific information should be interpreted as illustrative for discussing exemplary embodiments. Such specific information is not provided to limit the invention.

[0051] In the various embodiments in accordance with the present invention, embodiments are implemented as a method, system, and/or apparatus. As one example, exemplary embodiments and steps associated therewith are implemented as one or more computer software programs to implement the methods described herein. For example, the software is implemented as one or more modules. The location of the software will differ for the various alternative embodiments. The software programming code, for example, is accessed by a processor or processors of the computer or server from long-term storage media of some type, such as a CD-ROM drive or hard drive. The software programming code is embodied or stored on any of a variety of known media for use with a data processing system or in any memory device such as semiconductor, magnetic and optical devices, including a disk, hard drive, CD-ROM, ROM, etc. The code is distributed on such media, or is distributed to users from the memory or storage of one computer system over a network of some type to other computer systems for use by users of such other systems. Alternatively, the programming code is embodied in the memory and accessed by the processor using the bus. The techniques and methods for embodying software programming code in memory, on physical media, and/or distributing software code via networks are well known and will not be further discussed herein.

[0052] The above discussion is meant to be illustrative of the principles and various embodiments of the present invention. Numerous variations and modifications will become apparent to those skilled in the art once the above disclosure is fully appreciated. It is intended that the following claims be interpreted to embrace all such variations and modifications.