US 8,443,134 B2Grant
Apparatus, system, and method for graceful cache device degradation
Issue Date:2013-05-14
•22 Claims
•8 Drawing Sheets
Abstract
Apparatuses, systems, and methods are disclosed for implementing a cache policy. A method may include determining a risk of data loss on a cache device. The cache device may comprise a non-volatile storage device configured to perform cache functions for a backing store. The cache device may implement a cache policy. A method may include determining that a risk of data loss on the cache devices exceeds a threshold risk level. A method may include implementing a modified cache policy for the cache device in response to the risk of data loss exceeding the threshold risk level. The modified cache policy may reduce the risk of data loss below the threshold level.
Metadata
Assignee
- Fusion-io, Inc.
Inventor
- David Flynn
Application Information
Application Number:US 12/885,285
Filing Date:2010-09-17
Priority Date:2006-12-06
Art Unit:2188
Classifications
IPC:
G06F12/00
Patent Drawings (8 sheets)
Description
Cross-References to Related Applications
[0001] This application is a continuation in part of, and claims priority to, the following U.S. patents, each of which is incorporated by reference:
[0002] U.S. patent application Ser. No. 11/952,123 entitled “Apparatus, System, and Method for Solid-State Storage as Cache for High-Capacity, Non-Volatile Storage” and filed Dec. 6, 2007 for David Flynn, et al., which patent application claims priority to U.S. provisional patent application No. 60/873,111 entitled “Elemental Blade System” filed on Dec. 6, 2006 for David Flynn et al., and U.S. provisional patent application No. 60/974,470 entitled “Apparatus, System, and Method for Object-Oriented Solid-State Storage” filed on Sep. 22, 2007 for David Flynn et al;
[0003] U.S. patent application Ser. No. 12/877,971 entitled “Apparatus, System, and Method for Caching Data on a Solid-State Storage Device” filed on Sep. 8, 2010 for David Flynn et al., which patent application claims priority to U.S. provisional patent application No. 61/240,573 entitled “Apparatus, System, and Method for an Improved Nonvolatile Caching Device” filed on Sep. 8, 2009 for David Flynn et al., and U.S. provisional patent application No. 61/240,966 entitled “Apparatus, System, and Method for Allocating Solid-State Storage” filed on Sep. 9, 2009, for Jonathan Thatcher et al., and U.S. provisional patent application No. 61/373,271 entitled “Apparatus, System, and Method for Caching Data” filed on Aug. 12, 2010 for David Flynn; and
[0004] U.S. patent application Ser. No. 12/847,952 entitled “Apparatus, System, and Method for Redundant Write Caching” filed on Jul. 30, 2010 for David Flynn et al., which patent application is a continuation-in-part of and claims priority to U.S. patent application Ser. No. 11/952,123 entitled “Apparatus, System, and Method for Solid-State Storage as Cache for High-Capacity, Non-Volatile Storage” and filed on Dec. 6, 2007 for David Flynn, et al., and U.S. provisional patent application No. 61/240,573 entitled “Apparatus, System, and Method for an Improved Nonvolatile Caching Device” and filed on Sep. 8, 2009 for David Flynn, et al. U.S. patent application Ser. No. 11/952,123 is a continuation in part of and claims priority to U.S. provisional patent application No. 60/873,111 filed on Dec. 6, 2006 and U.S. provisional patent application No. 60/974,470 filed on Sep. 22, 2007.
This application also claims priority to the following U.S. provisional patent applications, each of which is incorporated by reference:
[0005] U.S. provisional patent application No. 61/368,564 entitled “Apparatus, System, and Method for Writing Data to Storage Media in a Single Atomic Operation” filed on Jul. 28, 2010 for David Flynn et al.; and
[0006] U.S. provisional patent application No. 61/373,271 entitled “Apparatus, System, and Method for Caching Data” filed on Aug. 12, 2010 for David Flynn.
Background
[0007] 1. Field of the Invention
[0008] This invention relates to allocation of data storage and more particularly relates to allocation of data storage at a storage device.
[0009] 2. Description of the Related Art
[0010] Computers and storage systems (such as storage area networks (SANs)) often make use of cache to speed data writing and retrieval. Due to the cost associated with devices that have sufficient speed to act as effective cache, the amount of memory available as cache is often limited. Solid state storage (such as Flash) may be used to provide relatively affordable, high-speed and fast caching memory. Hard drives may be used as cache in front of slower media such as tape.
[0011] However, many types of storage media degrade with time. For example, Flash memory has a limited life cycle; the reliability of the Flash memory degrades as the read-write cycles increase. The device becomes increasingly unreliable as a storage device with use, and eventually dies completely. The limited life cycle of physical devices (whether solid state memory or hard disk) reduces its ability to act as effective cache. Since an effective storage system should be able to guarantee, with a high degree of reliability, that data written to the system will not be lost, using a cache device that degrades with time can cause difficulty. The possibility of losing data is an impediment to the acceptance and implementation of nonvolatile memory as cache. It can also increase costs as administrators need to replace cache devices (even if they may be functional) because the risk of data loss associated with continued use of the cache device is simply too high. Furthermore, attempting to write data to a cache device, and read data from a cache device, when that data is no longer trustworthy, unnecessarily increases traffic and overhead on a storage system.
Summary
[0012] Methods for caching using a nonvolatile storage device are presented. In one embodiment, a method includes determining a risk of data loss on a cache device. The cache device, in certain embodiments, is a non-volatile storage device configured to perform cache functions for a backing store. The cache device, in a further embodiment, implements a cache policy. A method, in one embodiment, includes determining that the risk of data loss on the cache device exceeds a threshold risk level. In a further embodiment, a method includes implementing a modified cache policy for the cache device in response to the risk of data loss exceeding the threshold risk level. The modified cache policy, in certain embodiments, reduces the risk of data loss below the threshold risk level.
[0013] Apparatuses are presented for caching using a nonvolatile storage device. In one embodiment, a direct cache in configured for communication with a cache device and a backing store. A cache device, in certain embodiments, is a non-volatile storage device configured to perform cache functions for a backing store. A direct cache, in one embodiment, includes a monitor module configured for determining a risk of data loss on the cache device. In a further embodiment, a direct cache includes a cache policy module configured for implementing a modified cache policy for the cache device if a risk of data loss on the cache device exceeds a threshold risk level. A direct cache, in certain embodiments, is configured to receive, from the cache device, one or more indicators associated with the risk of data loss on the cache device.
[0014] Systems are presented for caching using a nonvolatile storage device. In one embodiment, a backing store is configured for storing data received from one or more storage clients. A backing store, in certain embodiments, comprises one or more non-volatile storage devices. A cache device, in one embodiment, is configured for performing cache functions for the backing store. A cache device, in a further embodiment, is a non-volatile storage device with faster access characteristics than those of the backing store. In another embodiment, a cache device is configured to implement a cache policy. A direct cache, in one embodiment, is configured for determining a risk of data loss on a cache device. A direct cache, in a further embodiment, is configured for determining that the risk of data loss on a cache device exceeds a threshold risk level. In another embodiment, a direct cache is configured for implementing a modified cache policy for a cache device in response to the risk of data loss exceeding the threshold risk level. The modified cache policy, in certain embodiments, reduces the risk of data loss below the threshold risk level.
[0015] Reference throughout this specification to features, advantages, or similar language does not imply that all of the features and advantages that may be realized with the present invention should be or are in any single embodiment of the invention. Rather, language referring to the features and advantages is understood to mean that a specific feature, advantage, or characteristic described in connection with an embodiment is included in at least one embodiment of the present invention. Thus, discussion of the features and advantages, and similar language, throughout this specification may, but do not necessarily, refer to the same embodiment.
[0016] Furthermore, the described features, advantages, and characteristics of the invention may be combined in any suitable manner in one or more embodiments. One skilled in the relevant art will recognize that the invention may be practiced without one or more of the specific features or advantages of a particular embodiment. In other instances, additional features and advantages may be recognized in certain embodiments that may not be present in all embodiments of the invention.
[0017] These features and advantages of the present invention will become more fully apparent from the following description and appended claims, or may be learned by the practice of the invention as set forth hereinafter.
Brief Description of the Drawings
[0018] In order that the advantages of the invention will be readily understood, a more particular description of the invention briefly described above will be rendered by reference to specific embodiments that are illustrated in the appended drawings. Understanding that these drawings depict only typical embodiments of the invention and are not therefore to be considered to be limiting of its scope, the invention will be described and explained with additional specificity and detail through the use of the accompanying drawings, in which:
[0019] FIG. 1 is a schematic block diagram illustrating one embodiment of a system with graceful cache degradation;
[0020] FIG. 2 is a second schematic block diagram illustrating one embodiment of a system with graceful cache degradation;
[0021] FIG. 3 is a schematic block diagram illustrating one embodiment of a direct cache for realizing graceful cache degradation;
[0022] FIG. 4 is an illustration of two exemplary tables specifying possible cache policies and associated cache device states;
[0023] FIG. 5 is a schematic block diagram illustrating one embodiment of a software stack in a system with graceful cache degradation;
[0024] FIG. 6 is an illustrative flow chart diagram showing one embodiment of a method for realizing graceful cache degradation; and
[0025] FIG. 7 is an illustrative block diagram of one embodiment of a mapping module and remapping module in a storage device.
Detailed Description
[0026] Many of the functional units described in this specification have been labeled as modules, in order to more particularly emphasize their implementation independence. For example, a module may be implemented as a hardware circuit comprising custom VLSI circuits or gate arrays, off-the-shelf semiconductors such as logic chips, transistors, or other discrete components. A module may also be implemented in programmable hardware devices such as field programmable gate arrays, programmable array logic, programmable logic devices or the like.
[0027] Modules may also be implemented in software for execution by various types of processors. An identified module of executable code may, for instance, comprise one or more physical or logical blocks of computer instructions which may, for instance, be organized as an object, procedure, or function. Nevertheless, the executables of an identified module need not be physically located together, but may comprise disparate instructions stored in different locations which, when joined logically together, comprise the module and achieve the stated purpose for the module.
[0028] Indeed, a module of executable code may be a single instruction, or many instructions, and may even be distributed over several different code segments, among different programs, and across several memory devices. Similarly, operational data may be identified and illustrated herein within modules, and may be embodied in any suitable form and organized within any suitable type of data structure. The operational data may be collected as a single data set, or may be distributed over different locations including over different storage devices, and may exist, at least partially, merely as electronic signals on a system or network. Where a module or portions of a module are implemented in software, the software portions are stored on one or more computer readable media.
[0029] Reference throughout this specification to “one embodiment,” “an embodiment,” or similar language means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, appearances of the phrases “in one embodiment,” “in an embodiment,” and similar language throughout this specification may, but do not necessarily, all refer to the same embodiment.
[0030] Reference to a computer readable medium may take any form capable of storing machine-readable instructions on a digital processing apparatus. A computer readable medium may be embodied by a transmission line, a compact disk, digital-video disk, a magnetic tape, a Bernoulli drive, a magnetic disk, a punch card, flash memory, integrated circuits, or other digital processing apparatus memory device. The word “may” is used in this application to express possibility, and should not be interpreted as expressing opportunity, permission, or contingency.
[0031] Furthermore, the described features, structures, or characteristics of the invention may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided, such as examples of programming, software modules, user selections, network transactions, database queries, database structures, hardware modules, hardware circuits, hardware chips, etc., to provide a thorough understanding of embodiments of the invention. One skilled in the relevant art will recognize, however, that the invention may be practiced without one or more of the specific details, or with other methods, components, materials, and so forth. In other instances, well-known structures, materials, or operations are not shown or described in detail to avoid obscuring aspects of the invention.
[0032] The schematic flow chart diagrams included herein are generally set forth as logical flow chart diagrams. As such, the depicted order and labeled steps are indicative of one embodiment of the presented method. Other steps and methods may be conceived that are equivalent in function, logic, or effect to one or more steps, or portions thereof, of the illustrated method. Additionally, the format and symbols employed are provided to explain the logical steps of the method and are understood not to limit the scope of the method. Although various arrow types and line types may be employed in the flow chart diagrams, they are understood not to limit the scope of the corresponding method. Indeed, some arrows or other connectors may be used to indicate only the logical flow of the method. For instance, an arrow may indicate a waiting or monitoring period of unspecified duration between enumerated steps of the depicted method. Additionally, the order in which a particular method occurs may or may not strictly adhere to the order of the corresponding steps shown.
[0033] FIG. 1 illustrates one embodiment of a system 100 which may benefit from graceful cache degradation. In the depicted embodiment, the system 100 includes clients 110, a computer network 112, a server 108, a direct cache 120, a cache device 102, and a backing store 130. Other configurations of a system 100 are also possible and may similarly benefit from graceful cache degradation. Exemplary system 100 include, but are not limited to, a storage area network (SAN), network attached storage (NAS), a desktop computer, a laptop computer, or other computing environment which may beneficially implement a cache.
[0034] The clients 110 (also referred to as storage clients) are broadly defined as devices that send and receive requests for data. The clients may be computers (such as laptops, desktops, servers, mainframes, etc), mobile devices, or other categories of devices that send and receive digital data. The clients 110 may be directly connected to storage for the digital data (for example, by a bus) or may connected over a network (for example, the Internet).
[0035] The computer network 112 is a series of computers and devices connected by communications channels that facilitate communications between users of the attached devices. The network 112 may allow the users to share resources, such as storage. The network 112 may include, for example, network interface cards (NICs), repeaters, hubs, bridges, switches, routers, etc. The computer network 112 may be any of a large area network (LAN), wide area network (WAN), or other variety of computer network. The network 112 may be, for example, implemented in a data center, over the Internet, or in another environment. As noted above, in certain embodiments, the system 100 does not include a computer network 112.
[0036] The server 108 may be a workstation, a mainframe, a laptop, or other computing device. In certain embodiments, the server 108 has access to, and is communication with, the cache device 102 and the backing store 130. In certain embodiments, the server 108 manages communications between the clients 110 and the cache device 102 and the backing store 130. For example, the server 108 may be a storage manager. The server 108 may be directly connected to the cache device 102 (for example, via a bus), or indirectly connector (for example, via a network).
[0037] The server 108 may implement a file server to facilitate accessing and writing data to the cache device 102 and the backing store 130. The file server may manage files for the client 110. For example, clients 110 may direct the file server to read, write, delete, modify, or perform another operation for a particular file, object, logical block, or other data unit. The server 108 may view the cache device 102 and the backing store 130 as separate devices. The server 108 may also be presented with a view of unitary storage (such as a logical address space) with the existence and functioning of the cache device 102 hidden from the server 108.
[0038] The cache device 102 is a nonvolatile storage device that can store data digitally. The cache device 102 acts as a cache for the backing store 130, which is described below. In certain embodiments, the cache device 102 also includes a driver; in certain embodiments, the direct cache 120 may communicate directly with the cache device 102 without using an intermediate device driver for the cache device 102. For example, the direct cache 120 may interact with a VolumeManager (VM) either by polling or notification in order to determine when a backing store 130 is added to the system 100. In certain embodiments, the driver gets authorization to activate a direct cache 120 feature. The cache device 102 typically has faster access characteristics than those of the backing store 130; thus data can generally be read from the cache device 102 faster than data can be read from the backing store 130. The increased speed of the cache device 102 may result from the use of a cache device 102 that uses technology that is generally faster than that of the backing store (e.g., Flash memory cache device and hard disk drive backing store). The increase speed may also result from the nature of the connection; for example, the cache device 102 may be connected using a bus of the server 108 while the backing store 130 is connected using a slower network connection.
[0039] The cache device 102 may be realized as a card that connects to the PCIe bus of the server 108. In one embodiment, the cache device 102 is a solid state storage device implemented using NAND Flash memory. For example, the cache device 102 may be a solid state storage device such as those manufactured by Fusion-io®. In certain embodiments, the cache device 102 may comprise one or more physical devices that cooperate such that they serve as a single logical cache device; for example, multiple memory cards that attach to the PCIe bus of the server 108 may be used as the cache device 102. In such embodiments, the direct cache 120 may be configured to manage multiple physical devices that make up the cache device 102.
[0040] The backing store 130 provides storage for data in the system 100. The backing store 130 may comprise one or more devices. For example, the backing store 130 may be made up of hard disk drives, solid state storage, tape, or other non-volatile media for storing data. The backing store 130 may be a SAN, a NAS, a RAID array, an ISCSI client, a JBOD, a volume or partition, or other storage device. In certain embodiments, the backing store 130 is a block device and interacts with the direct cache 120 by way of a block device interface.
[0041] The direct cache 120 enables graceful degradation of the cache device 102. The direct cache 120 may be implemented as a driver on the server 108, as software, as firmware, or as hardware. The caching device 120 may be implemented on the cache device 102. The caching device 120 may be a filter driver that interacts with the file system and/or operating system of a server 108, or other storage client 110 such as a database management system (DBMS). Components of the caching device 120 may also be distributed in the system 100. In certain embodiments, the direct cache 120 dynamically changes the cache policy implemented on the cache device 102 as the cache device experience wear out and failure. The cache policies may be implemented to provide the best data protection and cache performance possible in view of the health of the cache device 102.
[0042] The direct cache 120 may manage the use of the cache device 102 as a cache for data stored in the backing store 130. The direct cache 120 may implement any number of caching policies to make advantageous use of the faster access characteristics of the cache device 102. A caching policy specifies how data is written into, and read out of, the cache device 102 and the backing store 130. The cache policy may, for example, specify how the cache device 102 responds to read requests and/or write requests. The direct cache 120 may present a block device interface to storage clients such as the server 108. The server 108 may, in certain embodiments, interact with the direct cache 120 as it would with a standard block device. In certain embodiments, the direct cache 120 advertises to a storage client such as the server 108 an amount of storage capacity that is equal to the capacity of the backing store 130. In other embodiments, the direct cache 120 may advertise a storage capacity that is equal to the combined storage capacity of the cache device 102 and the backing store 130. In other embodiments, the direct cache 120 may present a sparse address space. The cache device 102 and backing store 130 may be transparent to the server 108, which may only see and interact with the direct cache 120.
[0043] The direct cache 120 may be configured to communicate with both the cache device 102 and the backing store 130. In certain embodiments, the direct cache 120 communicates commands like TRIM to the cache device 102. Commands may be sent in parallel to the cache device 102 and the backing store 130 when the commands are relevant to both. For example, where the cache device 102 and the backing store 130 are both solid state storage devices, the direct cache 120 may send TRIM commands to both the backing store 130 and the cache device 102.
[0044] In certain embodiments, the direct cache 120 determines the risk of data loss on the cache device 102, and determines that the risk of data loss exceeds a threshold risk level. The direct cache 120 may implement a modified cache policy for the cache device 102 if the direct cache 120 determines that the risk of data loss exceeds the threshold risk level. A modified cache policy chosen by the direct cache 120 is one that reduces the risk of data loss to below the threshold risk level. The direct cache 120 may dynamically determine the risk of data loss and dynamically adjust the cache policy of the cache device 102.
[0045] The direct cache 120 may determine the risk of data loss on the cache device 102 by monitoring the cache device 102 directly. The direct cache 120 may also determine the risk of data loss based on information given to it by the cache device 102; in certain embodiments, the cache device 102 reports the risk of data loss to the direct cache 120.
[0046] In certain embodiments, as the physical health of the cache device 102 decreases, the reliability of the cache device 102 also decreases (thus increasing the risk of data loss). For example, as Flash memory can grow increasingly unreliable and wear out completely as it is used over time. The direct cache 120 may gauge the physical health of the cache device 102 in determining the risk of data loss on the cache device 102. In certain embodiments, the cache device 102 reports metrics on physical health to the direct cache 120. In other embodiments, the direct cache 120 determines the heath of the cache device 102 directly. For example, the direct cache 120 may track the number of read and write operations executed by the cache device 102; the direct cache 120 may track the frequency of errors in data read from the cache device 102 corrected using error correction code (ECC); the direct cache 120 may track the frequency of errors in data read from the cache device 102 which cannot be corrected. Other approaches to monitoring and determining the physical health of a cache device 102 may also be used.
[0047] The direct cache 120 may dynamically adjust the caching policies for the system 100 in response to the changing risk of data loss on the cache device 102. For example, the direct cache 120 may dynamically adjust the cache policy as the cache device 102 ranges from a cache device state of fully operational to non-operational. As used herein, the term “non-operational” means that the cache device 102 is no longer capable of performing its primary function of caching data for the backing store 130. In addition, in certain embodiments, a non-operational cache device 102 may mean that the cache device 102 can no longer store data in a persistent manner. In certain embodiments, a non-operational cache device 102 may mean that the cache device 102 can no longer write new data in a manner that guarantees a threshold level of data reliability. In certain embodiments, a non-operational cache device 102 may mean that the cache device 102 has retired a sufficient number of storage cells of the non-volatile storage media 106 that there is no more storage capacity for new data (either user read misses or user writes). In other embodiments, a non-operational cache device 102 may mean that the cache device 102 is not communicating properly with a cache device driver 118, has lost power, has otherwise failed, or the like. The direct cache 120 may, for example, send all read and write requests to the backing store 130 and bypass the cache device 102 when the cache device 102 is non-operational. In certain embodiments, depending on the nature of the cache device 102 being non-operational, bypassing the cache device 102 may comprise sending read and write requests to the cache device 102 which then simply passes the requests on to the backing store 130. (e.g. the cache device 102 acts as a pass through component) These changes in cache policy may be hidden from the rest of the system 100 such that the server 108 and the clients 110 are unaware of the changes in the cache policy and the availability of the cache device 102. In certain embodiments, the cache device 102 is transparent to the server 108 and the clients 110.
[0048] As noted above, the direct cache 120 may determine that the risk of data loss on the cache device 102 exceeds a threshold risk level. The threshold risk level defines the maximum risk of data loss that is acceptable for the system 100. The way in which the threshold risk level may be implemented may vary. In one embodiment, the threshold risk level is a defined possibility of data loss; for example, the direct cache 120 may not tolerate a 2% risk of data loss. The direct cache 120 may implement various algorithms to determine a current risk of data loss for the direct cache 120, which can be compared against the threshold risk of data loss.
[0049] In other embodiments, the risk of data loss may be derived from a cache device state for the cache device 102. The direct cache 120 may define which caching policies are acceptable for a particular cache device state of the cache device 102. Examples of cache device states are given in FIG. 4 . The direct cache 120 may determine that the risk of data loss exceeds a threshold risk level by determining that there is a change in the cache device state, and that the cache policy currently implemented on the cache device 102 is not acceptable for that particular cache device state. Such an approach is discussed in greater detail below. Other approaches may also be used to determine that the risk of data loss on the cache device 102 exceeds a threshold risk level.
[0050] The threshold risk level may be user defined, pre-defined by a manufacturer of the cache device 102, defined by a creator of the direct cache 120, or some combination thereof. For example, a user may define which caching policies are appropriate for which cache device states. The creator of the direct cache 120 may set defaults and recommend certain settings to the user.
[0051] The direct cache 120 may thereby allow the cache device 102 to gracefully degrade, adjusting the role of the cache device 102 as the cache device 102 ages and becomes increasingly unreliable. Thus, the direct cache 120 may adjust the role of the cache device 102 in the system 100 as it goes from a fully functional cache device 102 with no risk of data loss to non-operational. The role of the cache device 102 may be adjusted dynamically in response to the changes in the cache device state, and the changing role of the cache device 102 (as defined by the cache policies) may be hidden from the server 108 and the clients 110. Thus, the cache device 102 may initially take a vigorous role in the system 100 and, eventually, be bypassed completely when the cache device 102 becomes non-operational.
[0052] FIG. 2 presents a second embodiment of a system 200 which may benefit from graceful cache degradation. The system 200 includes a host 208, a file server 214, a direct cache 120, a backing store driver 222, a cache device 102, a caching interface 230, a cache device driver 118, a storage controller 104, and storage media 106. The system 200, in certain configurations, may include different elements than those shown in FIG. 2 .
[0053] The host 208 may be a computer (such as a laptop, blade server, desktop, etc). In the depicted embodiment, both the cache device 102 and the backing store 130 may be directly attached to a bus of the host 208. The cache device 102 may be used as a cache for the backing store 130.
[0054] The file server 214 may facilitate writing and reading data in the system 200. The file server 214 may be a file system responsible for organizing blocks and maintaining an address system. For example, the file server 214 may associate file names with blocks of storage. The file server 214 may use a file allocation table (FAT) filing system, hierarchical file system (HFS) Plus, or other file management approach. The operating system on the host 208 may provide the file server 214. The file server 214 may communicate with the direct cache 120 using a storage interface.
[0055] The backing store driver 222 is a driver for the backing store 130. The direct cache 120 may invoke routines of the backing store driver 222 which, in response, may issue commands to the backing store 130. The backing store 130, in response, may send data back to the backing store driver 222, which in turn may send data to the direct cache 120 or otherwise invoke a routine in the direct cache 120. For example, the direct cache 120 may receive a write request from the file server 214, and determine that the data should be written to the backing store 130. The direct cache 120 may invoke routines of the backing store driver 222 causing it to issue commands to the backing store which result in the data being written on the physical storage media of the backing store 130. The backing store 130 may provide the backing store driver 222 with an acknowledgement (commonly referred to as an ACK) indicating that the data was successfully written. The backing store driver 222 may send this ACK to the direct cache 120. A similar process may be used for reads.
[0056] The direct cache 120 may similarly interface with a cache device driver 118 that is a driver for the cache device 102. The cache device driver 118 may be implemented as a driver on the host 208. In certain embodiments, the direct cache 120 may include the cache device driver 118. The direct cache 120 may also be configured to communicate directly with the storage controller 104 without an intermediate driver. The cache device driver 118 may be shared among multiple storage devices with individual storage controllers 104.
[0057] The storage controller 104 enables the storage and retrieval of data on the storage media 106. The storage controller 104 may perform various low-level operations necessary to write and read data from the storage media 106. The storage controller 104 may map logical identifiers to physical addresses on the storage media 106. The storage controller 104 may be a field programmable gate array (FPGA) such as a Xilinx® FPGA. The storage controller 104 may be an application specific integrated circuit (ASIC) or other customer logic solution. Various approaches may be used to create and configure a storage controller 104.
[0058] The storage media 106 is the physical media on which the data is stored. The storage media 106 may be disks, solid state media (such as NAND Flash), or other suitable storage media. In one embodiment, the storage media 106 is NAND Flash and is made up of banks of solid-state storage elements. The banks may be arranged in parallel. The cache device 102 may be configured to sequence execution of solid-state storage commands (such as reads and writes) such that a command of a first execution duration executes on a first bank concurrent with execution of a command of a second execution duration on a second bank. More detail on the possible arrangements of banks of solid-state storage can be found in U.S. patent application Ser. No. 11/952,095, entitled “Apparatus, System, and Method for Managing Commands of Solid-State Storage Using Bank Interleave” filed on Dec. 6, 2007 for David Flynn, Bert Lagerstedt, John Strasser, Jonathan Thatcher, and Michael Zappe, which application is hereby incorporated by reference.
[0059] The cache device 102 may also support error correcting code (ECC) chunks to protect data stored in the storage media 106. In certain embodiments, the cache device 102 generates ECC chunks comprising at least a portion of the received data, and an ECC code that is capable of correcting an error in the ECC chunk. The cache device 102 may store a portion of the ECC chunk on one solid-state storage element and also store a portion of the ECC chunk on a second solid-state storage element.
[0060] As noted above, the cache device 102 may interact with the direct cache 120 by way of a block device interface. The direct cache 120 and the cache device 102 may also interact using a caching interface 230. In certain embodiments, the caching interface 230 provides an interface by which the direct cache 120 may communicate with the cache device driver 118. In other embodiments, the direct cache 120 communicates with the storage controller 104 directly using the caching interface 230.
[0061] The caching interface 230 may provide support for various functions in support of graceful cache degradation. The cache interface 230 may provide the necessary support for reporting the risk of data loss on the cache device 102. For example, the cache interface 230 may allow the cache device 102 to report degrading cache device state.
[0062] The caching interface 230 may support reporting the actual physical capacity of the cache device 102. The reported actual physical capacity may exclude reserved amounts of capacity, such as the memory space necessary for reserve space and maintenance. Such a function may be used when the cache device 120 and cache device 102 support sparse addressing, and where the advertised storage is very large.
[0063] The caching interface 230 may also support reporting indicators that are associated with the risk of data loss on the cache device 102 and which can be used to determine the health of the cache device 102. The caching interface 230 may, for example, support reporting indicators such as the frequency of errors correctable using ECC, the frequency of errors uncorrectable using ECC, the number of read and write operations executed, and other indicators that are relevant to the health of the cache device 102. The cache interface 230 may further support the cache device 102 asynchronously reporting to the direct cache 120 when the cache device 102 transitions from one cache device state to another cache device state. In one embodiment, this is accomplished using a callback function. Other indicators may also be shared that are also associated with the risk of data loss on the cache device 102.
[0064] The caching interface 230 may also support advanced eviction features. In certain embodiments, the caching interface 230 allows the direct cache 120 to request that a cache member in the cache device 102 be evicted. The caching interface 230 may allow the cache device 102 to provide recommendations of which cache members to evict. In certain embodiments, the direct cache 120 simply tells the cache device 102 to evict cache members and allows the cache device 102 to determine which cache members to evict. The cache device 102 may determine which cache members are candidates for eviction based on the temperature of the cache member. The cache device 102 may also use efficiency related factors to determine cache members that are candidates for eviction. For example, where the cache device 102 is a solid state storage device, the cache device 102 may determine that cache members are candidates for eviction in order to make garbage collection more efficient. The cache device 102 may also make eviction decisions so as to maintain the highest possible write performance in the solid state storage device.
[0065] FIG. 3 shows one embodiment of a direct cache 120 comprising a monitor module 310, a data loss module 320, and a cache policy module 322. In one embodiment, the monitor module 310 determines a risk of data loss on the cache device 102. Determining the risk of data loss may comprise gauging the physical health of the cache device 102. As described above, the cache device 102 may share information with the direct cache 120 that is relevant to the health of the cache device 102.
[0066] The monitor module 310 may also determine the cache device state of the cache device 102. For example, the cache device state may be “healthy”, or may be “failure.” In certain embodiments, determining the cache device state comprises the cache device 102 reporting the cache device state to the direct cache 120. The monitor module 310 may determine the risk of data loss in response to the cache device state. For example, if the cache device state is healthy, the monitor module 310 may determine that the risk of data loss is low.
[0067] The data loss module 320 may determine whether the risk of data loss on the cache device 102 exceeds a threshold risk level. As noted above, the data loss module 320 may do so by determining that the cache policy currently implemented on the cache device 102 is deemed too risky for the cache device state. Other approaches to determining whether the risk of data loss exceeds a threshold risk level may also be used.
[0068] The cache policy module 322 may implement a modified cache policy for the cache device 102 if the risk of data loss exceeds the threshold risk level. The cache policy module 322 may implement the modified cache policy dynamically. The change in cache policy may also be made transparently to the host 208 and the clients 110.
[0069] In certain embodiments, the multiple cache policies may reduce the risk of data loss below the threshold risk level. In such circumstances, the cache policy module 322 may select the modified cache policy that provides the highest level of cache functionality. Such a cache policy provides the best performance for the current risk of data loss associated with the cache device 102.
[0070] In certain embodiments, the cache policy module 322 contains the logic to implement the cache policy. The cache policy module 322 may, for example, be responsible for maintaining the data structures necessary to support the cache device 102. In one embodiment, the cache policy module 322 tracks clean and dirty data in the cache device 102. The cache policy module 322 may report dirty data to the cache device 102 such that the dirty data can be deleted. The cache policy module 322 may also report dirty data to the cache device 102 such that the dirty data can be marked as invalid (and therefore candidate for garbage collection) in cache devices 102 that are solid state storage devices. The validity map and the clean/dirty may be maintained in the direct cache 120, or may be maintained in the cache device 102, or a combination thereof.
[0071] The cache policy module 322 may also set the size of the cache members (also referred to in the art as cache lines). In certain embodiments, the user may adjust the size of the cache members. The size of the cache members may also be set by default by a maker of the direct cache 120. The size of the cache member may be a tunable attribute. In one embodiment, the size of the cache member may be set to be the same as the size of an erase unit size (such as the size of a logical erase block) such that entire erase blocks are marked invalid when cache members are evicted. Such an embodiment may increase the efficiency of the cache device 102 and optimize the cache device 102 for grooming. In other embodiments, the size of the cache member may be made much larger to minimize cache misses. The cache policy module 322 may also be configured to take efficiency of internal operations (such as grooming) into consideration when making decisions on which cache members to evict.
[0072] FIG. 4 includes two tables 410 and 420. The table 410 shows possible cache policies and the write and read behavior associated with the cache policies. The cache policies may include additional actions and detail than that shown in FIG. 4 . In one embodiment, the cache policies include: a write back cache policy; a write through cache policy; a write around cache policy; a read only cache policy; and a bypass cache policy. The cache policies may further specify that the acknowledgement for a data write operation is sent when the data is preserved on a sufficiently stable storage device. In certain embodiments, writing the data to the cache device 102 will be sufficiently safe to merit sending an acknowledgement once the data is written to the cache device 102. In other embodiments, the acknowledgement is not sent until the data is written to the backing store 130.
[0073] In certain embodiments, the cache policy is a write back cache policy (also referred to as a copy back cache). Under such a policy, the direct cache 120 may direct write requests to the cache device 102 such that the data is written to the cache device 102. The data may be later migrated to the backing store 130. For example, the data may be moved to the backing store 130 when the cache member is evicted from the cache device 102 (an approach frequently called lazy write back). In such embodiments, the direct cache 120 may also track whether data written to the cache device 102 is clean or dirty to facilitate migrating data from the cache device 102 to the backing store 130. Other approaches to moving data from the cache device 102 to the backing store 130 (such as opportunistic write back) may also be used. For example, where the cache device 102 is a solid state storage device (such as a Flash device), evictions may be triggered as part of a grooming operation. Such a cache device 102 may be allowed to store data in circumstances that would not be advisable for a volatile storage device since the risk of data loss is very low for a Flash memory device in good condition. In the event that the cache device state changes, the direct cache 120 may cause all clean data to be migrated to the backing store 130; for example, the direct cache 120 may implement a modified cache policy that is a write through cache policy and, in preparation for the implementation of the modified cache policy, migrate all clean data that was stored under a write back cache policy.
[0074] In certain embodiments, under a write back cache policy, the acknowledgement is sent once the data is written to the cache device 102. The cache device 120 may be configured to provide an additional notification when the data is moved from the cache device 102 to the backing store 130. Reads may be executed against the cache device 102 such that, when the cache device 102 has the requested data, the read is performed from the cache device 102 and not the backing store 130.
[0075] The cache policy may also include a write through cache policy. Under a write through cache policy, data may be synchronously written to the cache device 102 and the backing store 130. In one embodiment, the cache device 120 ensures that the write request is executed against both the cache device 102 and the backing store 130 in a manner that is transparent to the host 208 or clients 110. In certain embodiments, the acknowledgement is sent when the data is written on the backing store 130, or when the data is written on both the backing store 130 and the cache device 102. Thus, in certain embodiments, the cache policy module 322, when implementing a write through cache policy, may send write requests to the cache device 102 and to the backing store 130, with acknowledgments being sent when the backing store 130 fulfills the request. The read policy associated with the write through cache policy may be identical to that described above for the write back cache policy.
[0076] The cache policy may also include a write around cache policy. In certain embodiments, the write around cache policy directs writes only to the backing store 130. Acknowledgements may be sent when the data is stored in the backing store 130. In one embodiment, the data coming from the clients 110 is not written to the cache device 102, but data may still be written to the cache device 102. For example, the cache device 120 may inject cache members into the cache device 102 from the backing store 130 even if data coming from the clients 110 is not written to the cache device 102. Thus, the cache policy module 322 may, when implementing a write around cache policy, send write requests exclusively to the backing store 130 such that the write requests bypass the cache device 102, with acknowledgements sent when the backing store 130 fulfills the write request. Reads may be executed against the cache device 102 as described above under the write around cache policy.
[0077] The cache policy may also include a read only cache policy. Under the read only cache policy, writes are directly only to the backing store 130. Acknowledgements may be sent when the data is stored in the backing store 130. Under the read only cache policy, the direct cache 120 may not inject cache members into the cache device 120. Reads may be executed against the cache device 102 as described above.
[0078] The cache policy may also include a bypass cache policy. Under the bypass cache policy, writes are directly only to the backing store 130. Acknowledgements may be sent when the data is stored in the backing store 130. Reads may be directed to the backing store 130 instead of the cache device 120. The cache policy module 322 may implement the bypass cache policy for the cache device 102 by sending write requests exclusively to the backing store 130, and by sending read requests exclusively to the backing store 130.
[0079] Table 420 displays one embodiment of a relationship between cache device state and cache functionality. Noted above, one way to determine that the risk of data loss on the cache device 102 exceeds a threshold risk level is to associate cache device states with permissible cache policies. Table 420 is one example of a possible association of cache device states and cache policies.
[0080] In one embodiment, the cache device state may be a “healthy” state. What constitutes a healthy state may vary based on the parameters set forth by a device manufacturer. In one embodiment, a healthy device may be defined by the likelihood of error. In some embodiments, a healthy device is defined by the number of spare chips or dies available to it for use when blocks go bad and need to be replaced. Other parameters may be used to specify when a cache device 102 is considered to be in a healthy cache device state.
[0081] In one embodiment, when the cache device state is healthy, the direct cache 120 may implement any of the possible cache policies. In certain embodiments, the default cache policy implemented for the cache device 102 in a healthy cache device state is the cache policy which maximizes performance. The choice of which cache policy to implement may be a user-tunable attribute. In one embodiment, the direct cache 120 implements a write back cache policy tuned to maximize input/output operations when the cache device 102 is in a healthy cache device state. As noted, under the write back policy, reads may be performed against the cache device 102 if the cache device 102 has the requested cache member.
[0082] In one embodiment, the cache device 102 may eventually degrade to a lesser cache device state. For example, a cache device 102 that is a Fusion-io NAND Flash memory device may degrade to a flashback mode where one or more spare chips or dies are in use on the cache device 102. In certain embodiments, the cache device 102 in the Flashback mode remains sufficiently reliable to implement any of the five cache policies shown in the table 410. The user may choose a conservative approach and implement a cache policy such as the write through cache policy to be implemented when the cache device 102 transitions to Flashback mode.
[0083] In certain embodiments, the cache device 102 reports when there is a change in the cache device state. The cache device 102 may communicate the change in the cache device state using the caching interface 230. In certain embodiments, the direct cache 120 may poll for changes in the cache device state. The cache device 102 may report a change in cache device state using an asynchronous notification.
[0084] In one embodiment, the cache device state also includes a write reduced mode. A cache device 102 may be in write reduced mode when the cache device 102 exhausts the available logical erase blocks (LEBs), which may result in slow writes. The cache policy module 322 may be configured to implement a modified cache policy in response to this change in cache device state; for example, the cache policy module 322 may implement the write through cache policy when the cache device 102 transitions to write reduced mode. The cache policy may alternatively be configured to implement the write around cache policy in response to the change.
[0085] In certain embodiments, multiple cache policies would reduce the risk of data loss below the threshold risk level. For example, in write reduced mode, the write through cache policy may sufficiently reduce the risk of data loss, as would the bypass cache policy. In certain embodiments, when multiple cache policies would sufficiently reduce the risk of data loss, the cache policy module 322 selects the cache policy that provides the highest level of cache functionality. Thus, while the bypass cache policy reduces the risk of data loss on the cache device 102 to zero (by logically removing the cache device 102), the cache policy module 322 would select the write around cache policy for implementation as the write around cache policy provides higher levels of cache functionality.
[0086] The cache device 102 may also have an associated read only mode. The read only mode may occur when there are sufficiently large numbers of errors in writing and/or reading data that writing data to the cache device 102 is not recommended. The maker of the direct cache 120 and/or cache device 102 may determine how frequent and severe errors must be before moving the cache device 102 into the read only mode cache device state. And, as above, the parameters for determining when the cache device 102 is in the read only mode may be configurable by a user. In one embodiment, the write around cache policy is associated with the read only mode cache device state. The cache policy module 322 may implement the write around cache policy for the cache device 102 to prevent data loss.
[0087] As noted above, the write around cache policy may allow for data to be injected as cache members into the cache device 102 from the backing store 130. Because such data is, by definition, stored on the backing store 130, even if the cache member is corrupted on the cache device 102 due to the degraded cache device state, there is no permanent loss of data. In the event that the injected cache member is corrupted, there may be a performance penalty in that the read operation on the cache device 102 fails, and the data ultimately has to be retrieved from the backing store 130.
[0088] In certain embodiments, the cache device 102 continues to degrade to such an extent that even injecting cache members into the cache device 102 is not advisable. In response, the cache policy module 322 may implement the read only cache policy for the cache device 102. In certain embodiments, the cache policy module 322 implements the read only cache policy when the cache device 102 is in the read only mode cache device state. The cache policy module 322 may monitor the error rates for injected cache members when the cache device 102 is in the read only mode and, once the error rates become large enough to assume that the injection is negatively impacting performance in the system, implement the read only cache policy.
[0089] In certain embodiments, the cache device state may be failure. The failure may be a hardware failure, software failure, or some combination thereof. The cache policy module 322 may, in response, implement the bypass cache policy which logically removes the cache device 102 from the system. Reads and writes are executed against the backing store 130 and not against the cache device 102.
[0090] When the risk of data loss on the cache device 102 is so high that the bypass cache policy is implemented for the cache device 102, the direct cache 120 may remove the cache device 102 from the software stack. In certain embodiments, the direct cache 120 remains in the call stack and routes all reads and writes to the backing store 130 without performing any cache-related functions. In certain embodiments, the direct cache 120 may be automatically removed and the backing store driver 222 put in its place. In certain embodiments, this may be accomplished through a reboot.
[0091] The tables in FIG. 4 are examples of how the cache policy for a cache device 102 may be altered to account for changes in the cache device state. Other approaches to determining the risk of data loss and adjusting the risk of data loss below a threshold level may also be used. In certain embodiments, these changes are made transparently to the host 208 and/or any clients 110. Thus, a cache device 102 may move through its life cycle and dynamically adjust the caching policy for the cache device 102 in order to protect data in the system 100. In certain embodiments, the only change from the perspective of clients 110 and a host 208 are changes in the amount of time necessary to execute read and write operations that result from the changing role of the cache device 102. As a result, a user may install a cache device 102 in a system 100 without committing to replacing the cache device 102 when it begins to degrade. In certain embodiments, the direct cache 120 may be configured to send a notification (such as an email, a system notification, or other) to the user when replacing the cache device 102 is recommended. The direct cache 120 may also provide updates to the user about the status of the cache device 102 and how the cache device 102's current state is affecting performance in the system 100.
[0092] FIG. 5 is an illustration of one embodiment of a software stack representing an exemplary system in which a direct cache 120 and a cache device 102 may be implemented. The system 500 may include a user application 502, a storage client 504, a direct cache 120, a storage controller 506, a cache device 102 (including storage devices 550a-c), and a backing store 130.
[0093] User applications 502 may be any variety of application that may generate read and write requests. For example, the user application 502 may be a database management system (DBMS), a web browser, or other of the many varied applications that read and write data from storage. The storage client 504 manages file systems, files, data, and the like. The storage client 504 may be, for example, a server, a file system, an operating system, a volume manager, a DBMS, and the like. The user applications 502 may access the storage using the storage client 504.
[0094] The storage client 504 may be in communication with the direct cache 120. In certain embodiments, the direct cache 120 presents a block interface to the storage client 504 such that the direct cache 120 appears to be a standard block storage device. In one embodiment, the direct cache 120 communicates with the storage client 504 through an I/O interface represented by the block I/O emulation layer 508. The direct cache 120 may identify itself as a standard block device to the storage client 504. As noted above, the direct cache 120 may present a sparse address space to the storage client 504. The direct cache 120 may similarly ensure that the cache device 102 and the backing store 130 are transparent to the storage client 504. The direct cache 120 may similarly provide caching services in front of one or more physical devices (represented as storage devices 550a-c) that make up the cache device 102. The direct cache 120 may hide from the storage client 504 that there are multiple physical storage devices 550a-c (for example, multiple Fusion-io cards connected to a PCIe bus).
[0095] In certain embodiments, the direct cache 120 also interfaces with the cache device 102 (by way of the storage controller 506) using a block I/O emulation layer 508. The direct cache 120 may, for example, communicate with the cache device driver 118. In such embodiments, the direct cache 120 may be able to work with a variety of different cache devices 102 by communicating with the driver for the cache device 102 using conventional protocols. In other embodiments, the direct cache 120 may communicate directly with the storage controller 506 without an intermediate cache device driver 118.
[0096] As noted above, the direct cache 120 may communicate with the cache device 102 using a caching interface 230. The caching interface 230 may allow the direct cache 120 and the cache device 102 to exchange information that may be valuable in making decisions regarding the status of the cache device 102 (such as the cache device state), which cache members are candidates for eviction, and other parameters discussed above.
[0097] Certain conventional block storage devices divide the storage media into volumes or partitions. Each volume or partition may include a plurality of sectors. One or more sectors are organized into a logical block. In certain storage systems, such as those interfacing with the Windows® operating systems, the logical blocks are referred to as clusters. In other storage systems, such as those interfacing with UNIX, Linux, or similar operating systems, the logical blocks are referred to simply as blocks. A logical block or cluster represents a smallest physical amount of storage space on the storage media that is managed by the storage manager. A block storage device may associate n logical blocks available for user data storage across the storage media with a logical block address, numbered from 0 to n. In certain block storage devices, the logical block addresses may range from 0 to n per volume or partition. In conventional block storage devices, a logical block address maps directly to a particular logical block. In conventional block storage devices, each logical block maps to a particular set of physical sectors on the storage media.
[0098] The direct cache 120, the cache device 102, and/or the backing store 130 may not directly or necessarily associate logical block addresses with particular physical blocks. The direct cache 120, the cache device 102, and/or the backing store 130 may emulate a conventional block storage interface to maintain compatibility with block storage clients 504 and with conventional block storage commands and protocols.
[0099] A direct cache 120 may store data on the cache device 102 and/or the backing store 130 as blocks, sectors, pages, logical blocks, logical pages, erase blocks, logical erase blocks, ECC chunks or in any other format or structure advantageous to the technical characteristics of the cache device 102 and/or the backing store 130. For example, in one embodiment, the backing store 130 comprises a hard disk drive and the direct cache 120 stores data on the backing store as contiguous sectors of 512 bytes, or the like, using physical cylinder-head-sector addresses for each sector, logical block addresses for each sector, or the like. The direct cache 120 may receive a logical address and a command from the storage client 504 and perform the corresponding operation in relation to the cache device 102 and/or the backing store 130. The direct cache 120, the cache device 102, and/or the backing store 130 may support a block I/O emulation layer 508, a caching interface 230, or both a block I/O emulation layer 508 and a caching interface 230.
[0100] As described above, certain storage devices, while appearing to a storage client 504 to be a block storage device, do not directly associate particular logical block addresses with particular physical blocks, also referred to in the art as sectors. Such storage devices may use a logical-to-physical translation layer 512. The cache device 102 and the backing store 130 may each include a logical-to-physical translation layer 512. In another embodiment, the direct cache 120 maintains a single logical-to-physical translation layer 512 for the cache device 102 and the backing store 130. In another embodiment, the direct cache 120 maintains a distinct logical-to-physical translation layer 512 for each of the storage devices 550a-c and the backing store 130.
[0101] The logical-to-physical translation layer 512 provides a level of abstraction between the logical block addresses used by the storage client 504 and the physical block addresses at which the cache device 120 and the backing store 130 store the data. This mapping allows data to be referenced in a logical address space using logical identifiers, such as a logical block address. A logical identifier may not indicate the physical location of data in the cache device 102; a logical identifier may be an abstract reference to the data. For example, contiguous logical block addresses may be stored in non-contiguous physical block addresses as the logical-to-physical translation layer 512 determines the location on the physical media of the cache device 102 at which to perform data operations. The logical-to-physical translation layers 512 may each include a map or index that maps logical block addresses to physical block addresses. The map may be in the form of a b-tree, a content addressable memory (CAM), a binary tree, and/or a hash table, or the like. The logical-to-physical translation layer 512 may be a tree with nodes that represent logical block addresses and include references to corresponding physical block addresses.
[0102] The mapping module and the remapping module of FIG. 4 in U.S. patent application Ser. No. 12/847,952 (referenced and incorporated above), are one example of a logical-to-physical translation layer 510. A mapping module 724 and remapping module 730 are also explained in greater detail in connection with FIG. 7 below. The storage controller 506 may also include a read data pipeline 516 and a write data pipeline 518, which are also described in the aforementioned application. The write data pipeline 518 may be used to generate ECC check bits may be stored with the associated data packets to constitute ECC chunks. The ECC check bits may be used to detect and correct errors in the ECC chunk. The write data pipeline 518 may also include input buffers, write synchronization buffers, and write buffers to enable sequencing execution of solid-state storage commands as described above. The read data pipeline 516 may include an ECC correction module, a depacketizer, an alignment module, read synchronization buffers, output buffers, and other components described in the aforementioned application which may be used for ECC correction and sequencing execution of solid-state storage commands as described above.
[0103] As discussed above, the direct cache 120 may interface with drivers for the cache device 102 and the backing store 130. In certain embodiments, the caching algorithms are implemented at the direct cache 120 such that the direct cache 120 determines where read and write operations should be directed. Such an embodiment may prevent unnecessary traffic in the system by reducing duplicative and/or unnecessary commands directed to the backing store 130 and/or the cache device 102. Thus, the direct cache 120 may be responsible for injecting cache members from the backing store 130 into the cache device 102 and for evicting cache members from the cache device 102. As discussed above, the cache device 102 may, in certain embodiments, suggest cache members as candidates for eviction.
[0104] In certain embodiments, when the cache policy implemented in the system 500 is a bypass cache policy, read requests and write requests are sent to the backing store 130 exclusively and the cache device 102 is no longer used in the system. In such embodiments, the direct cache 120 may act simply as a pass-through component, forwarding all requests to the backing store driver 222. In certain implementations, the direct cache 120 may remove itself from the software stack when the cache policy is a bypass cache policy. In other embodiments, the direct cache 120 remains in the software stack in order to facilitate the introduction of a new cache device 102 into the system 500. The direct cache 120 may determine, after the introduction of the new cache device 102, that the risk of data loss on the cache device 102 is no longer so high that the bypass cache policy is necessary, and implement a cache policy that improves the performance of the system 500.
[0105] FIG. 6 is a figure showing one embodiment of a method 600 for graceful cache degradation. Other methods may be used, and the steps do not necessarily need to be performed in the depicted order. The method 600 may begin with determining 602 a risk of data loss on the cache device 102. As noted above, this step may be performed by a monitor module 310. Determining includes, but is not limited to, polling the cache device 102 for parameters that are indicative of a change in the risk of data loss and also includes listening for notifications sent from the cache device 102 to the direct cache 120. Thus, determining (as used in this application) encompasses both pulling information and receiving pushed information. In one embodiment, the risk of data loss is determined based on the cache device state.
[0106] The method 600 may also include the step of determining 604 whether the risk of data loss exceeds a threshold risk level. This step may involve determining the cache device state of the cache device 102 and determining the risk of data loss for the cache device 102 in response to the cache device state. Thus, the cache device state may be one or many parameters considered in determining the risk of data loss. Other parameters may include, for example, the error rate and the number of read and/or write operations executed on the cache device 102.
[0107] If the risk of data loss exceeds the threshold risk level, the method may further involve identifying 606 cache policies that reduce the risk of data loss below the threshold risk level. In some instances, more than one cache policy will reduce the risk of data loss below the threshold risk level. In such instances, the method may further involve selecting the cache policy that provides the highest level of cache functionality/performance. Regardless of whether there are multiple satisfactory cache policies, or only one, the method may involve selecting 608 a cache policy to implement for the cache device 102. The method may conclude with implementing 610 the modified cache policy, as selected at step 608, for the cache device 102. The risk of data loss is thus reduced to below the threshold risk level, and the method begins anew at step 602, with determining whether the cache device performance 102 has again degraded sufficiently to require implementation of a new cache policy. In certain embodiments, this method 600 continues to iterate until the cache device 102 cannot guarantee performance sufficient to justify its use in the system, and the cache device 102 is logically removed. As discussed above, this entire process of transitioning from a fully functional cache to no cache may be transparent to clients 110 and to the host 208.
[0108] FIG. 7 illustrates one embodiment of how a mapping module 724 and a remapping module 730 may operate.
Bank Interleave
[0109] FIG. 7 is a schematic block diagram illustrating one embodiment 700 of a bank interleave controller 7344 in the storage controller 104. The bank interleave controller 7344 is connected to the control and status registers 7340 and to the storage I/O bus 7210 and storage control bus 7212 through the MUX 7350, storage bus controller 7348, and synchronization buffer 7346, which are described below. The bank interleave controller 7344 includes a read agent 702, a write agent 704, an erase agent 706, a management agent 708, read queues 710a-n, write queues 712a-n, erase queues 714a-n, and management queues 716a-n for the banks in the solid-state storage media 106, bank controllers 718a-n, a bus arbiter 720, and a status MUX 722, which are described below. The storage bus controller 7348 includes a mapping module 724 with a remapping module 730, a status capture module 726, and a NAND bus controller 728, which are described below.
[0110] The bank interleave controller 7344 directs one or more commands to two or more queues in the bank interleave controller 7344 and coordinates among the banks of the storage media 106 execution of the commands stored in the queues, such that a command of a first type executes on one bank while a command of a second type executes on a second bank. The one or more commands are separated by command type into the queues. Each bank of the storage media 106 has a corresponding set of queues within the bank interleave controller 7344 and each set of queues includes a queue for each command type.
[0111] The bank interleave controller 7344 coordinates among the banks of the storage media 106 execution of the commands stored in the queues. For example, a command of a first type executes on one bank while a command of a second type executes on a second bank. Typically the command types and queue types include read and write commands and queues 710, 712, but may also include other commands and queues that are storage media specific. For example, in the embodiment depicted in FIG. 7 , erase and management queues 714, 716 are included and would be appropriate for flash memory, NRAM, MRAM, DRAM, PRAM, etc.
[0112] For other types of storage media 106 (which is also referred to as solid-state storage media in connection with FIG. 7 ), other types of commands and corresponding queues may be included without straying from the scope of the invention. The flexible nature of an FPGA solid-state storage controller 104 allows flexibility in storage media. If flash memory were changed to another solid-state storage type, the bank interleave controller 7344, storage bus controller 7348, and MUX 7350 could be altered to accommodate the media type without significantly affecting the data pipelines 516, 518 and other storage controller 104 functions.
[0113] In the embodiment depicted in FIG. 7 , the bank interleave controller 7344 includes, for each bank, a read queue 710 for reading data from the solid-state storage media 106, a write queue 712 for write commands to the solid-state storage media 106, an erase queue 714 for erasing an erase block in the solid-state storage, an a management queue 716 for management commands. The bank interleave controller 7344 also includes corresponding read, write, erase, and management agents 702, 704, 706, 708. In another embodiment, the control and status registers 7340 and control queues 7342 or similar components queue commands for data sent to the banks of the solid-state storage media 106 without a bank interleave controller 7344.
[0114] The agents 702, 704, 706, 708, in one embodiment, direct commands of the appropriate type destined for a particular bank to the correct queue for the bank. For example, the read agent 702 may receive a read command for bank-1 and directs the read command to the bank-1 read queue 710b. The write agent 704 may receive a write command to write data to a location in bank-0 of the solid-state storage media 106 and will then send the write command to the bank-0 write queue 712a. Similarly, the erase agent 706 may receive an erase command to erase an erase block in bank-1 214b and will then pass the erase command to the bank-1 erase queue 714b. The management agent 708 typically receives management commands, status requests, and the like, such as a reset command or a request to read a configuration register of a bank 214, such as bank-0 214a. The management agent 708 sends the management command to the bank-0 management queue 716a.
[0115] The agents 702, 704, 706, 708 typically also monitor status of the queues 710, 712, 714, 716 and send status, interrupt, or other messages when the queues 710, 712, 714, 716 are full, nearly full, non-functional, etc. In one embodiment, the agents 702, 704, 706, 708 receive commands and generate corresponding sub-commands. In one embodiment, the agents 702, 704, 706, 708 receive commands through the control & status registers 7340 and generate corresponding sub-commands which are forwarded to the queues 710, 712, 714, 716. One of skill in the art will recognize other functions of the agents 702, 704, 706, 708.
[0116] The queues 710, 712, 714, 716 typically receive commands and store the commands until required to be sent to the solid-state storage banks. In a typical embodiment, the queues 710, 712, 714, 716 are first-in, first-out (“FIFO”) registers or a similar component that operates as a FIFO. In another embodiment, the queues 710, 712, 714, 716 store commands in an order that matches data, order of importance, or other criteria.
[0117] The bank controllers 718 typically receive commands from the queues 710, 712, 714, 716 and generate appropriate subcommands. For example, the bank-0 write queue 712a may receive a command to write a page of data packets to bank-0. The bank-0 controller 718a may receive the write command at an appropriate time and may generate one or more write subcommands for each data packet stored in the write buffer to be written to the page in bank-0. For example, bank-0 controller 718a may generate commands to validate the status of bank 0 and the solid-state storage array, select the appropriate location for writing one or more data packets, clear the input buffers within the solid-state storage memory array, transfer the one or more data packets to the input buffers, program the input buffers into the selected location, verify that the data was correctly programmed, and if program failures occur do one or more of interrupting the master controller, retrying the write to the same physical location, and retrying the write to a different physical location. Additionally, in conjunction with example write command, the storage bus controller 7348 will cause the one or more commands to multiplied to each of the each of the storage I/O buses with the logical address of the command mapped to a first physical addresses for storage I/O bus, and mapped to a second physical address for storage I/O bus, and so forth as further described below.
[0118] Typically, bus arbiter 720 selects from among the bank controllers 718 and pulls subcommands from output queues within the bank controllers 718 and forwards these to the storage bus controller 7348 in a sequence that optimizes the performance of the banks. In another embodiment, the bus arbiter 720 may respond to a high level interrupt and modify the normal selection criteria. In another embodiment, the master controller can control the bus arbiter 720 through the control and status registers 7340. One of skill in the art will recognize other means by which the bus arbiter 720 may control and interleave the sequence of commands from the bank controllers 718 to the solid-state storage media 106.
[0119] The bus arbiter 720 typically coordinates selection of appropriate commands, and corresponding data when required for the command type, from the bank controllers 718 and sends the commands and data to the storage bus controller 7348. The bus arbiter 720 typically also sends commands to the storage control bus 7212 to select the appropriate bank. For the case of flash memory or other solid-state storage media 106 with an asynchronous, bi-directional serial storage I/O bus 7210, only one command (control information) or set of data can be transmitted at a time. For example, when write commands or data are being transmitted to the solid-state storage media 106 on the storage I/O bus 7210, read commands, data being read, erase commands, management commands, or other status commands cannot be transmitted on the storage I/O bus 7210. For example, when data is being read from the storage I/O bus 7210, data cannot be written to the solid-state storage media 106.
[0120] For example, during a write operation on bank-0 the bus arbiter 720 selects the bank-0 controller 718a which may have a write command or a series of write sub-commands on the top of its queue which cause the storage bus controller 7348 to execute the following sequence. The bus arbiter 720 forwards the write command to the storage bus controller 7348, which sets up a write command by selecting bank-0 through the storage control bus 7212, sending a command to clear the input buffers of the solid-state storage elements 106 associated with the bank-0, and sending a command to validate the status of the solid-state storage elements associated with the bank-0. The storage bus controller 7348 then transmits a write subcommand on the storage I/O bus 7210, which contains the physical addresses including the address of the logical erase block for each individual physical erase solid-stage storage element as mapped from the logical erase block address. The storage bus controller 7348 then muxes the write buffer 320 through the write sync buffer 308 to the storage I/O bus 7210 through the MUX 7350 and streams write data to the appropriate page. When the page is full, then storage bus controller 7348 causes the solid-state storage elements associated with the bank-0 to program the input buffer to the memory cells within the solid-state storage elements. Finally, the storage bus controller 7348 validates the status to ensure that page was correctly programmed.
[0121] A read operation is similar to the write example above. During a read operation, typically the bus arbiter 720, or other component of the bank interleave controller 7344, receives data and corresponding status information and sends the data to the read data pipeline 516 while sending the status information on to the control and status registers 7340. Typically, a read data command forwarded from bus arbiter 720 to the storage bus controller 7348 will cause the MUX 7350 to gate the read data on storage I/O bus 7210 to the read data pipeline 516 and send status information to the appropriate control and status registers 7340 through the status MUX 422.
[0122] The bus arbiter 720 coordinates the various command types and data access modes so that only an appropriate command type or corresponding data is on the bus at any given time. If the bus arbiter 720 has selected a write command, and write subcommands and corresponding data are being written to the solid-state storage media 106, the bus arbiter 720 will not allow other command types on the storage I/O bus 7210. Beneficially, the bus arbiter 720 uses timing information, such as predicted command execution times, along with status information received concerning bank status to coordinate execution of the various commands on the bus with the goal of minimizing or eliminating idle time of the busses.
[0123] The master controller through the bus arbiter 720 typically uses expected completion times of the commands stored in the queues 710, 712, 714, 716, along with status information, so that when the subcommands associated with a command are executing on one bank, other subcommands of other commands are executing on other banks. When one command is fully executed on a bank, the bus arbiter 720 directs another command to the bank. The bus arbiter 720 may also coordinate commands stored in the queues 710, 712, 714, 716 with other commands that are not stored in the queues 710, 712, 714, 716.
[0124] For example, an erase command may be sent out to erase a group of erase blocks within the solid-state storage media 106. An erase command may take 10 to 1000 times more time to execute than a write or a read command or 10 to 100 times more time to execute than a program command. For N banks, the bank interleave controller 7344 may split the erase command into N commands, each to erase a virtual erase block of a bank. While bank-0 is executing an erase command, the bus arbiter 720 may select other commands for execution on the other banks. The bus arbiter 720 may also work with other components, such as the storage bus controller 7348, the master controller, etc., to coordinate command execution among the buses. Coordinating execution of commands using the bus arbiter 720, bank controllers 718, queues 710, 712, 714, 716, and agents 702, 704, 706, 708 of the bank interleave controller 7344 can dramatically increase performance over other solid-state storage systems without a bank interleave function.
[0125] In one embodiment, the solid-state controller 104 includes one bank interleave controller 7344 that serves all of the storage elements of the solid-state storage media 106. In another embodiment, the solid-state controller 104 includes a bank interleave controller 7344 for each column of storage elements. For example, one bank interleave controller 7344 serves one column of storage elements, a second bank interleave controller 7344 serves a second column of storage elements, etc.
Storage-Specific Components
[0126] The solid-state storage controller 104 includes a synchronization buffer 7346 that buffers commands and status messages sent and received from the solid-state storage media 106. The synchronization buffer 7346 is located at the boundary between the solid-state storage clock domain and the local bus clock domain and provides buffering to account for the clock domain differences. The synchronization buffer 7346, write synchronization buffer, and read synchronization buffer may be independent or may act together to buffer data, commands, status messages, etc. In one embodiment, the synchronization buffer 7346 is located where there are the fewest number of signals crossing the clock domains. One skilled in the art will recognize that synchronization between clock domains may be arbitrarily moved to other locations in order to optimize some aspect of design implementation.
[0127] The solid-state storage controller 104 may include a storage bus controller 7348 that interprets and translates commands for data sent to and read from the solid-state storage media 106 and status messages received from the solid-state storage media 106 based on the type of solid-state storage media 106. For example, the storage bus controller 7348 may have different timing requirements for different types of storage, storage with different performance characteristics, storage from different manufacturers, etc. The storage bus controller 7348 also sends control commands to the storage control bus 7212.
[0128] In one embodiment, the solid-state storage controller 104 includes a MUX 7350 that comprises an array of multiplexers 7350a-n where each multiplexer is dedicated to a row in the solid-state storage array 106. For example, multiplexer 7350a is associated with solid-state storage elements 216a, 218a, 220a. MUX 7350 routes the data from the write data pipeline 518 and commands from the storage bus controller 7348 to the solid-state storage media 106 via the storage I/O bus 7210 and routes data and status messages from the solid-state storage media 106 via the storage I/O bus 7210 to the read data pipeline 516 and the control and status registers 7340 through the storage bus controller 7348, synchronization buffer 7346, and bank interleave controller 7344.
[0129] In one embodiment, the solid-state storage controller 104 includes a MUX 7350 for each row of solid-state storage elements (e.g. SSS 0.1 216a, SSS 0.2 218a, SSS 0.N 220a). A MUX 7350 combines data from the write data pipeline 518 and commands sent to the solid-state storage media 106 via the storage I/O bus 7210 and separates data to be processed by the read data pipeline 516 from commands. Packets stored in the write buffer are directed on busses out of the write buffer through a write synchronization buffer for each row of solid-state storage elements to the MUX 7350 for each row of solid-state storage elements. The commands and read data are received by the MUXes 7350 from the storage I/O bus 7210. The MUXes 7350 also direct status messages to the storage bus controller 7348.
[0130] The storage bus controller 7348 includes a mapping module 724. The mapping module 724 maps a logical address of an erase block to one or more physical addresses of an erase block. For example, a solid-state storage media 106 with an array of twenty storage elements per block may have a logical address for a particular erase block mapped to twenty physical addresses of the erase block, one physical address per storage element. Because the storage elements are accessed in parallel, erase blocks at the same position in each storage element in a row of storage elements will share a physical address. To select one erase block instead of all erase blocks in the row, one bank is selected.
[0131] This logical-to-physical mapping for erase blocks is beneficial because if one erase block becomes damaged or inaccessible, the mapping can be changed to map to another erase block. This mitigates the loss of losing an entire virtual erase block when one element's erase block is faulty. The remapping module 730 changes a mapping of a logical address of an erase block to one or more physical addresses of a virtual erase block (spread over the array of storage elements). For example, virtual erase block 1 may be mapped to erase block 1 of storage element SSS 0.0 216a, to erase block 1 of storage element SSS 1.0 216b, . . . , and to storage element M.0 216m, virtual erase block 2 may be mapped to erase block 2 of storage element SSS 0.1 218a, to erase block 2 of storage element SSS 1.1 218b, . . . , and to storage element M.1 218m, etc. Alternatively, virtual erase block 1 may be mapped to one erase block from each storage element in an array such that virtual erase block 1 includes erase block 1 of storage element SSS 0.0 216a to erase block 1 of storage element SSS 1.0 216b to storage element M.0 216m, and erase block 1 of storage element SSS 0.1 218a to erase block 1 of storage element SSS 1.1 218b, . . . , and to storage element M.1 218m, for each storage element in the array up to erase block 1 of storage element M.N 220m.
[0132] If erase block 1 of a storage element SSS0.0 216a is damaged, experiencing errors due to wear, etc., or cannot be used for some reason, the remapping module 730 could change the logical-to-physical mapping for the logical address that pointed to erase block 1 of virtual erase block 1. If a spare erase block (call it erase block 221) of storage element SSS 0.0 216a is available and currently not mapped, the remapping module 730 could change the mapping of virtual erase block 1 to point to erase block 221 of storage element SSS 0.0 216a, while continuing to point to erase block 1 of storage element SSS 1.0 216b, erase block 1 of storage element SSS 2.0 (not shown), and to storage element M.0 216m. The mapping module 724 or remapping module 730 could map erase blocks in a prescribed order (virtual erase block 1 to erase block 1 of the storage elements, virtual erase block 2 to erase block 2 of the storage elements, etc.) or may map erase blocks of the storage elements 216, 218, 220 in another order based on some other criteria.
[0133] In one embodiment, the erase blocks could be grouped by access time. Grouping by access time, meaning time to execute a command, such as programming (writing) data into pages of specific erase blocks, can level command completion so that a command executed across the erase blocks of a virtual erase block is not limited by the slowest erase block. In other embodiments, the erase blocks may be grouped by wear level, health, etc. One of skill in the art will recognize other factors to consider when mapping or remapping erase blocks.
[0134] In one embodiment, the storage bus controller 7348 includes a status capture module 726 that receives status messages from the solid-state storage media 106 and sends the status messages to the status MUX 722. In another embodiment, when the solid-state storage media 106 is flash memory, the storage bus controller 7348 includes a NAND bus controller 728. The NAND bus controller 728 directs commands from the read and write data pipelines 518, 516 to the correct location in the solid-state storage media 106, coordinates timing of command execution based on characteristics of the flash memory, etc. If the solid-state storage media 106 is another solid-state storage type, the NAND bus controller 728 would be replaced by a bus controller specific to the storage type. One of skill in the art will recognize other functions of a NAND bus controller 728.
[0135] The present invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The described embodiments are to be considered in all respects only as illustrative and not restrictive. The scope of the invention is, therefore, indicated by the appended claims rather than by the foregoing description. All changes which come within the meaning and range of equivalency of the claims are to be embraced within their scope.
Claims
What is claimed is:
1. A method for caching using a nonvolatile storage device, the method comprising:
determining a risk of data loss on a cache device, wherein the cache device is a non-volatile storage device configured to perform cache functions for a backing store and implements a cache policy;
determining that the risk of data loss on the cache device exceeds a threshold risk level; and
implementing a modified cache policy for the cache device in response to the risk of data loss exceeding the threshold risk level, the modified cache policy reducing the risk of data loss below the threshold risk level.
2. The method of claim 1, wherein the cache policy specifies how the cache device responds to one or more of read requests and write requests.
3. The method of claim 1, wherein determining the risk of data loss comprises gauging a physical health of the cache device, and wherein the risk of data loss increases as the physical health of the cache device degrades.
4. The method of claim 1, further comprising identifying a plurality of cache policies that reduce the risk of data loss below the threshold risk level, and selecting a modified cache policy that provides a highest level of cache functionality from the plurality of cache policies.
5. The method of claim 1, wherein implementing the modified cache policy for the cache device comprises dynamically implementing the modified cache policy that reduces the risk of data loss below the threshold risk level.
6. The method of claim 5, wherein the modified cache policy reduces the risk of data loss by ensuring that data associated with write requests is preserved on the backing store.
7. The method of claim 1, wherein the modified cache policy is one of: a write through cache policy; a write around cache policy; a read only cache policy; and a bypass cache policy.
8. The method of claim 1, wherein the cache device is a solid state storage device having a plurality of banks of solid-state storage elements, wherein the cache device sequences execution of solid-state storage commands such that a first command of a first execution duration executes on a first bank concurrently with execution of a second command of a second execution duration on a second bank.
9. The method of claim 8, wherein the cache device generates an error-correcting code (ECC) chunk comprising at least a portion of received data and an ECC code capable of correcting an error in the ECC chunk, and wherein the cache device further stores a first portion of the ECC chunk on a first solid-state storage element and a second portion of the ECC chunk on a second solid-state storage element.
10. The method of claim 1, wherein the cache device comprises a plurality of physical devices cooperating such that the plurality of physical devices serve as a single logical cache device.
11. The method of claim 1, further comprising:
determining a cache device state of the cache device; and
determining the risk of data loss for the cache device in response to the cache device state.
12. An apparatus for caching using a nonvolatile storage device, the apparatus comprising:
a direct cache configured for communication with a cache device and a backing store, wherein the cache device is a non-volatile storage device configured to perform cache functions for the backing store, the direct cache comprising:
a monitor module configured for determining a risk of data loss on
the cache device;
a cache policy module configured for implementing a modified cache policy for the cache device if a risk of data loss on the cache device exceeds a threshold risk level; and
wherein the direct cache is configured to recieve, from the cache device,
one or more indicators associated with the risk of data loss on the cache device.
13. The apparatus of claim 12, wherein the cache policy module is further configured to send write requests to the cache device, send the write requests to the backing store, and receive acknowledgements in response to the backing store fulfilling the write requests.
14. The apparatus of claim 12, wherein the cache policy module is further configured to send write requests exclusively to the backing store such that the write requests bypass the cache device and to receive acknowledgements in response to the backing store fulfilling the write requests.
15. The apparatus of claim 14, wherein the cache policy module is further configured to first direct read requests to the cache device for fulfillment.
16. The apparatus of claim 12, wherein the cache policy module is further configured to bypass the cache device such that write requests are sent exclusively to the backing store, and such that read requests are sent exclusively to the backing store.
17. The apparatus of claim 12, wherein the direct cache communicates with the cache device using a caching interface.
18. The apparatus of claim 12, wherein the direct cache is a stand-alone driver.
19. A system for caching using a nonvolatile storage device, the system comprising:
a backing store configured for storing data received from one or more storage clients,
wherein the backing store comprises one or more non-volatile storage devices;
a cache device configured for performing cache functions for the backing store,
wherein the cache device is a non-volatile storage device with faster access characteristics than those of the backing store, and wherein the cache device is configured to implement a cache policy;
a direct cache configured for:
determining a risk of data loss on the cache device;
determining that the risk of data loss on the cache device exceeds a threshold risk level; and
implementing a modified cache policy for the cache device in response to the risk of data loss exceeding the threshold risk level, the modified cache policy reducing the risk of data loss below the threshold risk level.
20. The system of claim 19, wherein the direct cache is configured to communicate with the cache device using a caching interface.
21. The system of claim 19, wherein the cache policy specifies how the cache device responds to one or more of read requests and write requests received from the one or more storage clients.
22. The system of claim 19, wherein the modified cache policy reduces the risk of data loss by ensuring that data associated with write requests is preserved on the backing store.
Patent Citations (350)
| Patent | Date | Inventor | Cited By |
|---|---|---|---|
| US4571674(A) | 1986-02-01 | Hartung | Applicant |
| US4980861(A) | 1990-12-01 | Herdt et al. | Applicant |
| US5261068(A) | 1993-11-01 | Gaskins et al. | Applicant |
| US5291496(A) | 1994-03-01 | Andaleon et al. | Applicant |
| US5307497(A) | 1994-04-01 | Feigenbaum et al. | Applicant |
| US5313475(A) | 1994-05-01 | Cromer et al. | Applicant |
| US5325509(A) | 1994-06-01 | Lautzenheiser | Applicant |
| US5392427(A) | 1995-02-01 | Barrett et al. | Applicant |
| US5404485(A) | 1995-04-01 | Ban | Applicant |
| US5438671(A) | 1995-08-01 | Miles | Applicant |
| US5469555(A) | 1995-11-01 | Ghosh et al. | Applicant |
| US5499354(A) | 1996-03-01 | Aschoff et al. | Applicant |
| US5504882(A) | 1996-04-01 | Chai et al. | Applicant |
| US5551003(A) | 1996-08-01 | Mattson et al. | Applicant |
| US5559988(A) | 1996-09-01 | Durante et al. | Applicant |
| US5586291(A) | 1996-12-01 | Lasker et al. | Applicant |
| US5594883(A) | 1997-01-01 | Pricer | Applicant |
| US5596736(A) | 1997-01-01 | Kerns | Applicant |
| US5603001(A) | 1997-02-01 | Sukegawa et al. | Applicant |
| US5680579(A) | 1997-10-01 | Young et al. | Applicant |
| US5701434(A) | 1997-12-01 | Nakagawa | Applicant |
| US5734861(A) | 1998-03-01 | Cohn et al. | Applicant |
| US5745671(A) | 1998-04-01 | Hodges | Applicant |
| US5745792(A) | 1998-04-01 | Jost | Applicant |
| US5754563(A) | 1998-05-01 | White | Applicant |
| US5787486(A) | 1998-07-01 | Chin et al. | Applicant |
| US5797022(A) | 1998-08-01 | Shimotono et al. | Examiner |
| US5809527(A) | 1998-09-01 | Cooper et al. | Applicant |
| US5809543(A) | 1998-09-01 | Byers et al. | Applicant |
| US5822759(A) | 1998-10-01 | Treynor | Applicant |
| US5835935(A) | 1998-11-01 | Estakhri et al. | Applicant |
| US5845313(A) | 1998-12-01 | Estakhri et al. | Applicant |
| US5845329(A) | 1998-12-01 | Onishi et al. | Applicant |
| US5860083(A) | 1999-01-01 | Sukegawa | Applicant |
| US5893138(A) | 1999-04-01 | Judd et al. | Applicant |
| US5907856(A) | 1999-05-01 | Estakhri et al. | Applicant |
| US5924113(A) | 1999-07-01 | Estakhri et al. | Applicant |
| US5930815(A) | 1999-07-01 | Estakhri et al. | Applicant |
| US5960462(A) | 1999-09-01 | Solomon et al. | Applicant |
| US5961660(A) | 1999-10-01 | Capps, Jr. et al. | Applicant |
| US6000019(A) | 1999-12-01 | Dykstal et al. | Applicant |
| US6061511(A) | 2000-05-01 | Marantz et al. | Applicant |
| US6073232(A) | 2000-06-01 | Kroeker et al. | Applicant |
| US6101601(A) | 2000-08-01 | Matthews et al. | Applicant |
| US6105076(A) | 2000-08-01 | Beardsley et al. | Applicant |
| US6128695(A) | 2000-10-01 | Estakhri et al. | Applicant |
| US6141249(A) | 2000-10-01 | Estakhri et al. | Applicant |
| US6145051(A) | 2000-11-01 | Estakhri et al. | Applicant |
| US6170039(B1) | 2001-01-01 | Kishida | Applicant |
| US6170047(B1) | 2001-01-01 | Dye | Applicant |
| US6172906(B1) | 2001-01-01 | Estakhri et al. | Applicant |
| US6173381(B1) | 2001-01-01 | Dye | Applicant |
| US6185654(B1) | 2001-02-01 | Van Doren | Applicant |
| US6209088(B1) | 2001-03-01 | Reneris | Applicant |
| US6223308(B1) | 2001-04-01 | Estakhri et al. | Applicant |
| US6230234(B1) | 2001-05-01 | Estakhri et al. | Applicant |
| US6240040(B1) | 2001-05-01 | Akaogi et al. | Applicant |
| US6278633(B1) | 2001-08-01 | Wong et al. | Applicant |
| US6279069(B1) | 2001-08-01 | Robinson et al. | Applicant |
| US6289413(B1) | 2001-09-01 | Rogers et al. | Applicant |
| US6330688(B1) | 2001-12-01 | Brown | Applicant |
| US6334173(B1) | 2001-12-01 | Won et al. | Applicant |
| US6356986(B1) | 2002-03-01 | Solomon et al. | Applicant |
| US6370631(B1) | 2002-04-01 | Dye | Applicant |
| US6385710(B1) | 2002-05-01 | Goldman et al. | Applicant |
| US6393513(B2) | 2002-05-01 | Estakhri et al. | Applicant |
| US6404647(B1) | 2002-06-01 | Minne | Applicant |
| US6412080(B1) | 2002-06-01 | Fleming et al. | Applicant |
| US6418478(B1) | 2002-07-01 | Ignatius et al. | Applicant |
| US6418509(B1) | 2002-07-01 | Yanai et al. | Applicant |
| US6424872(B1) | 2002-07-01 | Glanzer et al. | Applicant |
| US6477617(B1) | 2002-11-01 | Golding | Applicant |
| US6515928(B2) | 2003-02-01 | Sato et al. | Applicant |
| US6516380(B2) | 2003-02-01 | Kenchammana-Hoskote et al. | Applicant |
| US6519185(B2) | 2003-02-01 | Harari et al. | Applicant |
| US6523102(B1) | 2003-02-01 | Dye et al. | Applicant |
| US6552955(B1) | 2003-04-01 | Miki | Applicant |
| US6567889(B1) | 2003-05-01 | DeKoning et al. | Applicant |
| US6587915(B1) | 2003-07-01 | Kim | Applicant |
| US6601211(B1) | 2003-07-01 | Norman | Applicant |
| US6608793(B2) | 2003-08-01 | Park et al. | Applicant |
| US6625685(B1) | 2003-09-01 | Cho et al. | Applicant |
| US6633956(B1) | 2003-10-01 | Mitani | Applicant |
| US6671757(B1) | 2003-12-01 | Multer et al. | Applicant |
| US6675349(B1) | 2004-01-01 | Chen | Applicant |
| US6710901(B2) | 2004-03-01 | Pastor | Applicant |
| US6715046(B1) | 2004-03-01 | Shoham et al. | Applicant |
| US6728851(B1) | 2004-04-01 | Estakhri et al. | Applicant |
| US6735546(B2) | 2004-05-01 | Scheuerlein | Applicant |
| US6745292(B1) | 2004-06-01 | Stevens | Applicant |
| US6751155(B2) | 2004-06-01 | Gorobets | Applicant |
| US6754774(B2) | 2004-06-01 | Gruner et al. | Applicant |
| US6754800(B2) | 2004-06-01 | Wong et al. | Applicant |
| US6757800(B1) | 2004-06-01 | Estakhri et al. | Applicant |
| US6760806(B2) | 2004-07-01 | Jeon | Applicant |
| US6775185(B2) | 2004-08-01 | Fujisawa et al. | Applicant |
| US6779088(B1) | 2004-08-01 | Benveniste et al. | Applicant |
| US6779094(B2) | 2004-08-01 | Selkirk et al. | Applicant |
| US6785776(B2) | 2004-08-01 | Arimilli et al. | Applicant |
| US6785785(B2) | 2004-08-01 | Piccirillo et al. | Applicant |
| US6801979(B1) | 2004-10-01 | Estakhri | Applicant |
| US6804755(B2) | 2004-10-01 | Selkirk et al. | Applicant |
| US6816917(B2) | 2004-11-01 | Dicorpo et al. | Applicant |
| US6839819(B2) | 2005-01-01 | Martin | Applicant |
| US6845053(B2) | 2005-01-01 | Chevallier | Applicant |
| US6845428(B1) | 2005-01-01 | Kedem | Applicant |
| US6871257(B2) | 2005-03-01 | Conley et al. | Applicant |
| US6877076(B1) | 2005-04-01 | Cho et al. | Applicant |
| US6880049(B2) | 2005-04-01 | Gruner et al. | Applicant |
| US6883068(B2) | 2005-04-01 | Tsirigotis et al. | Applicant |
| US6883069(B2) | 2005-04-01 | Yoshida | Applicant |
| US6883079(B1) | 2005-04-01 | Priborsky | Applicant |
| US6910170(B2) | 2005-06-01 | Choi et al. | Applicant |
| US6912537(B2) | 2005-06-01 | Selkirk et al. | Applicant |
| US6912618(B2) | 2005-06-01 | Estakhri et al. | Applicant |
| US6922754(B2) | 2005-07-01 | Liu et al. | Applicant |
| US6928505(B1) | 2005-08-01 | Klingman | Applicant |
| US6938133(B2) | 2005-08-01 | Johnson et al. | Applicant |
| US6977599(B2) | 2005-12-01 | Widmer | Applicant |
| US6978342(B1) | 2005-12-01 | Estakhri et al. | Applicant |
| US6981070(B1) | 2005-12-01 | Luk | Applicant |
| US6996676(B2) | 2006-02-01 | Megiddo | Applicant |
| US7000063(B2) | 2006-02-01 | Friedman et al. | Applicant |
| US7010652(B2) | 2006-03-01 | Piccirillo et al. | Applicant |
| US7010662(B2) | 2006-03-01 | Aasheim et al. | Applicant |
| US7010663(B2) | 2006-03-01 | George et al. | Applicant |
| US7013376(B2) | 2006-03-01 | Hooper, III | Applicant |
| US7013379(B1) | 2006-03-01 | Testardi | Applicant |
| US7043599(B1) | 2006-05-01 | Ware et al. | Applicant |
| US7047366(B1) | 2006-05-01 | Ezra | Applicant |
| US7050337(B2) | 2006-05-01 | Iwase et al. | Applicant |
| US7057936(B2) | 2006-06-01 | Yaegashi et al. | Applicant |
| US7058769(B1) | 2006-06-01 | Danilak | Applicant |
| US7069380(B2) | 2006-06-01 | Ogawa et al. | Applicant |
| US7076723(B2) | 2006-07-01 | Saliba | Applicant |
| US7082495(B2) | 2006-07-01 | DeWhitt et al. | Applicant |
| US7082512(B2) | 2006-07-01 | Aasheim et al. | Applicant |
| US7085879(B2) | 2006-08-01 | Aasheim et al. | Applicant |
| US7089391(B2) | 2006-08-01 | Geiger et al. | Applicant |
| US7093101(B2) | 2006-08-01 | Aasheim et al. | Applicant |
| US7096321(B2) | 2006-08-01 | Modha | Applicant |
| US7111140(B2) | 2006-09-01 | Estakhri et al. | Applicant |
| US7130956(B2) | 2006-10-01 | Rao | Applicant |
| US7130957(B2) | 2006-10-01 | Rao | Applicant |
| US7143228(B2) | 2006-11-01 | Iida et al. | Applicant |
| US7149947(B1) | 2006-12-01 | MacLellan et al. | Applicant |
| US7162571(B2) | 2007-01-01 | Kilian et al. | Applicant |
| US7167944(B1) | 2007-01-01 | Estakhri | Applicant |
| US7167953(B2) | 2007-01-01 | Megiddo | Applicant |
| US7177197(B2) | 2007-02-01 | Cernea | Applicant |
| US7178081(B2) | 2007-02-01 | Lee et al. | Applicant |
| US7181572(B2) | 2007-02-01 | Walmsley | Applicant |
| US7194577(B2) | 2007-03-01 | Johnson et al. | Applicant |
| US7197567(B1) | 2007-03-01 | Fitzgerald | Applicant |
| US7203815(B2) | 2007-04-01 | Haswell | Applicant |
| US7215580(B2) | 2007-05-01 | Gorobets | Applicant |
| US7219197(B2) | 2007-05-01 | Hatakeyama | Applicant |
| US7219238(B2) | 2007-05-01 | Saito et al. | Applicant |
| US7227777(B2) | 2007-06-01 | Roohparvar | Applicant |
| US7234082(B2) | 2007-06-01 | Lai et al. | Applicant |
| US7237141(B2) | 2007-06-01 | Fredin | Applicant |
| US7243203(B2) | 2007-07-01 | Scheuerlein | Applicant |
| US7246179(B2) | 2007-07-01 | Camara | Applicant |
| US7248691(B1) | 2007-07-01 | Pandit et al. | Applicant |
| US7257129(B2) | 2007-08-01 | Lee et al. | Applicant |
| US7263591(B2) | 2007-08-01 | Estakhri et al. | Applicant |
| US7280536(B2) | 2007-10-01 | Testardi | Applicant |
| US7305520(B2) | 2007-12-01 | Voigt et al. | Applicant |
| US7337201(B1) | 2008-02-01 | Yellin et al. | Applicant |
| US7340558(B2) | 2008-03-01 | Lee et al. | Applicant |
| US7340566(B2) | 2008-03-01 | Voth | Applicant |
| US7340581(B2) | 2008-03-01 | Gorobets et al. | Applicant |
| US7356651(B2) | 2008-04-01 | Liu et al. | Applicant |
| US7360015(B2) | 2008-04-01 | Matthews et al. | Applicant |
| US7360037(B2) | 2008-04-01 | Higaki et al. | Applicant |
| US7366808(B2) | 2008-04-01 | Kano et al. | Applicant |
| US7370163(B2) | 2008-05-01 | Yang et al. | Applicant |
| US7392365(B2) | 2008-06-01 | Selkirk et al. | Applicant |
| US7424593(B2) | 2008-09-01 | Estakhri et al. | Applicant |
| US7437510(B2) | 2008-10-01 | Rosenbluth et al. | Applicant |
| US7441081(B2) | 2008-10-01 | Humlicek | Applicant |
| US7441090(B2) | 2008-10-01 | Estakhri et al. | Applicant |
| US7447847(B2) | 2008-11-01 | Louie et al. | Applicant |
| US7450420(B2) | 2008-11-01 | Sinclair | Applicant |
| US7460432(B2) | 2008-12-01 | Warner | Applicant |
| US7463521(B2) | 2008-12-01 | Li | Applicant |
| US7480766(B2) | 2009-01-01 | Gorobets | Applicant |
| US7487235(B2) | 2009-02-01 | Andrews et al. | Applicant |
| US7487320(B2) | 2009-02-01 | Bansal et al. | Applicant |
| US7500000(B2) | 2009-03-01 | Groves et al. | Applicant |
| US7523249(B1) | 2009-04-01 | Estakhri et al. | Applicant |
| US7529905(B2) | 2009-05-01 | Sinclair | Applicant |
| US7536491(B2) | 2009-05-01 | Kano et al. | Applicant |
| US7549013(B2) | 2009-06-01 | Estakhri et al. | Applicant |
| US7552271(B2) | 2009-06-01 | Sinclair | Applicant |
| US7580287(B2) | 2009-08-01 | Aritome | Applicant |
| US7631138(B2) | 2009-12-01 | Gonzalez et al. | Applicant |
| US7640390(B2) | 2009-12-01 | Iwamura et al. | Applicant |
| US7644239(B2) | 2010-01-01 | Ergan et al. | Applicant |
| US7660911(B2) | 2010-02-01 | McDaniel | Applicant |
| US7689803(B2) | 2010-03-01 | Karr et al. | Applicant |
| US7725628(B1) | 2010-05-01 | Phan | Applicant |
| US7725661(B2) | 2010-05-01 | Liu et al. | Applicant |
| US7856528(B1) | 2010-12-01 | Frost | Applicant |
| US7873782(B2) | 2011-01-01 | Terry | Applicant |
| US7908501(B2) | 2011-03-01 | Kim et al. | Applicant |
| US2002/0069317(A1) | 2002-06-01 | Chow et al. | Applicant |
| US2002/0069318(A1) | 2002-06-01 | Chow et al. | Applicant |
| US2002/0194451(A1) | 2002-12-01 | Mukaida et al. | Applicant |
| US2003/0061296(A1) | 2003-03-01 | Craddock et al. | Applicant |
| US2003/0093741(A1) | 2003-05-01 | Argon et al. | Applicant |
| US2003/0131182(A1) | 2003-07-01 | Kumar et al. | Applicant |
| US2003/0163633(A1) | 2003-08-01 | Aasheim et al. | Applicant |
| US2003/0198084(A1) | 2003-10-01 | Fujisawa et al. | Applicant |
| US2004/0059870(A1) | 2004-03-01 | Ash et al. | Applicant |
| US2004/0093463(A1) | 2004-05-01 | Shang | Applicant |
| US2004/0186946(A1) | 2004-09-01 | Lee | Applicant |
| US2004/0268359(A1) | 2004-12-01 | Hanes | Applicant |
| US2005/0002263(A1) | 2005-01-01 | Iwase et al. | Applicant |
| US2005/0015539(A1) | 2005-01-01 | Horii et al. | Applicant |
| US2005/0027951(A1) | 2005-02-01 | Piccirillo et al. | Applicant |
| US2005/0055497(A1) | 2005-03-01 | Estakhri et al. | Applicant |
| US2005/0076107(A1) | 2005-04-01 | Goud et al. | Applicant |
| US2005/0132259(A1) | 2005-06-01 | Emmot et al. | Applicant |
| US2005/0144361(A1) | 2005-06-01 | Gonzalez et al. | Applicant |
| US2005/0149618(A1) | 2005-07-01 | Cheng | Applicant |
| US2005/0149819(A1) | 2005-07-01 | Hwang | Applicant |
| US2005/0177672(A1) | 2005-08-01 | Rao | Applicant |
| US2005/0177687(A1) | 2005-08-01 | Rao | Applicant |
| US2005/0193166(A1) | 2005-09-01 | Johnson et al. | Applicant |
| US2005/0229090(A1) | 2005-10-01 | Shen et al. | Applicant |
| US2005/0235132(A1) | 2005-10-01 | Karr et al. | Applicant |
| US2005/0240713(A1) | 2005-10-01 | Wu | Applicant |
| US2005/0246510(A1) | 2005-11-01 | Retnamma et al. | Applicant |
| US2005/0257213(A1) | 2005-11-01 | Chu et al. | Applicant |
| US2005/0276092(A1) | 2005-12-01 | Hansen et al. | Applicant |
| US2006/0004955(A1) | 2006-01-01 | Ware et al. | Applicant |
| US2006/0026339(A1) | 2006-02-01 | Rostampour | Applicant |
| US2006/0075057(A1) | 2006-04-01 | Gildea et al. | Applicant |
| US2006/0090048(A1) | 2006-04-01 | Okumoto et al. | Applicant |
| US2006/0090948(A1) | 2006-05-01 | Allen | Applicant |
| US2006/0106968(A1) | 2006-05-01 | Wooi Teoh | Applicant |
| US2006/0106990(A1) | 2006-05-01 | Benhase et al. | Applicant |
| US2006/0143396(A1) | 2006-06-01 | Cabot | Applicant |
| US2006/0152981(A1) | 2006-07-01 | Ryu | Applicant |
| US2006/0184736(A1) | 2006-08-01 | Benhase et al. | Applicant |
| US2006/0224849(A1) | 2006-10-01 | Islam et al. | Applicant |
| US2006/0230295(A1) | 2006-10-01 | Schumacher et al. | Applicant |
| US2006/0248387(A1) | 2006-11-01 | Nicholson et al. | Applicant |
| US2006/0265624(A1) | 2006-11-01 | Moshayedi et al. | Applicant |
| US2007/0016699(A1) | 2007-01-01 | Minami | Applicant |
| US2007/0050571(A1) | 2007-03-01 | Nakamura | Applicant |
| US2007/0061515(A1) | 2007-03-01 | Kano | Applicant |
| US2007/0118676(A1) | 2007-05-01 | Kano | Applicant |
| US2007/0124474(A1) | 2007-05-01 | Margulis | Applicant |
| US2007/0124540(A1) | 2007-05-01 | van Riel | Applicant |
| US2007/0150689(A1) | 2007-06-01 | Pandit | Applicant |
| US2007/0162830(A1) | 2007-07-01 | Stek et al. | Applicant |
| US2007/0198770(A1) | 2007-08-01 | Horii et al. | Applicant |
| US2007/0204128(A1) | 2007-08-01 | Lee | Applicant |
| US2007/0204197(A1) | 2007-08-01 | Yokokawa | Applicant |
| US2007/0230253(A1) | 2007-10-01 | Kim | Applicant |
| US2007/0233455(A1) | 2007-10-01 | Zimmer et al. | Applicant |
| US2007/0233937(A1) | 2007-10-01 | Coulson et al. | Applicant |
| US2007/0233938(A1) | 2007-10-01 | Cho et al. | Applicant |
| US2007/0234021(A1) | 2007-10-01 | Ruberg et al. | Applicant |
| US2007/0245094(A1) | 2007-10-01 | Lee et al. | Applicant |
| US2007/0245217(A1) | 2007-10-01 | Valle | Applicant |
| US2007/0250660(A1) | 2007-10-01 | Gill et al. | Applicant |
| US2007/0271468(A1) | 2007-11-01 | McKenney et al. | Applicant |
| US2007/0271572(A1) | 2007-11-01 | Gupta et al. | Applicant |
| US2007/0274150(A1) | 2007-11-01 | Gorobets | Applicant |
| US2007/0276897(A1) | 2007-11-01 | Tameshige et al. | Applicant |
| US2007/0300008(A1) | 2007-12-01 | Rogers | Applicant |
| US2008/0005748(A1) | 2008-01-01 | Mathew et al. | Applicant |
| US2008/0010395(A1) | 2008-01-01 | Mylly et al. | Applicant |
| US2008/0016301(A1) | 2008-01-01 | Chen | Applicant |
| US2008/0043769(A1) | 2008-02-01 | Hirai | Applicant |
| US2008/0052483(A1) | 2008-02-01 | Rangarajan et al. | Applicant |
| US2008/0059752(A1) | 2008-03-01 | Serizawa | Applicant |
| US2008/0059820(A1) | 2008-03-01 | Vaden et al. | Applicant |
| US2008/0080243(A1) | 2008-04-01 | Edahiro et al. | Applicant |
| US2008/0091876(A1) | 2008-04-01 | Fujibayashi et al. | Applicant |
| US2008/0098159(A1) | 2008-04-01 | Song | Applicant |
| US2008/0104344(A1) | 2008-05-01 | Shimozono et al. | Applicant |
| US2008/0117686(A1) | 2008-05-01 | Yamada | Applicant |
| US2008/0120469(A1) | 2008-05-01 | Kornegay et al. | Applicant |
| US2008/0123211(A1) | 2008-05-01 | Chng et al. | Applicant |
| US2008/0126700(A1) | 2008-05-01 | El-Batal et al. | Applicant |
| US2008/0126852(A1) | 2008-05-01 | Brandyberry et al. | Applicant |
| US2008/0133963(A1) | 2008-06-01 | Katano et al. | Applicant |
| US2008/0137658(A1) | 2008-06-01 | Wang | Applicant |
| US2008/0140819(A1) | 2008-06-01 | Bailey et al. | Applicant |
| US2008/0183965(A1) | 2008-07-01 | Shiga et al. | Applicant |
| US2008/0195801(A1) | 2008-08-01 | Cheon et al. | Applicant |
| US2008/0201535(A1) | 2008-08-01 | Hara | Applicant |
| US2008/0205286(A1) | 2008-08-01 | Li et al. | Applicant |
| US2008/0209090(A1) | 2008-08-01 | Kano et al. | Applicant |
| US2008/0229046(A1) | 2008-09-01 | Raciborski | Applicant |
| US2008/0235443(A1) | 2008-09-01 | Chow et al. | Applicant |
| US2008/0235528(A1) | 2008-09-01 | Kim et al. | Applicant |
| US2008/0266973(A1) | 2008-10-01 | Sekar et al. | Applicant |
| US2008/0276040(A1) | 2008-11-01 | Moritoki | Applicant |
| US2008/0294847(A1) | 2008-11-01 | Maruyama et al. | Applicant |
| US2008/0313364(A1) | 2008-12-01 | Flynn et al. | Applicant |
| US2009/0043952(A1) | 2009-02-01 | Estakhri et al. | Applicant |
| US2009/0070541(A1) | 2009-03-01 | Yochai | Applicant |
| US2009/0083478(A1) | 2009-03-01 | Kunimatsu et al. | Applicant |
| US2009/0083485(A1) | 2009-03-01 | Cheng | Applicant |
| US2009/0089485(A1) | 2009-04-01 | Yeh | Applicant |
| US2009/0089518(A1) | 2009-04-01 | Hobbet et al. | Applicant |
| US2009/0091996(A1) | 2009-04-01 | Chen et al. | Applicant |
| US2009/0125650(A1) | 2009-05-01 | Sebire | Applicant |
| US2009/0144496(A1) | 2009-06-01 | Kawaguchi | Applicant |
| US2009/0150605(A1) | 2009-06-01 | Flynn et al. | Applicant |
| US2009/0150621(A1) | 2009-06-01 | Lee | Applicant |
| US2009/0157956(A1) | 2009-06-01 | Kano | Applicant |
| US2009/0204750(A1) | 2009-08-01 | Estakhri et al. | Applicant |
| US2009/0228637(A1) | 2009-09-01 | Moon et al. | Applicant |
| US2009/0235017(A1) | 2009-09-01 | Estakhri et al. | Applicant |
| US2009/0276654(A1) | 2009-11-01 | Butterworth | Applicant |
| US2009/0300277(A1) | 2009-12-01 | Jeddeloh | Applicant |
| US2009/0313453(A1) | 2009-12-01 | Stefanus et al. | Applicant |
| US2009/0327602(A1) | 2009-12-01 | Moore et al. | Applicant |
| US2009/0327804(A1) | 2009-12-01 | Moshayedi | Applicant |
| US2010/0017556(A1) | 2010-01-01 | Chin et al. | Applicant |
| US2010/0023674(A1) | 2010-01-01 | Aviles | Applicant |
| US2010/0023676(A1) | 2010-01-01 | Moon et al. | Applicant |
| US2010/0023682(A1) | 2010-01-01 | Lee et al. | Applicant |
| US2010/0030946(A1) | 2010-02-01 | Kano et al. | Applicant |
| US2010/0077194(A1) | 2010-03-01 | Zhao et al. | Applicant |
| US2010/0102999(A1) | 2010-04-01 | Lee et al. | Applicant |
| US2010/0106917(A1) | 2010-04-01 | Ruberg et al. | Applicant |
| EP1100001 | 2001-05-01 | Applicant | |
| EP1418502 | 2004-05-01 | Applicant | |
| EP1814039 | 2007-08-01 | Applicant | |
| GB1995018407 | 1995-07-01 | Applicant | |
| GB123416.0 | 2002-11-01 | Applicant | |
| JP4-242848 | 1992-08-01 | Applicant | |
| JP8-153014 | 1996-06-01 | Applicant | |
| JP2000259525 | 2000-09-01 | Applicant | |
| JP2009122850 | 2009-06-01 | Applicant | |
| KR1020100022811 | 2010-03-01 | Applicant | |
| WO9419746 | 1994-09-01 | Applicant | |
| WO9612225 | 1996-04-01 | Applicant | |
| WO131512 | 2001-05-01 | Applicant | |
| WO201365 | 2002-01-01 | Applicant | |
| WO2008070173 | 2008-06-01 | Applicant | |
| WO2008073421 | 2008-06-01 | Applicant | |
| WO2010036757 | 2010-04-01 | Applicant |
Non-Patent Literature (67)
- Application No. PCT/US2011/047659, International Search Report and Written Opinion, Apr. 9, 2012.Applicant
- Application No. 200780050973.4, First Office Action, Jan. 26, 2011.Applicant
- Application No. 07867661.6, Office Action, Jun. 10, 2011.Applicant
- “Windows PC Accelerators”, Nov. 30, 2006, pp. 1-16, XP002476842, URL:http://download.microsoft.com/download/9/c/5/9c5b2167-8017-4bae-9fde-d599bac8184a/perfaccel.doc.Applicant
- Application No. PCT/US2007/025049, International Preliminary Report on Patentability, Mar. 11, 2009.Applicant
- Application No. PCT/US2007/025049, International Search Report and Written Opinion, May 14, 2008.Applicant
- “Am29DL322D/323D/324D”, Data Sheet, Spansion, XP-002476181, Oct. 7, 2004, Publication No. 21534, Revision D, Amendment #7.Applicant
- Ari, Ismail, “Performance Boosting and Workload Isolation in Storage Area Networks with SANCache”, Proceedings of the 23rd IEEE/14th NASA Goddard Conference on Mass Storage Systems and Technologies, May 2006, pp. 263-273, College Park, Maryland, US.Applicant
- “ASPMC-660 Rugged IDE Flash Disk PMC Module”, Asine, downloaded Nov. 18, 2009, pp. 3, http://www.asinegroup.com/products/aspmc660html.Applicant
- Bandulet, Christian, “Object-Based Storage Devices”, Sun Developer Network, Jul. 2007, pp. 7, http://developers.xun/com/solaris/articles/osd.html.Applicant
- “BiTMICRO Introduces E-Disk PMC Flash Disk Module”, Military and Aerospace Electronics East, 2004, pp. 2, http://www.bitmicro.com/press—news—releases?20040518/prt.php.Applicant
- Bulletproof Memory for RAID Servers, Part 1-3, agigatech com., 2009, pp. 12, http://agigatech.com/blog/bulletproof-memory-for-raid-servers-part-1/3.Applicant
- Casey, Michael, “Solid State File-Caching for Performance and Scalability”, Solidata, downloaded Jan. 5, 2011, pp. 5, http://www.storagesearch.com/3dram.html.Applicant
- “S2A9550 Overview”, White Paper, DataDirect Networks, 2007, pp. 17, Chatsworth, California, US.Applicant
- Casey, Michael, “Disk I/O Performance Scaling: the File Caching Solution”, Solid Data Systems, Inc., Paper #528, pp. 8.Applicant
- “EMC Virtual Infrastructure for Microsoft Applications—Data Center Solution Enabled by EMC Symmetrix V-Max and VMware ESC 3.5”, White Paper, EMC, Apr. 2009, pp. 30.Applicant
- Feresten, Paul, “NETAPP Thin Provisioning: Better for Business”, NETAPP White Paper, Network Appliance, Inc., Mar. 2007, WP-7017-0307, pp. 11.Applicant
- “File System Primer”, CoolSolutionsWiki, downloaded Oct. 18, 2006, pp. 5, http://wiki.novell.com/index.php/File—System—Primer.Applicant
- Gill, Binny S., “WOW: Wise Ordering for Writes—Combining Spatial and Temporal Locality in Non-Volatile Caches”, IBM Almaden Research Center, San Jose, California, US.Applicant
- Van Hensbergen, Eric, “Dynamic Policy Disk Caching for Storage Networking”, IBM Research Report, Nov. 28, 2006, pp. 13, RC24123 (W0611-189).Applicant
- “How NTFS Works”, Updated Mar. 28, 2003, pp. 34, http://technet.microsoff.com/en-us/library/cc781134(WS.10).aspx.Applicant
- Dan, Raz, “Implementing MLC NAND Flash for Cost-Effective, High-Capacity Memory”, White Paper M-Systems, Sep. 2003, pp. 13, 92-SR-014-02-8L, Rev. 1.1, Newark, California, US.Applicant
- Weber, Ralph O., “Information Technology-SCSI Object-Based Storage Device Commands (OSD)”, Seagate Technology, Jul. 30, 2004, pp. 171, Project T10/1355-D, Revision 10, Reference No. ISO/IEC 14776-391 : 200x ANSI INCITS.***:200x.Applicant
- “Intel Turbo Memory with User Pinning”, Intel NAND Storage Solutions, 2008, pp. 4, www.intel.com/go/nand.Applicant
- “Introduction to Samsung's Linux Flash File System—RFS”, Samsung Electronics Application Note, Nov. 2006, Version 1.0, pp. 6.Applicant
- “ioDrive—Frequently Asked Questions”, Clustered Storage Solutions: Products, downloaded Feb. 16, 2010, pp. 2, http://www.clusteredstorage.com/clustered—storage—solutions.html.Applicant
- Johnson, Michael K., “An Introduction to Block Device Drivers”, Jan. 1, 1995, pp. 6.Applicant
- Kawaguchi, Atsuo, “A Flash-Memory Based File System”, Advanced Research Laboratory, Hitachi, Ltd., Hatoyama, Saitama, JP.Applicant
- Leventhal, Adam, “Flash Storage Memory”, Communications of the ACM, Jul. 2008, pp. 47-51, vol. 51, No. 7.Applicant
- “Method for Fault Tolerance in Nonvolatile Storage”, ip.com PriorArtDatabase Technical Disclosure, Feb. 3, 2005, pp. 6, IPCOM000042269D, www.ip.com.Applicant
- IBM, “Method to Improve Reliability of SSD Arrays”, ip.com PriorArtDatabase Technical Disclosure, Nov. 5, 2009, pp. 6, IPCOM000189338D, www.ip.com.Applicant
- Volos, Haris, “Mnemosyne: Lightweight Persistent Memory”, ASPLOS'11, Mar. 5-11, 2011, pp. 13, Newport Beach, California, US.Applicant
- Morgenstern, David, “Is There a Flash Memory RAID in Your Future?” Ziff Davis Enterprise Holdings, Inc., Nov. 8, 2006, pp. 4, http://www.eweek.com.Applicant
- “NAND Flash 101: An Introduction to NAND Flash and How to Design It in to Your Next Product”, Micron Technical Note, Rev. A 11/06, pp. 28, TN-29-19: NAND Flash 101.Applicant
- Coburn, Joel, “NV-Heaps: Making Persistent Objects Fast and Safe with Next-Generation, Non-Volatile Memories”, ASPLOS'II, Mar. 5-11, 2011, pp. 13, Newport Beach, California, US.Applicant
- Wu, Michael, “eNVy: A Non-Volatile, Main Memory Storage System”, Rice University, 1994, pp. 86-97, San Jose, California, US.Applicant
- Yerrick, Damian, “Block Device”, downloaded Mar. 1, 2010, pp. 3, http://www.pineight.com/ds/block/.Applicant
- Application No. PCT/US2012/022244, International Search Report and Written Opinion, Jul. 30, 2012.Applicant
- Application No. 11952123, Office Action, May 5, 2010.Applicant
- Application No. PCT/US2011/046005, International Search Report and Written Opinion, Apr. 6, 2012.Applicant
- Application No. 11952123, Office Action, May 10, 2010.Applicant
- Application No. 07867661.6, Office Action, Oct. 10, 2011.Applicant
- “EEL-6892—Virtual Computers”, Lecture 18, http://www.acis.ufl.edu/ming/lectures.Applicant
- “Hystor: Making SSDs the ‘Survival of the Fittest’ in High-Performance Storage Systems”, pp. 14.Applicant
- Application No. 11952123, Notice of Allowance, May 5, 2011.Applicant
- “Datadirect Storage Systems Selected for NCSA Cluster”, High Performance Computing, Aug. 15, 2003, pp. 2, vol. 12, No. 32, http://www.hpcwire/com/hpcwire/hpcwireWWW/03/0815/105731.html.Applicant
- “Introducing Box Hill's Fibre Box”, Box Hill Systems Corporation, Jan. 16, 1997, p. 5.Applicant
- Makulowich, John, “Strategies, Players and Emerging Markets”, Washington Technology, Jun. 26, 1997, pp. 6, http://washingtontechnology.com/Articles/1997/06/26/Strategies-Players-and-Emerging-Ma. . .Applicant
- Application No. PCT/US2010/048321, International Search Report and Written Opinion, Apr. 28, 2011.Applicant
- “jZip A Free WinZip Alternative”, First jZip by Greg Kowal, 2012.Applicant
- Megiddo, Nimrod, “ARC: A Self-Tuning, Low Overhead Replacement Cache”, Proceedings of FAST '03: 2nd USENIX Conference on File and Storage Technologies, Mar. 31-Apr. 2, 2003, pp. 17, San Francisco, California, US.Applicant
- Whyte, Barry, “IBM SAN Volume Controller 4.2.1 Cache Partitioning”, Redpaper, Apr. 30, 2008, pp. 12, REDP-4426-00.Applicant
- Brandon, Jr., Daniel, “Sparse Matrices in CS Education”, Consortium for Computing Sciences in Colleges, May 2009, pp. 93-98.Applicant
- “Pivot3 RAIGE Storage Cluster”, White Paper Technology Overview, Jun. 2007, pp. 17, www.pivot3.com.Applicant
- Plank, James S., “A Tutorial on Reed-Solomon Coding for Fault-Tolerance in RAID-like Systems”, pp. 19, Technical Report CS-96-332, Department of Computer Science, University of Tennessee, http://www.cs.utk.edu/plank/papers/CS-03-504.html.Applicant
- Rosenblum, Mendel, “The Design and Implementation of a Log-structured File System”, 1992, pp. 101.Applicant
- Casey, Michael, “SAN Cache: SSD in the SAN”, Computer Technology Review's Storage Inc., Quarter 1, 2000, pp. 4, SolidData, Santa Clara, California, US.Applicant
- Hynix 48-GB Flash MCP, Slashdot, downloaded Nov. 4, 2010, pp. 17, http://hardware.slashdot.org/article.pl?sid=07/09/06/146218&from=rss.Applicant
- “Sybase: Maximizing Performance through Solid State File-Caching”, SolidData Systems Best Practices Guide, May 2000, pp. 4, Santa Clara, California, US.Applicant
- Stokes, Jon, “Intel's New Flash Tech to Bring Back Turbo Memory, For Real”, downloaded Apr. 23, 2012, pp. 2, http://arstechnica.com/hardware/news/2009/09intels-new-flash-tech-to-b . . .Applicant
- “Design and Use Considerations for NAND Flash Memory”, Micron Technical Note, 2006, pp. 8, TN-29-17: NAND Flash Design and Use Considerations.Applicant
- “Hamming Codes for NAND Flash Memory Devices”, Micron Technical Note, 2005, pp. 7, TN-29-08: Hamming Codes for NAND Flash Memory Devices.Applicant
- “The Advantages of Object-Based Storage—Secure, Scalable, Dynamic Storage Devices”, Seagate Research, Apr. 2005, pp. 7, TP-536.Applicant
- Wacha, Rosie, “Improving RAID-Based Storage Systems with Flash Memory”, First Annual ISSDM/SRL Research Symposium, Oct. 20, 21, 2009, pp. 21.Applicant
- Wang, Feng, “OBFS: A File System for Object-based Storage Devices”, 21st IEEE/12th NASA Goddard Conference on Mass Storage Systems and Technologies (MSST2004), Apr. 2004, pp. 18, College Park, Maryland, US.Applicant
- Woodhouse, David, “JFFS: The Journalling Flash File System”, Red Hat, Inc., pp. 12, [email protected].Applicant
- Mesnier, Mike, “Object-Based Storage”, IEEE Communications Magazine, Aug. 2003, pp. 84-90.Applicant