Method and apparatus for identifying a faulty component on a multiple component field replaceable unit

[0001] The present invention is directed to providing data storage. In particular, the present invention is directed to providing identifying information regarding a faulty component in a multiple component field replaceable unit.

[0002] The need to store digital files, documents, pictures, images and other data continues to increase rapidly. As a result, the demand for data storage continues to increase. In addition, recent legislation, such as the Sarbanes-Oxley Act in the United States, affecting the management of electronic records, has increased the need for data storage. As the demand for data storage has increased, the space or volume occupied by storage systems has become an important issue. In particular, data storage having high capacity, high density, and space efficiency has become increasingly desirable.

[0003] In order to provide increased storage space, storage devices with ever greater storage capacities are being developed. However, the storage needs of even small enterprises can easily exceed the storage capacity of a single data storage device. In addition, in order to safeguard data, systems providing data redundancy that include multiple storage devices are necessary.

[0004] Systems that provide at least some integration of individual storage devices, such as JBOD (just a bunch of disks), SBOD (switched bunch of disks) or RAID (redundant array of independent disks) systems have been developed. Such systems are typically deployed within enclosures to present an integrated component to the user. In order to facilitate serviceability and packaging, such systems may include sleds or carriers to which a number of storage devices are mounted. The sled may also provide interconnections to allow the attached storage devices to be operatively connected to a controller and/or a data bus. By providing storage devices attached to sleds, the removal and insertion of storage devices into a system enclosure can be facilitated. Accordingly, each sled and its associated storage devices can comprise an individual field replaceable unit (FRU) within a data storage system.

[0005] When there is a storage device failure within the data storage system, the FRU is replaced and returned to the manufacturer for servicing. Where there are a number of storage devices included in an FRU that have been removed due to a failure, all of the storage devices are usually treated as faulty, even though there may be only one storage device that is in fact faulty. As a result, fully operational storage devices may be permanently removed from service. The percentage of waste that results from this practice can be calculated by the number of good drives on a sled divided by the number of bad drives. For example, if there are two drives on a sled, there is a 100% waste if only one drive on that sled is actually bad. As a further example, if there are four drives on a sled and one drive is bad, there is a 300% waste.

[0006] If all of the storage devices are not treated as defective, full failure analysis must be performed for each drive on the sled. This task can consume large amounts of manpower and other resources. Furthermore, even if a full analysis is conducted, it cannot guarantee that the faulty drive will actually be detected. For example, failures can be transient, related to the data storage system with which the data storage device was associated, and/or related to the particular environment in which the data storage system operates. As a result, the practice of discarding all drives on a sled in response to detecting a failure associated with one of the interconnected drives has continued.

[0007] The present invention is directed to solving these and other problems and disadvantages of the prior art. In accordance with embodiments of the present invention, a storage system that includes one or more field replaceable units (FRUs) that each comprise a storage device sled, and one or more storage devices is provided. Further embodiments of the present invention include memory interconnected to each sled for storing information related to a failed component, such as failed storage device within the FRU associated with the sled. In accordance with still other embodiments of the present invention, information related to the identity of a failed component within an FRU is stored in bulk data storage and/or memory associated with one or more storage devices included in the FRU.

[0008] In accordance with embodiments of the present invention, a method for identifying a failed component, such as a failed storage device, within an FRU is provided. The method includes detecting or receiving an indication of a fault related to a component included as part of an FRU. Identification information related to the failed component is collected, as is information related to the location of the component on its associated sled. Information related to the identity and the location of the FRU having the failed component within the storage system may also be collected. Such information is stored in a storage facility provided as part of the FRU. The storage facility may comprise memory provided as part of electronic circuitry incorporated into or associated with the sled of the FRU, memory included in control circuitry associated with one or more of the storage devices included in the FRU, and/or memory comprising the data storage of one or more of the storage devices included in the FRU.

[0009] In accordance with still other embodiments of the present invention, in response to a component failure, an administrator or operator is notified of the failure. The FRU including the failed component is then removed from the storage system. The removed FRU may include not only the failed component, but other components that have not failed. In addition, the information collected regarding the identity and location of the failed component on the sled of the FRU is maintained in memory or storage included in the FRU. The administrator or operator can then return the storage system to full operational capabilities by replacing the FRU having a failed component (i.e., by providing a replacement FRU).

[0010] In accordance with further embodiments of the present invention, an FRU including a failed component can be serviced. Servicing may include interrogating the FRU to obtain information identifying the failed component. Repairs can then be made with respect to the identified component, while any other components included in the FRU can be left alone. For example, a storage device identified as being the component that has failed can be replaced, while other storage devices attached to the sled can be reused. The repaired FRU can then be redeployed to the field as a spare field replaceable unit.

[0011] Other features and advantages of embodiments of the present invention will become more readily apparent from the following description, particularly when taken together with the accompanying drawings.

[0018] The present invention is directed to providing a storage system for instructions or data, hereinafter referred to as a data storage system, which includes one or more field replaceable units (FRUs). Each of the FRUs may comprise multiple physical storage devices interconnected to a common sled. In addition, each of the FRUs includes memory capable of maintaining information related to the identification of a failed storage device in response to the detection of a failure of a storage device included in that field replaceable unit (FRU). Furthermore, an FRU in accordance with embodiments of the present invention may be configured for easy removal and replacement to facilitate the maintenance of a data storage system.

[0019] With reference now to FIG. 1, a computer system 100 incorporating a data storage system 104 in accordance with embodiments of the present invention is illustrated. In general, the computer system 100 includes the data storage system 104 and one or more host processors or computers 108 interconnected to the data storage system 104 by a bus and/or network 112. Accordingly, embodiments of the present invention have applications in association with single or multiple hosts 108 and storage area network (SAN) or direct connect environments.

[0020] The data storage system 104 may include an enclosure or chassis 116 to interconnect and/or house at least some of the other components of the data storage system 104. In addition, the data storage system 104 may include a number of FRUs 120. Each FRU 120 includes one or more physical storage devices 124 interconnected to a storage device sled 128. The data storage system 104 may additionally include one or more controllers 132, one or more power supplies 136, and one or more cooling units 140.

[0021] Each FRU 120 provided as part of a data storage system 104 comprises a unit in which the one more storage devices 124 are interconnected to the sled 128. By interconnecting each storage device 124 of an FRU 120 to a common sled 128, replacement of one or more of the storage devices 124 of an FRU 120 can be effected by removing the entire FRU 120 from the enclosure or chassis 116 and data system 104, and inserting a replacement FRU 120 in the data storage system 104. Accordingly, embodiments of the present invention may include field replaceable units 120 that can be easily removed from and added to a data storage system 104.

[0022] The controller 132 performs functions to obtain and store information identifying a failed storage device in response to detection of such a failure. In particular, information collected may include information identifying a failed storage device 124, identifying the position of a failed storage device 124 on its associated sled 128, and identifying the sled 128 containing a failed storage device 124. Furthermore, the controller 132 stores at least some of the collected information related to the detected failure on the FRU 120 of which the storage device 124 is a part. Storing information on an FRU 120 related to a detected failure of a storage device 124 included in that FRU 120 can include storing such information in memory provided as part of or interconnected to the sled 128, as part of control electronics associated with one or more of the storage devices 124 associated with the FRU 120, or on the data storage media of one or more of the storage devices 124 included in the FRU 120.

[0023] The controller 132 may also function to control the flow of data between the bus or network 112 and storage devices 124 included in the data storage system 104. Accordingly, the controller 132 may comprise a controller or adapter that allows the multiple storage devices 124 to appear as a single storage device to a host 108. Therefore, a controller 132 may additionally comprise a RAID controller, implementing any one of various RAID schemes, a JBOD system controller, an SBOD system controller, or a controller for any other storage system featuring a number of storage devices 124. However, such system level control functions do not need to be performed by a controller 132 provided as part of a data storage system 104. For example, a controller that implements a RAID array using storage devices 124 provided as part of the data storage system 104 may be implemented as a separate controller, or as software provided as part of or running on a host 108 or other computer system, and may operate in cooperation with the controller 132 as part of the data storage system 104.

[0024] The one or more power supplies 136 may be included to provide power in the format required by the other components of the data storage system 104. For example, a power supply 136 may convert 120 Volt line power to a 12 Volt direct current for supply to the data storage devices 124, and a 3 Volt direct current for supply to the controller 132. Although a single power supply 136 is illustrated in FIG. 1, it can be appreciated that multiple power supplies 136 may be included in order to meet power requirements, or to provide redundancy.

[0025] The one or more cooling units 140 may be provided to help ensure that the nominal operating temperatures of storage devices 124 and other components associated with the data storage system 104 are not exceeded. In accordance with embodiments of the present invention, the cooling unit 140 comprises a fan or blower powered by an electric motor. Although shown as separate from the power supply 136, the cooling unit 140 may be integrated with the power supply 136. In addition, multiple cooling units 140 may be provided. A cooling unit 140 may also use various other cooling techniques, including systems that incorporate a liquid cooling medium.

[0026] Each storage device 124 may be separately addressable by at least a controller 132 provided as part of the data storage system 104. As an example, the storage device 124 may comprise a hard disk drive, such as a serial advanced technology attachment (SATA) disk drive. Accordingly, it can be appreciated that embodiments of the present invention may utilize relatively inexpensive storage devices 124. Examples of other storage devices 124 that may be used in connection with embodiments of the present invention include small computer systems interface (SCSI), serial attached SCSI (SAS) and fiber channel (FC) hard disk drives. Embodiments of the present invention may also utilize storage devices 124 other than devices utilizing magnetic disks as a storage medium. For example, a storage device 124 may also include magnetic tape devices, optical storage devices or solid state disk devices.

[0027] A storage device sled 128 generally provides a structure that permits two or more storage devices 124 to be interconnected to the storage device sled 128 forming at least part of an FRU 120. Furthermore, the structure provided by the storage device sled 128 facilitates the removal and replacement of storage device units 120 and their associated storage devices 124 by allowing all of the components of a storage device unit 120 to be removed from (or installed in) a data storage system 104. Accordingly, each FRU 120 may be replaced as a unit by personnel in the field.

[0028] In addition to providing a physical structure that permits the interconnection of one or more storage devices 124 to the sled 128, and to one another when multiple storage devices 124 are interconnected to the sled 128, a sled 128 may provide electrical signal and power channels. For example, a storage device sled 128 may include conductive traces for providing signal lines required by attached storage devices 124 and conductive traces for supplying power to those storage devices 124. The storage device sled 128 may additionally provide circuitry for performing various functions, such as implementing a controller 132 and/or for receiving additional components, such as a controller 132 integrated circuit or chip. In addition, the storage device sled 128 may incorporate memory or other storage space for information related to the identification of a failed storage device 124.

[0029] With reference now to FIG. 2, a data storage system 104 in accordance with embodiments of the present invention is shown in a perspective view. In addition, the example data storage system 104 in FIG. 2 includes a chassis 116 comprising an enclosure, shown with the top and front covers removed. As shown, embodiments of the present invention can provide for densely packaged storage devices 124. In particular, where each storage device sled 128 has m storage devices 124 attached to it, and there are n storage device sleds 128, the total number of storage devices 124 in the data storage system 104 is n×m where n is the number of storage device sleds and m is the number of storage devices. In the example shown in FIG. 2, there are 24 storage device sleds 128 (i.e., n=24), and each storage device sled 128 has two interconnected storage devices 124 (i.e., m=2). Accordingly, the example data storage system 104 shown in FIG. 2 includes 48 storage devices 124 provided as part of 24 FRUs 120.

[0030] In accordance with embodiments of the present invention, each FRU 120 and its associated storage device sled 128 and storage devices 124 can be removed by releasing fasteners 204 and sliding the storage device sled 128 and attached storage devices 124 out of the enclosure 116. Furthermore, a failed FRU 120 can be replaced by inserting a spare FRU 120 in the slot within the enclosure 116 that had been occupied by the failed FRU 120. In accordance with still other embodiments of the present invention, a replacement FRU can be placed in any available slot in the enclosure 116.

[0031] Continuing the example shown in FIG. 2, power supplies 136 and cooling units 140 are mounted towards the rear of the enclosure 116. Furthermore, space between the power supplies 136 and cooling units 140 is provided for a controller 132 or other componentry.

[0032] With reference now to FIG. 3, a single FRU 120 having two storage devices 124 is illustrated. As seen in FIG. 3, the storage devices 124 are interconnected to the storage device sled 128 by storage device fasteners 304. In addition, the embodiment illustrated in FIG. 3 incorporates fasteners 204 for securing the FRU 120 in an enclosure or to a chassis 116 that are incorporated into the storage device sled 128. The example FRU 120 of FIG. 3 also includes a power and signal interface 308 to connect the FRU 120 to the bus or network 112, for example through a mating connector and complementary signal lines provided as part of or within the enclosure 116, either directly or through a controller 132 or other components when the FRU 120 is operatively connected to the remainder of the data storage system 104.

[0033] With reference now to FIG. 4, components of an FRU 120 in accordance with embodiments of the present invention are illustrated. In general, the FRU 120 may include a number of storage devices 124. Although three storage devices 124a, 124b and 124n are illustrated in FIG. 4, it should be appreciated that an FRU 120 is not limited to any particular number of storage devices 124. Each storage device 124 is interconnected to a storage device sled 128. Accordingly, where multiple storage devices 124 are included in an FRU 120, each of the storage devices 124 is interconnected to one another through the storage device sled 128. In addition, the FRU 120 may include memory 404, which can be used for storing information related to the failure of a storage device 124 or other component of an FRU 120, as described herein. In accordance with embodiments of the present invention, the memory 404 may comprise a serial EEPROM or other non-volatile memory. The FRU 120 may also include a power and signal interface 308. In accordance with embodiments of the present invention, the power and signal interface 308 may comprise a connector that operatively interconnects other components of the FRU 120, such as the storage devices 124 and memory 404, to the data storage system 104 controller 132 and/or the system bus or network 112 when the FRU 120 is interconnected to the data storage system 104.

[0034] Each storage device 124 may include a device controller 408 and data storage 412. The device controller 408 may generally control the operation of the storage device 124. In addition, the device controller 408 may include memory 416. The memory 416 can be used for storing information related to the failure of a storage device 124 or other component of an FRU 120, as described herein. As used herein, the data storage 412 may comprise the storage media used to store data and associated structures for writing data to or reading data from the storage media. For example, in connection with a storage device 124 comprising a hard disk drive, the data storage 412 generally includes storage media in the form of magnetic disks, and a structure for writing data to and reading data from the storage media comprising one or more head disk assemblies comprising movable actuator arms and associated read/write heads generally under the control of the device controller 408.

[0035] With reference now to FIG. 5, aspects of the operation of a data storage system 104 in accordance with embodiments of the present invention in response to the detection of a storage device 124 failure are illustrated. Initially, the storage system 104 is powered on (step 500). The data storage system 104 is then available for host input/output (IO) operations (step 504). At step 508, a determination is made as to whether a failure of a storage device 124 included in the data storage system 104 has been detected. If no failure has been detected, the process may return to step 504, and the data storage system 104 remains available for host IO operations.

[0036] If a failure of a storage device 124 is detected, the location of the FRU 120 with the failed storage device 124 within the data storage system 104 is determined (step 512). An identifier associated with the FRU having a failed storage device may also be determined. In addition, the location of the failed storage device 124 on the sled 128 of the FRU 120 is determined (step 516). At step 520, the identifier of the failed storage device 124 is determined. The determined parameters may then be saved to a location on the FRU 120 in which the failed storage device 124 is included (step 524). Saving the determined parameters to a location on the FRU 120 can include storing that information in memory 404 provided generally with respect to the FRU 120, on memory 416 included in or provided as part of one or more of the storage devices 124, or in memory comprising the data storage 412 provided as part of one or more of the storage devices 124. As can be appreciated by one of skill in the art from the description provided herein, where parameters regarding a failed storage device 124 are stored in memory 416 and/or data storage 412 associated with a storage device 124, such parameters may be stored in memory 416 or data storage 412 associated with a storage device 124 that is not determined to have failed.

[0037] At step 528, the FRU 120 with the failed storage device 124 is removed from the data storage system 104. In order to return the data storage system 104 to its original operating status, the removed FRU 120 can be replaced by a spare FRU 120 (step 532). For example, a spare FRU 120 can be interconnected to the data storage system 104 by placing it in the slot of the enclosure 116 that had been occupied by the FRU 120 on which a data storage device 124 failed.

[0038] As can be appreciated by one of skill in the art after consideration of the description provided herein, various steps described in connection with FIG. 5 need not be performed in the order presented. For instance, steps 516 through 524 can be reordered. In addition, although the example given in FIG. 5 relates to a failure associated with a storage device 124, embodiments of the present invention are not so limited. For example, a failure associated with another component of an FRU 120 can be detected, and information related to such failure can be stored on the FRU 120, prior to removal of the FRU 120 from the data storage system 104. Furthermore, it should be appreciated that all of the described parameters need not be determined and stored. However, at least a parameter or parameters identifying a failed storage device 124 (or other failed component of the FRU 120) should be stored somewhere on the FRU. Also, it should be appreciated that other parameters may be determined and stored if desired.

[0039] The determination of failure parameters such as described in connection with FIG. 6 may be performed in connection with the execution of instructions by the data storage device 104 controller 132. Furthermore, the storage of determined failure parameters on an FRU 120 associated with a detected failure can be performed by the controller 132. As can be appreciated by one of skill in the art after consideration of the disclosure provided herein, by storing failure parameters or information on a field replaceable unit 120 associated with a detected failure, errors that might otherwise be caused by a disassociation of such information with the affected FRU 120 can be avoided. Furthermore, administration of repair procedures can be facilitated by removing the need to associate separately maintained failure information with an FRU 120 having a failed component in order to retrieve information identifying the failed component.

[0040] With reference now to FIG. 6, aspects of a storage system repair operation in accordance with embodiments of the present invention are illustrated. In general, the process illustrated in FIG. 6 may be performed after an FRU 120 having a failed storage device 124 or that is otherwise associated with a failure has been removed from a data storage system 104. Furthermore, the process illustrated in connection with FIG. 6 is representative of steps that may be taken at or by a repair facility.

[0041] Initially, the FRU 120 associated with a failure is connected to a diagnostic system (step 604). As can be appreciated by one of skill in the art after appreciation of the description provided herein, a diagnostic system may comprise a general purpose computer or a test board capable of being interconnected to an FRU 120 and of retrieving the parameters that were stored on the FRU 120 pertaining to a detected failure. After connecting the FRU 120 a diagnostic system, the failure parameters are read from the FRU 120 (step 608). In general, where a storage device 124 has experienced a failure, the parameters stored on the FRU 120 identify the failed storage device 124 by its location on the sled 128 of the FRU 120, and may also include a storage device 124 identifier. Accordingly, by reading the stored parameters from the FRU 120, service personnel can be apprised of the storage device 124 requiring replacement. Furthermore, where the identifier of the storage device 124 that has failed is stored, the storage device 124 at the indicated position can be inspected or queried for its identifier, to confirm the identity of the storage device 124 before it is removed from the FRU 120. In accordance with embodiments of the present invention in which an identifier associated with the FRU 120 is stored, the stored identifier can be compared to an identifier embedded in the FRU 120 to confirm that the failure parameters pertain to the FRU 120 being serviced. The storage device 124 identified as the failed storage device 124 is then replaced with a fully operational storage device 124 (step 612). After replacement of the failed storage device 124, the FRU 120 can be returned to service, for example as a spare FRU 120 (step 616). Returning an FRU 120 to service can include erasing and/or reformatting included data storage devices 124.

[0042] Although various examples provided herein have discussed the failure of a storage device 124, it should be appreciated that embodiments of the present invention are not so limited. For example, parameters associated with the failure of other components included in an FRU 120 can be determined and stored on that FRU 120.

[0043] In accordance with embodiments of the present invention, an algorithm may be used to identify a storage device 124 associated with a fault. In accordance with an exemplary one of such embodiments, each storage device 124 in a data storage system 104 may be numbered sequentially with a device ID and may be zero based (i.e., the first storage device 124 on the first sled 128 is assigned the number 0 as a device ID). The location of a faulty storage device 124 can be determined by the formula: D modulus of N, where D is the sequentially assigned device ID, and where N is the number of storage devices 124 interconnected to each sled 128.

[0044] The location of a sled 128 included as part of the FRU 120 having the faulty storage device 124 is also determined. In particular, each sled 128 has an identifier, which is assigned sequentially and is based on the physical location of the sled 128 within the data storage system 104 enclosure 116. The location of a sled 128, which corresponds to the location of an FRU comprising that sled 128, can then be given by the following: (D−S)/N+M, where S is the starting drive identifier and where M is the number of missing sleds 128 below the sled 128 for which the location is being determined.

[0045] The sled location 128 and storage device 124 location on the sled 128 can then be stored in memory or data storage provided as part of the FRU 120 including the identified storage device 124 and sled 128. In addition, the storage device 124 identifier and/or serial number can be stored on the FRU 120. As can be appreciated by one of skill in the art, the example algorithms can be implemented using instructions executed by the controller 132. Furthermore, it should be appreciated that the example algorithms should be performed using whole number operations rather than floating point operations.

[0046] Accordingly, it can be appreciated that embodiments of the present invention facilitate the maintenance of data storage systems 104 having a number of data storage devices 124, and in which multiple storage devices 124 are interconnected to common sleds 128. More particularly, embodiments of the present invention permit an administrator or operator of a data storage system 104 to remove an FRU 120 having a number of attached storage devices 124, including at least one failed storage device 124, as a single unit. The administrator or operator may further return a data storage system 104 to full operational capabilities by replacing a removed FRU 120 with a spare FRU 120. As can be appreciated by one of skill in the art, returning a data storage system 104 to full operational capability after removing an FRU 120 with a failed storage device 124 may comprise rebuilding data according to a redundant data storage scheme.

[0047] Embodiments of the present invention further operate to reduce waste and/or servicing costs by storing information identifying a failed storage device 124 in memory or data storage included in the FRU 120. As a result, information related to the failure of an FRU 120 component is not disassociated from the FRU 120, and therefore maintenance personnel can access such information from the FRU 120 itself. Accordingly, repair personnel can identify a failed device 124 positively, without requireing testing of the individual storage devices 124. In addition, information identifying a failed storage device 124 may be redundant, for example by identifying a storage device 124 by its position on a storage device sled 128 and by an associated identifier. Furthermore, by maintaining information identifying a failed storage device 124 as part of the FRU 120 that includes the failed storage device 124, the accuracy and speed with which a failed storage device 124 can be identified can be increased.

[0048] The foregoing discussion of the invention has been presented for purposes of illustration and description. Further, the description is not intended to limit the invention to the form disclosed herein. Consequently, variations and modifications commensurate with the above teachings, within the skill or knowledge of the relevant art, are within the scope of the present invention. The embodiments described hereinabove are further intended to explain the best mode presently known of practicing the invention and to enable others skilled in the art to utilize the invention in such or in other embodiments and with the various modifications required by their particular application or use of the invention. It is intended that the appended claims be construed to include alternative embodiments to the extent permitted by the prior art.