Method and system for end-to-end problem determination and fault isolation for storage area networks

[0002] 1. Technical Field

[0003] The present invention relates to an improved data processing system and, in particular, to a method and apparatus for computer network managing.

[0004] 2. Description of Related Art

[0005] A Storage Area Network (SAN) is an “open system” storage architecture that allows multiple host computers to share multiple storage peripherals, and in particular, to share storage peripherals via a Fibre Channel (FC) network switch. The FC switch, host systems, and storage peripherals may be manufactured by different vendors and contain different operating environments.

[0006] Currently, there is a lack of an end-to-end problem determination capability or specification for an FC SAN. A complex configuration of multi-vendor systems, network switches, and peripherals makes it significantly more difficult to perform problem determination in a SAN environment than existing point-to-point storage configurations. As a result, failures in a SAN environment will cause an increase of system downtime as well as increasing cost of system maintenance.

[0007] It would be advantageous to have a method and apparatus that defines an “open system”, real-time, end-to-end, error detection architecture that incorporates fault isolation algorithms to identify failing systems and/or components connected to a SAN.

[0008] A method and system for problem determination and fault isolation in a storage area network (SAN) is provided. A complex configuration of multi-vendor host systems, FC switches, and storage peripherals are connected in a SAN via a communications architecture (CA). A communications architecture element (CAE) is a network-connected device that has successfully registered with a communications architecture manager (CAM) on a host computer via a network service protocol, and the CAM contains problem determination (PD) functionality for the SAN and maintains a SAN PD information table (SPDIT). The CA comprises all network-connected elements capable of communicating information stored in the SPDIT. The CAM uses a SAN topology map and the SPDIT are used to create a SAN diagnostic table (SDT). A failing component in a particular device may generate errors that cause devices along the same network connection path to generate errors. As the CAM receives error packets or error messages, the errors are stored in the SDT, and each error is analyzed by temporally and spatially comparing the error with other errors in the SDT. If a CAE is determined to be a candidate for generating the error, then the CAE is reported for replacement if possible.

[0009] The novel features believed characteristic of the invention are set forth in the appended claims. The invention itself, however, as well as a preferred mode of use, further objectives and advantages thereof, will best be understood by reference to the following detailed description of an illustrative embodiment when read in conjunction with the accompanying drawings, wherein:

[0010] FIG. 1 is a pictorial representation depicting a data processing system in which the present invention may be implemented;

[0011] FIG. 2 is an example block diagram illustrating internal components of a server-type data processing system that may implement the present invention;

[0012] FIG. 3 is a diagram depicting a communications architecture for data processing systems that participate in the SAN problem determination methodology implemented in accordance with a preferred embodiment of the present invention;

[0013] FIG. 4 is a table depicting a SAN Problem Determination Information Table (SPDIT) in accordance with a preferred embodiment of the present invention;

[0014] FIG. 5A is a simplified network topology diagram for a SAN;

[0015] FIG. 5B is a table providing a topology map for the SAN shown in FIG. 5A;

[0016] FIG. 6 is a diagram depicting a SAN Diagnostic Table for a SAN;

[0017] FIG. 7 is a table depicting the weightings to be used in real-time diagnostic analysis for various errors; and

[0018] FIGS. 8A-8D are flowcharts depicting a process for a real-time diagnostic algorithm for SAN end-to-end fault isolation of a single failing SAN element in accordance with a preferred embodiment of the present invention.

[0019] With reference now to FIG. 1, a pictorial representation depicts a data processing system in which the present invention may be implemented. A computer 100 is depicted, which includes a system unit 110, a video display terminal 102, a keyboard 104, storage devices 108, which may include floppy drives and other types of permanent and removable storage media, and mouse 106. Additional input devices may be included with computer 100. Computer 100 can be implemented using any suitable computer, for example, an IBM RISC/System 6000 system, a product of International Business Machines Corporation in Armonk, N.Y., running the Advanced Interactive Executive (AIX) operating system, also a product of IBM. Although the depicted representation shows a server-type computer, other embodiments of the present invention may be implemented in other types of data processing systems, such as workstations, network computers, Web-based television set-top boxes, Internet appliances, etc. Computer 100 also preferably includes a graphical user interface that may be implemented by means of system software residing in computer readable media in operation within computer 100.

[0020] FIG. 1 is intended as an example and not as an architectural limitation for the present invention.

[0021] With reference now to FIG. 2, a block diagram depicts a typical organization of internal components in a data processing system. Data processing system 200 employs a variety of bus structures and protocols. Although the depicted example employs a PCI bus, an ISA bus, and a 6XX bus, other bus architectures and protocols may be used.

[0022] Processor card 201 contains processor 202 and L2 cache 203 that are connected to 6XX bus 205. System 200 may contain a plurality of processor cards. Processor card 206 contains processor 207 and L2 cache 208.

[0023] 6XX bus 205 supports system planar 210 that contains 6XX bridge 211 and memory controller 212 that supports memory card 213. Memory card 213 contains local memory 214 consisting of a plurality of dual in-line memory modules (DIMMs) 215 and 216.

[0024] 6XX bridge 211 connects to PCI bridges 220 and 221 via system bus 222. PCI bridges 220 and 221 are contained on native I/O (NIO) planar 223 which supports a variety of I/O components and interfaces. PCI bridge 221 provides connections for external data streams through network adapter 224 and a number of card slots 225-226 via PCI bus 227. PCI bridge 220 connects a variety of I/O devices via PCI bus 228. Hard disk 229 may be connected to SCSI host adapter 230, which is connected to PCI bus 228. Graphics adapter 231 may also be connected to PCI bus 228 as depicted, either directly or indirectly.

[0025] ISA bridge 232 connects to PCI bridge 220 via PCI bus 228. ISA bridge 232 provides interconnection capabilities through NIO controller 233 via ISA bus 234, such as serial connections 235 and 236. Floppy drive connection 237 provides removable storage. Keyboard connection 238 and mouse connection 239 allow data processing system 200 to accept input data from a user. Non-volatile RAM (NVRAM) 240 provides non-volatile memory for preserving certain types of data from system disruptions or system failures, such as power supply problems. System firmware 241 is also connected to ISA bus 234 and controls the initial BIOS. Service processor 244 is connected to ISA bus 234 and provides functionality for system diagnostics or system servicing.

[0026] Service processor 244 detects errors and passes information to the operating system. The source of the errors may or may not be known to a reasonable certainty at the time that the error is detected. The operating system may merely log the errors or may otherwise process reported errors.

[0027] Those of ordinary skill in the art will appreciate that the hardware in FIG. 2 may vary depending on the system implementation. For example, the system may have more processors, and other peripheral devices may be used in addition to or in place of the hardware depicted in FIG. 2. The depicted examples are not meant to imply architectural limitations with respect to the present invention.

[0028] With reference now to FIG. 3, a diagram depicts a communications architecture for data processing systems that participate in the SAN problem determination methodology implemented in accordance with a preferred embodiment of the present invention. Network 300 comprises a set of computers, switches, and storage devices that may or may not participate in the communications architectures.

[0029] The Communications Architecture (CA) comprises all SAN-connected elements capable of communicating any or all of the information defined in a SAN Problem Determination Information Table (SPDIT), which is described in more detail further below.

[0030] Each SAN connected element participating in the CA is called a CA Element (CAE). Any element not participating in the CA is called a CA Non-participant (CAN). These elements are distinguished because they both participate in the SAN topology and thereby the problem determination (PD) capabilities of the system. Windows NT™ server 302, mainframe computer 304, Unix™ server 306, and Linux™ server 308 are computers that participate in the CA and are thus CAEs. Windows NT™ server 302, mainframe computer 304, and Unix

[0031] server 306 are also host computers that may support various clients, which may require access to the storage devices. Each of computers 302-306 has a Host Bus Attach (HBA), which is a type of network adapter for FC hosts. FC switches 311-313 are CAEs, and some of the storage devices are also CAEs. In the example, shared RAIDs (Redundant Array of Independent Disks) 321-323 and shared tape 324 are CAEs, while shared tape 325 is a CAN.

[0032] The CA can communicate via the FC switching fabric via in-band communication links 341-352 using the TCP/IP protocol and/or via an out-of-band TCP/IP communication network on communication links 331-334 that all SAN elements share. It should be noted that the communication links depicted in FIG. 3 may be logical connections that share a single physical connection. Alternatively, the devices may be connected by more than one physical communication link.

[0033] The protocols used by the CA to issue and/or collect information are defined to be both SNMP/MIB (Simple Network Management Protocol/Management Information Base, an SNMP structure that describes the particular device being monitored) and native FC based. The use of these two protocols allows both device/host-specific and SAN-specific information to be collected and subsequently used for end-to-end problem determination.

[0034] CA Managers (CAMs) are special CAEs in which the end-to-end PD capabilities of the system reside. The SPDIT resides in the CAM and every CAE is automatically registered with a CAM (via native FC and/or SNMP services). CAEs are those elements that successfully register, and CANs are those elements that cannot register with the CAM but are known to be present via the SAN topology discovery process, which is discussed in more detail further below. CAMs support any FC Extended Link Services (ELS) that are relevant to end-to-end problem determination.

[0035] CAMs may be categorized as a primary or active CAM and secondary or inactive CAMs. CAMs are highly available elements that replicate SPDIT and registration information. For example, a secondary CAM and a primary CAM may share a heartbeat signal so that a secondary CAM, operating in a redundant manner, may assume the duties of the primary CAM if the primary CAM appears to have failed by not responding to the heartbeat signal. The problem determination interface to the CAM is comprised of a SAN PD Application Programming Interface (SAN PD API). The SAN PD API defines the communication interface between the CAM and any other operating environment that can read CAM information or status.

[0036] With reference now to FIG. 4, a table depicts a SAN Problem Determination Information Table (SPDIT) in accordance with a preferred embodiment of the present invention. The SPDIT is comprised of all known products/elements and the information types that can be communicated on the CA. The format of the SPDIT may vary depending upon the number of devices in the CA, the type of products that are supported, the information associated with the devices, etc. For example, the SPDIT would contain information concerning each device shown in FIG. 3.

[0037] SPDIT 400 may contain the following record entries: vendor attribute 401, product identifier 402, info type 403, and description 404. Each record in SPDIT 400 contains data for these record entries. Vendor attribute 401 contains the manufacturer of a particular device on the CA. Product identifier 402 contains vendor-assigned information for identifying a particular device, such as model type, model number, product serial number, etc.

[0038] Information type 403 contains data related to the type of communication links supported by the device, the format of error conditions or error definitions supported by the device, etc. Description attribute 404 provides information about the type of error information that should be expected by the product. For example, if the description attribute record only contains an indication that the product is ELS Registered Link Incident Record (RLIR) compatible, then a CAM-related process would not expect to receive out-of-band MIBs for the product.

[0039] The SPDIT will generally contain all information used to indicate status/error conditions by SAN capable peripherals, hosts, and switches. This would include native FC link and extended link error definitions, and MIB definitions. These definitions can include field replaceable unit (FRU) component information, which can be located in a MIB or embedded in the error reporting protocol and can be used to determine the granularity to which failing components can be isolated.

[0040] As noted previously, CAMs are special CAs in which the end-to-end PD capabilities of the system reside, including the SPDIT. The CAM initialization process includes the discovery and registration of all FC nodes connected to both the in-band SAN and out-of-band network. The CAM initialization process uses FC in-band and CA out-of-band (via SNMP) discovery/registration processes. This process provides a topology map (TM) of the SAN that includes all registered and non-registered SAN connected elements along with knowledge of the element types (hosts, peripherals, switches), explicit connections/paths, and their relevant vendor and SPDIT information.

[0041] With reference now to FIG. 5A, a simplified network topology diagram for the SAN is shown. FC switch 501 contains ports 511-513 providing connection points between FC switch 501 and CAEs 521-523, also labeled CAE A, CAE B, and CAE C. From the perspective of the CA, FC switch ports 511-513 are CAEs because the ports are capable of failing or generating errors and could be replaced after being properly diagnosed as a source of errors.

[0042] With reference now to FIG. 5B, a table provides a topology map for the SAN shown in FIG. 5A. The TM is represented as a two-dimensional table with both the left column and the top row containing the SAN elements, both CAE and CAN devices, connected to the switch, such as FC switch 501 in FIG. 5A. The diagonal cells contain all the SPDIT/type information about the corresponding element and the switch port to which it is connected. The other cells contain the directional paths between the elements. For example, the table shows the direction path between CAE A and CAE C using the path between ports 3 and 1. Multiple paths are possible. The topology and registration discovery process are periodically repeated to ensure that the TM is current. The CAM will also register with any SAN elements providing Extended Link Services that can be used for PD.

[0043] With reference now to FIG. 6, a diagram depicts a SAN Diagnostic Table for a SAN. The TM of a SAN is used to create a SAN Diagnostic Table (SDT) that is used for First Error Data Collection (FEDC) and Real-time Diagnostic Analysis (RDA). The SDT shown in FIG. 6 is similar to the TM shown in FIG. 5B except that it contains an extra row for each switch/fabric element.

[0044] The diagonal SDT cells are used to hold the errors reported by the CAE corresponding to its row/column, including switch ports. Each point in a path, i.e. SDT cell, represents another SAN-connected element. Each cell contains the information collected in the TM so that specific product behaviors are known and proper diagnostic decisions can be made. For example, diagnostic queries may be presented, such as whether it is more likely that a storage device is causing a link error versus a Host Bus Attach (HBA) if out-of-band SCSI device errors accompany in-band HBA FC link errors.

[0045] The exemplary error information contained in FIG. 6 illustrates the utility of RDA using the SDT. Row 1 indicates that CAE A has reported an in-band FC link timeout. Row 3 indicates an out-of-band hardware controller error on CAE C. These two errors are related because they occurred in the same time frame, as shown by the timestamps associated with the error information. Row 5 indicates that an in-band FC link error has occurred, but given the stored timestamp, the error in row 5 is unrelated to the previous two. Therefore, the table depicts two separate problems: the first is related to a controller hardware failure in CAE C, and the second is a FC Link failure on CAE 2 in the FC Switch.

[0046] With reference now to FIG. 7, a table depicts the weightings to be used in real-time diagnostic analysis for various errors. The RDA algorithms traverse error reporting elements of the SDT whenever an FEDC event occurs in order to determine the appropriate response. The RDA uses weighted decision analysis in order to isolate the failing component. Two broad categories are illustrated with H=Highest, M=Middle, L=Lowest weighting.

[0047] The SDT traversal algorithms and error weightings are dynamic and would be changed to accommodate the complexity of the SAN topology and the nature of its connected elements.

[0048] The weighting table shown in FIG. 7 provides a simple illustration of the strong-to-weak weighting scale that applies to a typical SAN environment. If the SAN grows to just a few 16 port switches with its associated hosts and peripherals, the number of possible nodes that can report errors due to a single disk drive error or HBA timeout error can grow to a large number. Without global end-to-end RDA diagnostic capability, the task of isolating the failing component becomes hit-or-miss. In a multi-vendor SAN, it is common for multiple intermittent, recoverable, device errors, i.e. soft errors, to go unnoticed by the host. Eventually, the device may encounter an unrecoverable error, i.e. a hard error, that results in a system crash. The in-band and out-of-band mechanisms provided by the present invention would detect and report the recoverable errors as soon as they occur.

[0049] With reference now to FIGS. 8A-8D, flowcharts depict a process for a real-time diagnostic algorithm (RDA) for SAN end-to-end fault isolation of a single failing SAN element in accordance with a preferred embodiment of the present invention. The RDA uses two dynamic mechanisms to isolate faults:

[0050] 1. Temporal Correlation Window (TCW)—The TCW is scalar value, i.e. time range, used to constrain fault isolation searching of the SDT in the time dimension so that the probability of misdiagnosis is minimized in the time dimension.

[0051] 2. Spatial Correlation Path (SCP)—The SCP is a data structure that is used to constrain fault isolation searching in the spatial domain of the SDT so that only known system-to-subsystem associations are scrutinized and so that the probability of misdiagnosis is minimized in the spatial dimension. The SCP copies elements from the SDT during the RDA.

[0052] The goal of the RDA is to correlate all fault information received in time, location, and severity until the source of the fault is isolated with a high degree of certainty. This process terminates after a single reported fault or after a series of reported faults.

[0053] The general RDA for SAN end-to-end fault isolation of a single failing SAN element is described as follows. The process begins when a CAM initializes all connected paths in the CA into the SDT (step 801). The SDT is initialized with all connected paths, i.e., paths A→B, B→C, etc. Only those paths that should be able to make connections are entered. These paths are established by the topology mapping, such as a TM similar to the TM shown in FIG. 5. The SAN may not be fully connected in order to zone off certain connections that should not be able to connect. For example, certain hosts might be restricted to storing and retrieving data on particular storage devices. A system administrator may only allow NT hosts to store data on a particular device so that a mainframe does not have the ability to corrupt or destroy the NT data.

[0054] The process continues with a CAM initializing the TCW and SCP for the SAN (step 802). The TCW is a time window and requires a time value, usually on the order from seconds to minutes. The SCP contains all sets of paths chosen from the SDT. These paths reflect known host-to-storage, host-to-host, and storage-to-storage associations that are established by the topology mapping. Again, it should be noted that a secondary CAM maintains a replica of the data structures and values that are stored in the primary CAM.

[0055] The CAM then receives a new error (step 803) and processes the error using the RDA (step 804). A determination is then made as to whether the RDA process is being terminated (step 805), and if not, the process then loops back to step 803 to receive and process more errors. If so, then the process of initializing for SAN end-to-end fault isolation is complete.

[0056] Referring now to FIG. 8B, a process depicts the processing of a new error, such as step 804 in FIG. 8A, in more detail. The process begins by receiving a new error (step 810), and the SDT is updated to indicate the component reporting the error, the time the error occurred, and the severity (high, medium, low) of the error (step 811). A determination is made as to whether the error is a high severity error (step 812). If so, then this error is immediately reported as a fault that requires maintenance (step 813). The SPDIT is then interrogated to determine if the reported error is associated with a specific part that should be replaced (step 814). If not, then the processing of the high severity error is complete. If so, then the failing component is called out to be replaced (step 815), and the processing of the high severity error is complete.

[0057] If the error is not a high severity error, then a determination is made as to whether the error is a medium or low severity error (step 816). If so, then the low/medium severity error is processed (step 817), and the error processing is complete.

[0058] If the error is neither a high severity error or a low/medium severity error, then the error severity is determined to be faulty and the error ignored (step 818).

[0059] Referring now to FIG. 8C, a process depicts the processing of a new low/medium severity error, such as step 817 in FIG. 8B, in more detail. The SCP is used to determine the paths that can be affected by the reported error. Each of the SDT cells for the elements in these paths, including the element reporting the new error, are interrogated in turn for previous occurrences of errors (step 820), and it is determined if the occurrence of a previous error is spatially related to the current error (step 821). The interrogation then uses the TCW in order to determine if the occurrence of a previous error is related to the current error in time as well as space (step 822). If the previous errors are temporally and spatially related, then the errors are stored into the SCP (step 823). After the interrogation is finished, the SCP contains the mapping of all errors on the appropriate paths in the SDT that occur within the time constraint.

[0060] The manner in which the data structure for the SCP is organized and used may vary depending upon system implementation. For example, the elements from the SDT may be copied into the SCP, and as errors are determined not to be related in space or in time, the elements may be deleted from the SCP.

[0061] The algorithm must make an error correlation/severity assessment in order to isolate the location of the failing component. Referring now to FIG. 8D, a flowchart depicts several possible cases for failing components associated with low/medium severity errors.

[0062] The process begins with a determination of whether all errors emanate from the current element which generated the newly received error (step 830). If so, then a determination is made as to whether two or more errors are in the SCP (step 831). If not, then the processing of the current error is complete. If so, then the current element is indicated to require maintenance (step 832). The SPDIT is then interrogated to determine if the reported error is associated with a specific part that should be replaced (step 833). If so, then the failing component is called out to be replaced (step 834), and the newly received, low/medium severity error has been processed.

[0063] If all errors do not emanate from the current element, then a determination is made as to whether all (two or more) errors are contained in a single path (step 835). In this case, any element in the path may be the root cause of the reported errors, and device hardware related errors take precedence over link related or timeout related errors. A determination is made as to whether the errors contain a device hardware error (step 836). If a device hardware error is found, in a manner similar to steps 832-834 described above, the associated element is indicated to require maintenance, the SPDIT is then interrogated to determine if the reported error is associated with a specific part that should be replaced, and if so, the failing component is called out to be replaced.

[0064] If the errors on the single path do not contain a device hardware error, then only link or timeout errors are being reported. This situation can lead to degradation in performance and eventual failure of the link. In this case, the algorithm looks for the element that is reporting the error first (step 837), i.e., the first error takes precedence and the others are assumed to be related to the first occurrence. Once the element that is originating the chain of errors is found, in a manner similar to steps 832-834 described above, the associated element is indicated to require maintenance, the SPDIT is then interrogated to determine if the reported error is associated with a specific part that should be replaced, and if so, the failing component is called out to be replaced.

[0065] If two or more errors are not contained in a single path, then two or more errors are occurring on multiple paths. A determination is made as to whether there are any common elements on the paths of the multiple errors (step 838), and if so, then this case requires isolating the common element(s) on these paths (step 839) and performing an error correlation/severity assessment.

[0066] The common elements can either be SAN endpoint elements and/or SAN fabric elements. A determination is made as to whether a SAN endpoint or fabric element is the only common element (step 840). If so, then in a manner similar to steps 832-834 described above, this common element is indicated as failing and maintenance is required. The SPDIT is then interrogated to determine if the reported error is associated with a specific part that should be replaced. If so, the failing component is called out to be replaced.

[0067] Otherwise, if a SAN endpoint or fabric element is not the only common element, then both a SAN endpoint and a SAN fabric element are common. This situation is now equivalent to the result from the determination in step 835, and the process branches to step 836 for further processing.

[0068] If there are two or more errors that are not contained in a single path and there are no common elements on the paths of these errors, then each of the multiple errors are run through the real-time diagnostic algorithm (RDA) separately (step 841). This rare but possible scenario may occur when more than one error has been received within the TCW and the errors originate from separately failing components. At this point, the error process may branch back to step 804 to process each error as if each error were a newly received error.

[0069] The advantages of the present invention are apparent in view of the detailed description of the invention provided above. A SAN Diagnostic Table is created using the SAN topology, native Fibre Channel services, and vendor specific information. The present invention supports both FC native in-band and host/device specific out-of-band status/error data collection for SAN problem determination. A real-time diagnostic algorithm may then traverse the SAN Diagnostic Table to isolate a failing SAN component. The methodology is advantageous because it may be implemented on host operating environments such that special access to management terminals or device diagnostics are not required to isolate failing components. In addition, the methodology is platform-independent, and it supports both FC native in-band and host/device-specific out-of-band status/error data collection for SAN problem determination.

[0070] It is important to note that while the present invention has been described in the context of a fully functioning data processing system, those of ordinary skill in the art will appreciate that the processes of the present invention are capable of being distributed in the form of a computer readable medium of instructions and a variety of forms and that the present invention applies equally regardless of the particular type of signal bearing media actually used to carry out the distribution. Examples of computer readable media include recordable-type media such a floppy disc, a hard disk drive, a RAM, and CD-ROMs and transmission-type media such as digital and analog communications links.

[0071] The description of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiment was chosen and described in order to best explain the principles of the invention, the practical application, and to enable others of ordinary skill in the art to understand the invention for various embodiments with various modifications as are suited to the particular use contemplated.