Error injection apparatus and method

[0002] This application is related to U.S. patent applications entitled “On-line Switch Diagnostics” Ser. No. 09/393,587 and “Switch Adapter Testing,” Ser. No. 09/393,582 filed on even date, which are assigned to the assignee of the present patent application and are incorporated herein by reference.

[0003] The present invention relates generally to packet-switched computer networks, and specifically to methods and apparatus for testing and diagnosing malfunctions in such networks.

[0004] Packet-switched, source-routing computer networks are used in a growing range of applications. Such networks link multiple computer processors, or nodes, via multiple switches. Typically, a packet of data sent from one of the nodes to another passes through a number of different switches. Each switch along the way reads routing information, which is commonly contained in a header of the data packet, and passes the packet on to the next switch along the way, or to the destination node. Typically, there are multiple different paths available through the network over which any given pair of nodes can communicate. An example of this type of network is the well-known Asynchronous Transfer Mode (ATM) network, which is used in communications between separate computers. Such networks are also used in multi- processor computers, such as the RS/6000 Scalable POWERParallel System (SP) series of computers produced by International Business Machines Corporation (Armonk, N.Y.). In the SP computer, as well as in certain other networks, successive packets in a communication stream between the nodes may be sent over different routes.

[0005] Because of the complex topology and hardware of packet-switched networks, when a fault occurs in such a network it can be difficult to identify the exact location and nature of the fault. The difficulty is exacerbated by the fact, noted above, that by their nature such networks use multiple different paths between nodes and are fault-tolerant. A network fault will typically appear not as a total breakdown (which would be relatively easy to find), but rather will present more subtle symptoms. For example, there may be a reduction in throughput between some or all of the nodes, or an increase in the number of “bad packets”—data packets whose content is corrupted and must be discarded—at one or more of the nodes.

[0006] There are few efficient tools known in the art for diagnosis of such faults. The diagnostic process is time-consuming and heavily reliant on the intuition and experience of a human system administrator (or service engineer) in deciphering and drawing conclusions from the limited information that is available. This information is typically collected in various system files, such as topology files, error logs and trace files, as are known in the art. These files may be recorded at different nodes of the network and must somehow be collated and analyzed by the administrator. Because few network administrators have the know-how to perform this sort of diagnosis, costly service calls are frequently required.

[0007] A further problem in diagnosing network faults is non-deterministic failures, which may occur only under certain conditions, and may not arise at all while the diagnostic tests are being performed. Such failures are referred to with terms such as “sporadic,” “intermittent,” “overheating,” “lightning,” “aging,” or “statics,” which generally mean only that the cause of the problem is unknown. For example, a high-speed switch or adapter may behave normally in light traffic, and break down only under certain particular stress conditions. At times the only way to find such a problem is to systematically bombard each suspect component of the network with packets from different sources, at controlled rates, gradually eliminating components from consideration until the failure is found. Such a process is difficult to automate, and may require that the network be taken off-line for an extended period. The cost of such down-time for prolonged testing and repair can be enormous. There is therefore a need for systematic methods of diagnostic testing, which can be performed while the network is on-line.

[0008] There is a similar lack of tools and techniques for systematically testing the response of switch-related network software to hardware fault conditions. Such techniques are needed particularly in software development and testing stages, to ensure that the software responds properly when faults occur. Current methods of testing use specially-designed simulation hardware, such as cables with broken pins, together with debugging clauses that can be activated in the software itself and dedicated debugging fields in associated data structures. The fault situations created by such methods, however, are limited to a small range of scenarios, which are for the most part different from the real hardware faults that occur in actual networks. Similarly, the software used in debugging mode for fault simulation is different from the actual software product that will be used in the field. Moreover, these testing tools are incapable of simulating the type of transient, non-deterministic failures described above. They do not allow errors to be injected and altered on the fly during a simulation.

[0009] It is an object of some aspects of the present invention to provide improved methods for fault simulation and diagnostics in packet-switched data networks.

[0010] It is a further object of some aspects of the present invention to provide apparatus and methods for systematically injecting errors into a data network, for purposes of debugging and diagnostics.

[0011] Preferred embodiments of the present invention operate in the context of a packet data network, which comprises a plurality of nodes, or processors, mutually coupled by a plurality of switches, such that typically any one of the nodes can communicate with any other one of the nodes, preferably over multiple links. Each of the nodes is coupled to a respective port of one of the switches by a switch adapter, which performs data link functions, as are known in the art, with respect to each data packet sent or received through the network by the node. One of the nodes is a primary node, which manages the configuration of elements of the network, such as the other nodes and switches in the network.

[0012] In preferred embodiments of the present invention, the primary node controls testing and diagnosis of elements of the network in real time, while the network is on-line, or at least with minimal interruption of on-line operation, by appropriately setting parameters of the nodes and switches. The testing preferably includes diagnostic testing to locate suspected faults in the switches and switch adapters. Additionally or alternatively, for the purposes of testing, errors are intentionally injected into the network so as to simulate the response of the network elements to faults that may occur.

[0013] In some preferred embodiments of the present invention, one of the nodes of the network is used as an error injector, and is isolated for this purpose from the remaining nodes in the network. While the remaining nodes and switches carry out their normal functions on-line, the error injector injects errors into the network in order to test by simulation the response of elements of the network to actual errors that may occur. Generally, if the network is operating properly, the errors will be rejected and, preferably, reported to and logged by the primary node, while normal functions continue without substantial interruption.

[0014] In some of these preferred embodiments, the error injector is used to simulate the effect of a faulty switch adapter, by injecting bad packets into the network.

[0015] In other preferred embodiments, the error injector is used to simulate switch faults, such as failures in the ports or central queue or in send and/or receive logic of the switch. Preferably, the error injector sends a command packet to one of the switches, such as a packet instructing the switch to reset its queue or initialize operation, while the switch is in the midst of its normal functions. The error injector prepares the command packet in such a manner that it appears to have originated from the primary node, which is the only node ordinarily entitled to issue such commands. The primary mode generally issues a reset or initialization command when it is informed that a fault or error has occurred in a particular switch. Alternatively or additionally, the error injector sends an error reporting packet to the primary node, prepared so as to appear to have originated from one of the switches in the network.

[0016] There is therefore provided, in accordance with a preferred embodiment of the present invention, in a computer network system that includes a multiplicity of nodes interconnected by a network of switches, wherein data are normally conveyed in the network according to predetermined conventions, a method for simulation testing of the system, including:

[0017] selecting one of the nodes to serve as an error injector;

[0018] injecting data into the network from the error injector node in a manner that violates the predetermined conventions, so as to simulate an error condition in the system; and

[0019] observing operation of the system following the injection of the data so as to evaluate a response of the system to the error condition.

[0020] Preferably, selecting the one of the nodes includes taking the error injector node off-line, while normal data transmission continues among other nodes and switches in the system.

[0021] Further preferably, injecting the data includes sending a corrupted data packet to one of the nodes, and observing the operation of the system includes ascertaining that the corrupted packet has been detected. Most preferably, the nodes are linked to the network by respective data link adapters, and sending the corrupted data packet includes transmitting to the one of the nodes a packet having a corrupted data link header, so as to ascertain that the respective data link adapter detects the corrupted header.

[0022] In a preferred embodiment, injecting the data includes sending a command to one of the switches, wherein the command is of a type that is normally sent in response to an error in the network, and wherein sending the command includes choosing a command from the group of commands consisting of an initialization command, a reset command and a port disable command.

[0023] In another preferred embodiment, the system includes a primary node, which normally receives service messages from the switches in the network, and injecting the data includes sending data to the primary node having the form of a service message from one of the switches. Preferably, sending the data includes sending an error report. Alternatively or additionally, injecting the data includes sending a command to the one of the switches that causes the switch to convey service messages to the error injector node, rather than to the primary node.

[0024] There is further provided, in accordance with a preferred embodiment of the present invention, a manageable computer network system, including:

[0025] a network of switches, among which data are normally conveyed in the network according to predetermined conventions; and

[0026] a multiplicity of nodes interconnected by the switches, one of which nodes is selected to serve as an error injector, which injects data into the network in a manner that violates the predetermined conventions, so as to simulate an error condition in the system in order that a response of the system to the error condition can be observed.

[0027] Preferably, the error injector node is taken off-line while normal data transmission continues among other nodes and switches in the system.

[0028] In a preferred embodiment, the error injector node sends a corrupted data packet to a target node, so that it can be ascertained that the corrupted packet has been detected by the target node. Preferably, the system includes data link adapters, which link the nodes to the switches in the network, wherein the corrupted data packet has a corrupted data link header, which is detected by one of the data link adapters that is associated with the target node.

[0029] In another preferred embodiment, the error injector node sends a command to one of the switches of a type that is normally made in response to an error in the network.

[0030] In still another preferred embodiment, the multiplicity of nodes includes a primary node, which normally receives service messages from the switches in the network, and the error injector node sends data to the primary node having the form of a service message from one of the switches.

[0031] There is moreover provided, in accordance with a preferred embodiment of the present invention, a computer software product for simulation testing of a computer network system including a network of switches linking a plurality of processor nodes, wherein data are normally conveyed in the network according to predetermined conventions, the product including computer-readable code, which is read by one of the nodes selected to serve as an error injector node among a multiplicity of nodes coupled to the network and causes the error injector computer to inject data into the network in a manner that violates the predetermined conventions, so as to simulate an error condition in the system, wherein operation of the system following the injection of the data is observed in order to evaluate a response of the system to the error condition.

[0032] The present invention will be more fully understood from the following detailed description of the preferred embodiments thereof, taken together with the drawings in which:

[0033] FIG. 1 is a schematic block diagram of a packet-switched data network including elements for performing automated testing and diagnostics, in accordance with a preferred embodiment of the present invention;

[0034] FIG. 2 is a block diagram that schematically illustrates a data packet transmitted in the network of FIG. 1;

[0035] FIG. 3 is a flow chart that schematically illustrates a method for simulating errors in the network of FIG. 1, in accordance with a preferred embodiment of the present invention;

[0036] FIG. 4 is a flow chart that schematically illustrates a method for injecting errors into the network of FIG. 1, in accordance with a preferred embodiment of the present invention;

[0037] FIG. 5 is a schematic block diagram illustrating an erroneous data packet injected into the network using the method of FIG. 4;

[0038] FIG. 6 is a flow chart that schematically illustrates a method for injecting errors into the network of FIG. 1, in accordance with another preferred embodiment of the present invention;

[0039] FIG. 7 is a schematic block diagram illustrating a network system management architecture, useful in implementing preferred embodiments of the present invention;

[0040] FIG. 8 is a schematic block diagram illustrating an object-oriented framework for advanced network management, in accordance with a preferred embodiment of the present invention;

[0041] FIG. 9 is a schematic block diagram illustrating a finite state machine object model used in network diagnostics, in accordance with a preferred embodiment of the present invention;

[0042] FIG. 10 is a schematic diagram illustrating interaction of the objects in the model of FIG. 9;

[0043] FIG. 11 is a schematic block diagram illustrating a client-server network diagnostic architecture, in accordance with a preferred embodiment of the present invention; and

[0044] FIG. 12 is a schematic block diagram illustrating a hierarchy of classes used in the architecture of FIG. 11, in accordance with a preferred embodiment of the present invention.

[0045] FIG. 1 is a schematic block diagram of a computer system 20, built around a manageable packet-switched network 21, in accordance with a preferred embodiment of the present invention. System 20 is based generally on the IBM RS/6000 SP computer system, described in the Background of the Invention. In the example shown in FIG. 1, the system comprises five hundred twelve nodes 22, 24, . . . , and three hundred eighty-four switches, including one hundred twenty-eight node switches 29 and two hundred fifty-six intermediate switches 32 and 34. The architecture of network 21 is described in greater detail hereinbelow, to aid in understanding the application of preferred embodiments of the present invention in testing network performance and diagnosing network faults. It will be understood, however, that the present invention is in no way limited to this particular context. Rather, the principles of the present invention are applicable to switched networks of various types and topologies, including (but not limited to) both multi-processor computer systems, such as system 20, and data communication networks. Such networks include, for example, switched Ethernet, Fast Ethernet, Token Ring, FDDI and ATM networks, as are known in the art.

[0046] In system 20, nodes 22, 24, . . . , generally comprise processors, which are assigned various tasks in the system. Alternatively, the nodes may comprise other elements, such as gateways (or routers) or input/output extensions, as are known in the art. Each node is connected to a port on one of node switches 32 by a switch adapter 26, which performs data link functions, as are known in the art. Generally speaking, each switch adapter routes data packets from the respective node to their destination through network 21, and receives data packets addressed to the node from the network.

[0047] A primary node 66 is assigned the functions of monitoring system 20 and managing the configuration of nodes and switches therein. It is capable of setting various parameters of the switches and the other nodes in the system, as well as resetting and initializing these elements, by sending them special “service packets,” which contain service commands. The primary node monitors the system for errors and failures, by receiving error messages from the switches and other nodes, as well as by noting any elements that fail to respond and by collecting statistical measures of system performance. Preferably, system 20 also provides a user interface to an operator of the system via a terminal 68.

[0048] In some preferred embodiments of the present invention, one of the nodes, labeled N_ERR, is assigned to function as an error injector 36 for the purposes of diagnostics and testing. This function is described further hereinbelow.

[0049] Switches 29, 32 and 34 typically comprise custom, high-speed integrated circuit devices, preferably having eight ports each. Node switches 29 are arranged on thirty-two node switch boards 28, labeled NSB 1, NSB 2, . . . , NSB 32 in the figure. Intermediate switches 32 are similarly arranged on nodes switch boards 28, while intermediate switches 34 are arranged on sixteen intermediate switch boards 30, labeled ISB 1, ISB 2, . . . , ISB 16. Each of node switches 29 in the left-hand column of boards 28 (in the view shown in FIG. 1) is connected directly both to switch adapters 26 of four different nodes and to the four intermediate switches 32 in the right-hand column of the same board 28.

[0050] Based on the architecture shown in FIG. 1 (in which only a small subset of all the possible links in network 21 are illustrated), it will be observed that each node 22, 24, . . . , in system 20 can communicate with any other node over four different paths, or routes. Typically, switch adapters 26 are programmed so that successive data packets sent between any two of the nodes alternate in a “round robin” between the available paths, in order to level the loading of the different switches in the network. Thus, packets sent from node 22 (marked N1) to node 24 (N2) will pass in alternation via the uppermost switches on the intermediate switch boards 30 that are marked ISB 1, ISB 2 and ISB 3, along with ISB 4 (which is omitted in the figure, together with ISBs 5-15). Nodes 22 and 24 are linked via ISB 1, ISB 2 and ISB 3 over paths 31, 33 and 35 respectively. This sort of path alternation is well known in high-speed packet-switched networks.

[0051] FIG. 2 is a block diagram that schematically illustrates a data packet 40 transmitted in network 21, from node 22 to node 24, for example. This packet structure, which is generally known in the art, is presented here to aid in understanding the testing and diagnostic techniques described hereinbelow. A payload 44 includes data 46 provided by node 22, along with an error-correcting code 48, typically a CRC code, as is known in the art. Switch adapter 26 of node 22 adds on a header 42, which includes a route 50, a source address 52 and a destination address 54. Thus, in the example shown in FIG. 1 and described hereinabove, source address 52 will identify node 22 (N1); destination address 54 will identify node 24 (N2); and the route will alternate among paths 31, 33, 35, . . . Optionally, the header contains other information, as well.

[0052] Preferably, switch adapter 26 selects the paths from a route table, which is pre-programmed based on the known configuration of network 21 and stored in a memory (not shown) associated with the adapter. The table may be re-programmed from primary node 66. For any given node, the route table typically has the general form shown below in Table I. This table shows only a few of the 512 rows of the actual routing table, with pro forma table entries for purposes of illustration.

[0053] The switch adapter maintains a modulo-4 count of packets that it sends, and selects the packet route successively according to the packet count.

[0054] FIG. 4 is a flow chart that schematically illustrates a generic method for injection of errors into network 21, in accordance with a preferred embodiment of the present invention. This method makes use of error injector node (N_ERR) 36, shown in FIG. 1, which is isolated (“fenced”) from the normal program and message flow in network 21, as illustrated schematically by a dashed line 38 in the FIG. 1. Such fencing of node 36 is accomplished by giving appropriate system configuration instructions, through primary node 66. Program flow proceeds normally among the remaining nodes of system 20.

[0055] The main function of error injector node 36 is to inject simulated bad (corrupted) data packets and error reporting packets into system 20. In normal operation, hardware faults in the switch network cause packets of these types to be generated. Error injection in accordance with the method of FIG. 10 is used to simulate the effect of such hardware errors, typically for purposes of software testing and evaluation. The method enables a wide range of errors, occurring at substantially any desired location in the network, to be simulated quickly and easily, without the necessity of stopping the system to install “faulty” hardware.

[0056] As shown in FIG. 3, after error injector node 36 has been selected and isolated, it saves the current state of system 20, for restoration when the simulation is done. Node 36 then introduces a change or changes in the system, depending on the type of test that is planned, at a step 110. This step may include, for example, changing routes programmed in a route table of switch adapter (SA_ERR) 37 serving node 36, or resetting and changing internal registers in one or more of the switches in network 21. These examples are described in greater detail hereinbelow. Normal operation of system 20 is then initiated. During operation, node 36 uses the system change it has prepared to inject errors into the system, in a step 112. The response of the system to these errors is noted and recorded for subsequent evaluation, in order to determine whether the system was able to function properly despite the presence of (simulated) hardware errors. Upon conclusion of the testing, the error injection node restores the normal state of the system that it had earlier stored.

[0057] FIG. 4 is a flow chart that schematically illustrates a specific application of the method of FIG. 3, for testing the response of system 20 to the presence of bad packets, in accordance with a preferred embodiment of the present invention. Ordinarily, when one of switch adapters 26 (FIG. 1) receives a bad packet, it should generate an interrupt to management software running on its corresponding node and report the error. An error report of this sort will be generated, for example, when an adapter belonging to a given node receives a packet whose header 42 (FIG. 2) indicates that the packet was destined for a different node. The adapter should then go on with its normal program flow. The error reports are logged by the node, which may then take remedial action. If necessary, the adapter will be reset or disabled.

[0058] Let us assume that the error injector node N_ERRis to test the response of system 20 to transmission of a bad packet from node N1 to node N2 (FIG. 1). To prepare for the test, error injector node 36 sets its switch adapter 37 so that the adapter identifies itself as the switch adapter of node N2, rather that with its true identification as belonging to node 36. Furthermore, node 36 prepares a route table, of the general form shown hereinabove in Table I, which includes a route to node N1 that is entered in the table not in the row having N1 as its destination node index, but rather in the row of a different node, say Nx. This “illegal” route table is downloaded to switch adapter 37, and normal system operation is initiated.

[0059] FIG. 5 is a schematic block diagram illustrating an erroneous data packet 120 sent from error injector node 36 to receiving node N1, in accordance with the method of FIG. 4. To launch this packet, node 36 prepares payload 44 and instructs switch adapter 37 to send the packet to node Nx. In preparing header 42 of packet 120, adapter 37 writes N2 in source field 52, since this is the (erroneous) identification with which the adapter has been programmed, and Nx in destination field 54. The adapter refers to its route table, and finds the route from N_ERRto N1 opposite the index Nx. This route is entered in route field 50 and is used by network 21 to route packet 120 to node N1.

[0060] When packet 120 arrives at node N1, the switch adapter checks destination field 54, which contains the value Nx, against its own identification—N1—and determines that the packet is bad. The switch adapter at this point should generate an error interrupt, as described hereinabove. It is possible in this way to observe the behavior of all software layers and tasks running on system 20, to determine whether the bad packet was detected, whether the error interrupt was handled properly, and then whether the system recovered from the error and resumed normal operation. This is exactly the sequence of events that should occur if a bad packet is received as the result of a real, rather than simulated, hardware error. The simulation testing provided in this case using error injection node 36 is thus a completely realistic model of actual error situations.

[0061] In another preferred embodiment of the present invention, node 36 sends service packets to various switches in network 21. Ordinarily, such service packets are sent exclusively by primary node 66, in accordance with network management software running on system 20. Such packets may include, for example, commands to a given switch to initialize operational settings, to reset its central queue or switching logic, or to enable or disable certain of its ports. These commands are used by the system to recover from actual errors when they occur.

[0062] To prepare for sending such service packets to the switches, at step 110 of the method shown in FIG. 3, error injection node 36 programs its switch adapter 37 to identify itself as belonging to primary node 66. This false identification is necessary in order that the switches in network 21 recognize service packets sent by the error injection node as legitimate. Then, at step 112, node 36 sends a bogus service packet to one or more of the switches. For example, if a command is sent to one of the switches to reset its central queue, that switch will discard all packets currently in the queue. If any packet is in the course of being sent, it will be corrupted, and should be identified as such by the node that receives it. Recovery of the system from the loss of the discarded packets is also tested in this way. As another example, node 36 may send a service packet commanding a switch to disable certain of its ports, so that a bad switch error will be simulated.

[0063] In still another preferred embodiment of the present invention, error injection node 36 send error/status reporting packets to primary node 66. These packets are ordinarily sent to the primary node by the switches in network 21 when an error actually occurs or, alternatively, when they are queried by the primary node. In this case, in step 110, error injection node 36 programs adapter 37 so that it sends packets to primary node 66 that have one of the network switches as their source. Node 36 then sends error packets to the primary node at step 112. The error packets preferably contain all of the information that would be contained in a real error packet, including the proper packet source in the header and a payload reporting the type and cause of the error. The primary node logs and analyzes the simulated error that has been reported. If it considers corrective action to be necessary, such as a switch reset or port disablement, the primary node will send an appropriate service packet to the switch. Thus, this method tests the response of network management software running on the primary node to switch error reports. It also allows the behavior of all software layers and tasks running on system 20 to be observed under the conditions created by the remedial action instituted by the primary node under these circumstances.

[0064] FIG. 6 is a flow chart that schematically illustrates an application of the general method of error injection testing based on conveying false error packets to primary node 66, in accordance with another preferred embodiment of the present invention. In this case, at step 110, error injection node (N_ERR) 36 sends an initialization packet to one of the switches in network 21, say one of switches 34 (FIG. 1). The packet includes routing information in which the route to the primary node leads to node 36, instead of to node 66. As a result, error/status packets sent by the switch and intended for delivery to the primary node will instead be conveyed to the error injection node. The error injection node may then send error/status packets, formatted as though they were sent by switch 34, to primary node 66.

[0065] In operation of system 20, when primary node 66 sends a command to switch 34, following initialization of the switch by error injection node 36, the switch will perform the command. But instead of reporting back to the primary node, the switch will send an error/status packet to the error injection node, since the routing information of the switch was modified by the initialization packet sent by the error injection node. The error injection node receives the error/status packet and, at step 112, modifies the packet and sends the modified packet back to primary node 66. The modified content of the packet indicates to the primary node that the switch has not responded as desired, and the primary node may take further action accordingly to deal with the simulated error. In this case, the cycle of communication between the primary node, the switch and the error injection node can continue through multiple iterations, so as to simulate a range of different fault scenarios.

[0066] Although preferred embodiments are described hereinabove on the basis of a particular network configuration and certain specific combinations of error simulation tests and features, it will be appreciated that the principles of the present invention may be applied in a wide range of different computer networks and systems. The various testing and diagnostic techniques described herein may be used singly or in combination with other diagnostic methods, such as those described in the above-mentioned U.S. patent applications entitled “On-line Switch Diagnostics” and “Switch Adapter Testing. The capability of carrying out these techniques is preferably incorporated in firmware associated with a manageable switch network. Alternatively, software for carrying out such techniques may be supplied as a separate product for installation on the network, wherein the software code may be provided on tangible media, such as disks, or conveyed over a communications link.

[0067] It will thus be appreciated that the preferred embodiments described above are cited by way of example, and the full scope of the invention is limited only by the claims.

[0068] The following appendices describe methods and tools useful in implementation of the preferred embodiments described hereinabove in the context of the above-mentioned IBM SP computer system.

[0069] Networks have become a key component of the corporate infrastructure. Managing the growing complexity of networks is a major challenge imposing the necessity for better management tools.

[0070] This section presents an object-oriented framework for dynamic extension of network management software that goes beyond basic capabilities. The framework provides a common management interface for implementing advanced network management applications on top of an existing basic management system. It also defines an object model for such applications and the implementation of associated common logic.

[0071] The framework presented here was used in the implementation of switch diagnostics, as described hereinabove—a set of tools for isolating the cause of faults in a switched network. These diagnostic tools were implemented for IBM's Scalable POWERParallel System (SP) on top of the existing Communication Subsystem (CSS) switch network management software.

[0072] The size and complexity of corporate networks is growing fast. Modern networks consist of thousands of devices and multiple logical and physical layers. The availability of powerful management tools can be a critical factor in the success or failure of this new wave of networks.

[0073] Network Management includes fault management, configuration management, accounting, performance management, security management, and other areas. This document concentrates on the area of fault management: the process of locating and correcting network problems (faults). A lot of work has been done in order to facilitate building network management systems. Major management technologies include SNMP ([5]), OSI CMIP ([1], [2], [3]), CORBA ([8], [9]), DMTF DMI ([7]), JMAPI ([6]). (The bracketed numbers refer to references listed at the end of the appendix.)

[0074] Management technology specifications typically cover the communications protocols between manager and managed systems, and the information model. This includes the management information that defines requests for management operations, the results of the operations, and unsolicited reports such as alarms.

[0075] Some management technologies also define interfaces for implementing management applications within the management infrastructure according to its information model (JMAPI ([6]), DMI MI ([7])). These management interfaces define the interaction between the management application and the underlying management system. They allow the application to access management information, to invoke management operations, and to subscribe for unsolicited asynchronous events (e.g., error reports produced by network devices) to be delivered to the application by the system.

[0076] Using management interfaces has many advantages:

[0077] Independence of operating system and versions.

[0078] Consistent interface.

[0079] Compatibility with different revisions of management software.

[0080] Easy access to management information.

[0081] In a management system that exposes a well-defined management interface, management processing is no longer static, hard coded and resource-dependent; it becomes dynamically extensible to include new management services. In addition, it allows the developers of management applications to focus on their core expertise (a specific sub-domain of management) rather than spend time and energy studying the system and integrating their tools.

[0082] This appendix presents an object-oriented framework for implementing advanced network management. The framework's goal is to simplify the extension of the existing network management software which provides basic configuration, setup and fault recovery utilities. It provides a mechanism for dynamically extending the network management system by using “pluggable” components that use a common management interface.

[0083] Another way to simplify the development of management applications is to provide an object model that captures the applications' common logic. The framework presented in this appendix defines such an object model and provides its implementation. We chose to model a management application as a Finite State Machine (FSM). Finite State Machines provide a very effective way of describing and implementing the control logic for applications ([11]). They are commonly used to implement communication protocols, to control the interactions with a user, and so on. Finite State Machines are especially competent in areas of concurrent and real-time software, and in particular, network management. The framework handles an application's FSM state transitions, and, most importantly, it provides a per-state event subscription mechanism. Delivering asynchronous notifications to a subscribed application is a basic service provided in every network management solution. However, standard mechanisms for event subscription are not aware of the internal state of the application; registration is done per application. Our framework extends this mechanism by providing an additional subscription mechanism based on the current FSM state; different subscription information is defined for each FSM state. This mechanism makes the implementation straightforward, given a high-level definition of an FSM. It dramatically improves code quality and maintainability and greatly shortens the application development cycle.

[0084] The framework also provides a library of utility classes that can be used by all applications. These classes provide an abstraction layer between the network management system and the applications. They also improve code reuse and portability of applications.

[0085] The presented framework is implemented in IBM's Scalable POWERParallel System ([10]), on top of the existing Communication SubSystem (CSS) switch network management software. It has proven to be very useful in implementing SP switch network diagnostics—a set of tools for locating the cause of faults in an SP switch network, including tools that carry out the methods described hereinabove. These and other diagnostic applications have been developed using the framework. In addition, the general approach in definition, design and implementation of the framework allowed it to be used to implement an auxiliary testing tool.

[0086] The rest of this appendix is organized as follows:

[0087] Section 2 provides an overview of standard network management technologies.

[0088] Section 3 defines requirements for advanced management framework, and details a network management framework for an SP system.

[0089] Section 4 describes how SP switch diagnostics use the framework.

[0090] Finally, Section 5 summarizes the appendix.

[0091] The major technologies for network management are OSI (Open System Interconnection) Management (based on CMIP), Internet Management (based on SNMP) and CORBA/OMG.

[0092] 2.1 OSI Systems Management Overview

[0093] The OSI (CMIP) standards provide a efficient object-oriented management information model, support flexible distribution of management functionality, and provide common system management functions ([1], [2], [3]).

[0094] FIG. 7 is a schematic block diagram illustrating the OSI management architecture. The basic configuration has a manager system 200 that communicates with a managed system 202 in order to manage a resource 206 that is controlled by the managed system. The OSI architecture models these components as manager 200, agent 204, and managed object 206. The OSI management specifications cover the interface between manager and the agent. The main goal of the architecture is to define how to manage resources of all kinds. The standards have been divided into two areas:

[0095] Communications standards specify how manager and agent systems communicate with one another, using an OSI protocol stack and the CMIP application layer protocol. This includes specifying how to organize requests for operations on resources and reports on their activity, and how to transfer these requests and reports between open systems.

[0096] Management information standards are concerned with specifying how particular kinds of resources can be managed over a communications interface. This includes specifying which operations managers can perform on the resources and what unsolicited information are sent to managers when events occur. This kind of definition for a particular type of resource is referred to as a managed object definition.

[0097] The concept of managed objects has been adopted more widely than just in OSI. Managed objects have come to be regarded as a general technique for defining the management capability of a resource, applicable to OSI environments, Internet ([4]), CORBA, and so on.

[0098] 2.2 Internet Management

[0099] The IETF has specified a number of RFCs that define how network management should work in the TCP/IP environment. Its basic protocol is SNMP (Simple Network Management Protocol) ([5]).

[0100] The Internet Management model adopts a manager/agent approach, where the agents maintain information about resources and managers request information from the agents.

[0101] The Internet Structure of Management Information (SMI) standard specifies a methodology for defining the management information contained in the Management Information Base (MIB).

[0102] SNMP was designed to solve the communication problems between different types of networks, and it quickly became the network management protocol of choice for IP networks. Various developer kits are available for implementing SNMP managers; almost all major vendors of internetwork hardware, such as bridges and routers, design their products to support SNMP. However, non-IP devices, such as switches in our target environment, lack this support.

[0103] 2.3 CORBA

[0104] The Object Management Group (OMG) has developed an object-based environment for the development of distributed systems. This environment includes CORBA (the Common Object Request Broker) and IDL (an Interface Definition Language) which are used to specify the interface to objects. The ORB communication protocol is General Inter-ORB Protocol (GIOP) defined by the OMG; it is designed to work over different transport protocols. The Internet Inter-ORB Protocol (IIOP) is defined for TCP/IP transport.

[0105] CORBA is not a technology dedicated to network management. Rather, it is a distributed systems technology that can be used to specify objects that are related to the area of network management. CORBA also includes management-related services such as Naming and Event services.

[0106] 3.1 Management Frameworks

[0107] This section discusses the implementation of management applications and their integration within management infrastructures that provide basic configuration, setup, and fault management services. The objective of this discussion is to define the requirements for a framework that will allow a management system to be easily extended and simplify the deployment of extension applications to existing customer systems. Some management technologies explicitly define interfaces for implementing management applications within the infrastructure. Others do not define them, but rather leave the issue of providing management APIs to system implementers. Accordingly, the ability to extend a management product and to seamlessly integrate third-party applications within it varies greatly for different network management systems.

[0108] Common issues that have to be resolved when developing any management application are:

[0109] 1. Integrating the application within the infrastructure—Integration should be as seamless as possible. The system should be unaware of application specifics; the application dependency on the system version should be minimized.

[0110] 2. Access to management information—An application should have access to management information, including configuration information, real-time event notification, and so forth. The application should also be able to perform management functions, such as modifying the configuration or using services provided by the system.

[0111] 3. Remote invocation—In many cases, locally available management information is not sufficient for an application's needs. In these cases, the application will need a mechanism for invocation on remote machines.

[0112] 4. Interaction with user—Different components of a distributed management application may need to interact with users. A distributed input/output mechanism is therefore needed.

[0113] 5. Communication mechanism—Components of an application need to communicate. A reliable communication protocol should be implemented if the underlying communication mechanism is not reliable.

[0114] 6. Implementing asynchronous reactive behavior—An application typically must react to asynchronous events, such as error notifications from network devices. Therefore, it needs a methodology for representing and implementing this reactive behavior. In addition, the behavior on different types of servers may vary. A clean method of combining different behaviors within the same code is consequently needed. Providing common answers to these issues would shorten the application development cycle and time-to-market, thus giving management application developers the ability to be competitive and meet aggressive schedules. The remainder of this section describes the architecture and framework that give such answers for extending SP switch network management software.

[0115] 3.2 Architecture

[0116] FIG. 8 is a schematic block diagram illustrating an object-oriented framework for advanced network management, in accordance with a preferred embodiment of the present invention. It is assumed that the underlying management software is running on multiple nodes called management servers 212. An additional node called a management console 210 is used to run the management GUI. The architecture consists of components running on both of these types of nodes, as shown in the figure.

[0117] An I/O Server 216 is responsible for interaction with users, including invocation of an application 214 by the user. It is used as a proxy for exchanging relevant information between the user and the management software. The I/O Server communicates with Management Extender components.

[0118] A Management Extender 224 is a small component that provides support for extending the management servers. It runs as part of basic management software 222 and is responsible for the dynamic loading and unloading of applications 218 and the forwarding of events to these applications. The Management Extender interacts with a Management Abstraction Component (MAC) 220 and provides it with access to the underlying management software.

[0119] The Management Abstraction Component is a class library that provides an abstraction layer between the management software and the applications. It also defines an object model for event-driven applications, as discussed below in Section 3.3.

[0120] The rest of this section details the responsibilities of each component and their interactions.

[0121] 3.2.1 Management Extender

[0122] The Management Extender component runs as part of a management server. It listens to arriving events, identifies and processes special events sent to the Extender, and forwards other events to the MAC.

[0123] The Management Extender exposes the definition of several special events that are used for application invocation and termination. Other components (running on the management console or servers) can send these events to the Extender, in order to perform remote invocation or shutdown of an application.

[0124] The Extender keeps track of any applications it invokes. When no applications are running, it ignores all events (except its own). When applications are running, the Extender forwards these events to the MAC, where they are dispatched or delivered to applications if necessary.

[0125] 3.2.2 I/O Server

[0126] The I/O Server component runs on the management console and interacts in a unified manner with the Application Invocation Module provided by each application. The Application Invocation Module receives input parameters from the user and passes them to the I/O Server. The I/O Server sends an application invocation event to the Extender that is running on the default server. Similarly, the I/O Server provides an interface for aborting applications.

[0127] While an application is running, the I/O Server can display application messages on the Management Console, and send user responses back to the application using special events.

[0128] 3.2.3 Management Abstraction Component (MAC)

[0129] The MAC is a class library that provides an abstraction layer between the management software and the applications. It interacts with the Extender to access management information and basic services.

[0130] The MAC provides the applications with the following different types of services:

[0131] 1. Implementation of management classes that encapsulate management information and services, thus abstracting them from the system's internal details. For example, it can provide a class representing a network adapter and provide methods for the adapter's configuration.

[0132] 2. Exposing interfaces for exchanging messages with the user.

[0133] 3. Provision of an interface for exchanging messages with the other application components that run on additional servers. It implements a reliable communication protocol on top of an existing unreliable communication protocol provided by the underlying software. The interface is provided both for the reliable protocol and for the native unreliable one.

[0134] 4. Implementation of utility classes that abstract applications from the operating system. Examples include classes such as Thread, Mutex, and File that represent OS resources; Timer that provides a timeout mechanism; logging facilities class, etc.

[0135] 5. Implementation of a common logic of event-driven applications, as defined by the object model described in the next section.

[0136] 3.3 Object Model for Event-driven Applications

[0137] The common management services provided by the framework are described above. The next step in designing the framework is to define a methodology for implementing event-driven applications. The framework defines an object model that represents asynchronous behavior and implements the common logic.

[0138] An application is modeled as a Finite State Machine (FSM), in which input events are:

[0139] Events produced by the management system.

[0140] Events produced by the framework (e.g., events generated by the I/O Server, from user input).

[0141] Messages exchanged by the application components.

[0142] The FSM object model defines classes that represent: generic FSM, FSM state, FSM input event, and FSM state transition. The framework implements these generic objects. An application derives its own objects from these classes and thus reuses both design and implementation.

[0143] The framework handles FSM state transitions and provides the event subscription mechanism based on the current FSM state (i.e., different subscription information is defined for each FSM state). The framework handles all issues regarding event dispatching and guarantees that unexpected events will not be delivered.

[0144] FIG. 9 is a schematic block diagram that illustrates the main objects or classes defined in the FSM object model and the relationship between these classes, in accordance with a preferred embodiment of the present invention. The class descriptions are as follows:

[0145] FSM_Coordinator 230

[0146] This class is a “handle” to an application FSM. It supports the dynamic creation of FSMs and handles interactions with infrastructure by subscribing to relevant events, creating event objects for received events and forwarding the event objects to FSMs.

[0147] FSM 232

[0148] This class represents a Finite State Machine (FSM) and:

[0149] Provides an interface for registering FSM states and for subscription of common events (i.e., events that should be handled in the same manner in each state).

[0150] Is responsible for maintaining the current state and for state transitions.

[0151] Implements the dispatching of the events.

[0152] To do this, it looks for an appropriate event handler in the FSM event subscription information and in the subscription information of the current FSM State. It then invokes the handler (if found) or discards the event if neither the FSM nor the current state are subscribed to this event.

[0153] Users employ this class to derive their own classes representing application-specific behavior. This behavior may depend on the server's role (i.e., it may be different for different components of the same distributed application).

[0154] FSM_State 238

[0155] This class represents the state of a Finite State Machine. The FSM_State object provides an interface for subscription to state-specific events. It is used for registration of proper handlers for events which have to be processed in this state. Users derive their own states from this class and register these states in appropriate FSM objects.

[0156] FSM_Event 234

[0157] This class represents events in the framework. FSM_Event data includes the real event information and a unique ID that is used for event subscription.

[0158] FSM_EventHandler 236

[0159] This class represents the FSM state transition and defines an abstract method for event processing. All user defined handlers are derived from this class; they provide the implementation of the event processing. Application-specific FSM and state objects register relevant event handlers to be invoked by the framework.

[0160] The above classes allow applications to be implemented in a very simple manner. Given a high-level definition of an application-specific FSM, a corresponding class is derived from FSM class. For each state a corresponding class is derived from FSM_State and for each transition a corresponding class is derived from FSM_EventHandler. The FSM-derived object creates and registers its states at initialization time; each state creates and registers its event handlers at its initialization time. Each handler provides implementation for the abstract event processing method defined in the parent class FSM_EventHandler. This method implements an FSM transition; it returns the identifier of the next FSM state.

[0161] FIG. 10 is a schematic diagram that illustrates how FSM objects interact in order to process an event, in accordance with a preferred embodiment of the present invention. Once the application has defined its states and handlers according to the above object model, it is no longer concerned with maintaining state and deciding how to process an event. It need not examine various state flags to determine appropriate behavior, and to filter out irrelevant or unexpected events. This makes the implementation much more robust and maintainable. In addition, it is easy to verify that the application behavior matches the high-level design.

[0162] 3.4 Implementation Details

[0163] In this section, we describe several points of the framework implementation. The target platform for the implementation was the IBM SP, running AIX operating system, version 4.3. The implementation was done in C++ language using the IBM xlC C++ compiler.

[0164] 3.4.1 Dynamic Invocation Implementation

[0165] The framework implements dynamic application invocation using an AIX mechanism for dynamic loading of shared libraries. Applications are implemented as AIX shared libraries and provide a common interface used by the framework for application initialization. The Management Extender invokes an application upon receiving a special event. This event contains essential information about the application, including the path to the application library and application parameters. When such an event is received, the Extender loads the specified library, obtains a pointer to its entry point, and calls it.

[0166] The entry point is a function implemented by the MAC. Each application is statically linked with the MAC and specifies this function as its library entry point in link settings. The entry point is used to perform necessary initialization of the MAC and the application, and to resolve interface functions used by the Extender to forward events to the MAC.

[0167] The MAC initializes the application by calling a predefined initialization function. Since it is used to create an application-specific FSM object, each application must implement this function, which is later used by the MAC.

[0168] 3.4.2 Reliable Communication Protocol

[0169] The protocol used for network management by the SP switch network management software is unreliable. That is, there is no FIFO and no guarantee for delivery of the management packets. The framework we developed implements a reliable protocol over the native unreliable protocol. An application can use our protocol when it is necessary to guarantee an ordered delivery of the management packets, or can use the native unreliable protocol if no guarantee is required.

[0170] In order to support the reliability of the communication protocol, we used the following techniques:

[0171] Storage of buffers on the source server with a time-out.

[0172] Retransmission and acknowledgments for successful packet delivery.

[0173] Suppression of the duplicate packets in case of multiple sends.

[0174] Every packet that is sent between two servers carries a sequence number that is maintained independently between any two peer servers. These sequence numbers allow the packets to be delivered to the application in the same order in which they were transmitted by the sender, even if they were received out-of-order because they were sent over different routes. When a receiver delivers a packet to the application, it sends an acknowledgment for the received packet sequence number. If after a pre-defined timeout, the sender does not receive acknowledgment, it retransmits the retained copy of the lost packet (or discards it after predefined number of retries and reports an error to the application). If the receiver receives several packets with the same sequence number, it delivers the first one to the application and discards the rest. However, the receiver will send an acknowledgment for every copy of a delivered packet.

[0175] The reliable protocol of management packet transmission provides a duplicate- and loss-free FIFO channel between any two servers. It has proven to be a very effective feature greatly facilitating the design and implementation of management applications.

[0176] The framework described in the previous section was used to develop SP Switch Diagnostics—a set of tools used to isolate the cause of faults in SP switch networks, including the diagnostic functions described hereinabove.

[0177] SP Switch Management Software provides a basic mechanism for fault recovery. It receives error notification events and performs necessary changes in the network configuration and device setup. This allows the network to remain operational, but does not provide a complete solution for fault isolation. In many cases the same error report may be caused by a fault that occurred in different devices. For example, errors that were detected on a link may be caused by faults in devices on both sides of the link, or by a bad cable in this link. The basic mechanism of reacting to error reports would simply disable the link when the error threshold is exceeded or a critical fault occurs. An additional mechanism is needed to identify the faulty component, so that it can be replaced, and the network can continue to achieve maximum performance. SP Switch Diagnostics tools solve this and additional network management problems. The tools were implemented using the framework discussed in this appendix and using the FSM object model.

[0178] We presented an object-oriented framework for the dynamic extension of network management software. This framework resolves common issues that were listed in Section 3.1, including:

[0179] Provides applications with the ability to be seamlessly integrated within a management infrastructure through a dynamic invocation mechanism.

[0180] Provides applications with a unified interface to management software through MAC management classes.

[0181] Provides a remote invocation mechanism through the I/O Server and Management Extender.

[0182] Provides a user interaction mechanism through the I/O Server.

[0183] Implements a reliable packet delivery protocol within the MAC.

[0184] Defines and implements an object model for event-driven applications.

[0185] The framework is used to implement several SP switch diagnostic tools. The general approach used in the design and implementation of the framework also allow it to be used to develop auxiliary testing tools that were not originally defined. These testing tools provide functionality which is completely different from the diagnostic tools, namely, the implementation of error injection, simulation and/or stimulation of different network errors, and allowing the testing of diagnostic tools, management software and any distributed applications that use the network.

[0186] [1] Systems Management Overview, ISO/IEC 10040 (X.701).

[0187] [2] Management Framework for OSI, ISO/IEC 7498-4 (X.700).

[0188] [3] Common Management Information Protocol (CMIP), ISO/IEC 9596.

[0189] [4] The Common Management Information Services and Protocols for the Internet (CMOT and CMIP), RFC 1189.

[0190] [5] A Simple Network Management Protocol (SNMP), RFC 1157.

[0191] [6] Java Management API (JMAPI), http://java.sun.com/products/JavaManagement

[0192] [7] DMI 2.0s Specification, http://www.dmtf.org/spec/spec.html

[0193] [8] CORBA/IIOP 2.2, http://www.omg.org/library/specindx.html

[0194] [9] CORBAservices, http://www.omg.org/library/specindx.html

[0195] [10] The RS/6000 SP High-Performance Communication Network, http://www.rs6000.ibm.com/resource/technology/sp_sw1/spswp1.book_—1.html

[0196] [11] UML Finite State Machine Diagrams, Robert C. Martin, Engineering Notebook Column, C++ Report, June, 1998.

[0197] This appendix describes further aspects of methodology, architecture and a framework for developing diagnostics tests and diagnostics tools for the manageable switch network. It outlines a client-server model for the diagnostics architecture and its framework in the switch network, which allows easy creation of diagnostics tests that are able to access every resource of the network. The tests, which are incorporated in the switch management software, run concurrently with user applications without any disturbance to the user applications and to network management software. In this architecture, the server comprises switch management software that exports diagnostics interfaces to the clients, and the clients are different diagnostics tests.

[0198] Switch diagnostics are tests that will give service personnel the ability to troubleshoot switch problems during both normal operation of switch network (non-destructive tests) and during times when the switch network is not running customer applications (destructive tests). Technical personnel are able to run these tests from a Control Workstation (CWS) when they suspect that a component in the switch network is not performing properly. Switch diagnostics point to the failing component and make a decision as to whether this is a “real” hardware problem.

[0199] The following two issues summarizes the challenges in developing a new SP diagnostic test:

[0200] 1. Developing a well-defined model of the failures to be detected.

[0201] 2. Defining the test capabilities (what kind of failures will be detected).

[0202] 2.1. Properties of the Model

[0203] The model described hereinbelow is based on five properties: Coverage, Sensitivity, Operability, Performance and Scalability (referred to hereinafter as CSOPS). The following table depicts a simple definition for each of the five properties:

[0204] Overall, these properties provide a model for defining old and new tests, and become a measure of their effectiveness.

[0205] 1) Coverage

[0206] We use Coverage to define what failures the test should detect. The Coverage property of a test is defined using a model of the failures that are the target of this test. Such a model can be:

[0207] Deterministic failure of a single path through a switch chip, or

[0208] Heavy traffic through each path of a switch chip, or

[0209] All combinations of two overloaded paths through a switch chip.

[0210] This and other models can be defined, specifying what the particular test can detect.

[0211] The issue of whether the failure is deterministic or non-deterministic is also of relevance to the model. In the case of deterministic failure, Coverage will usually be specified in terms of the failure, or its cause. An example of this type of Coverage is: “the deterministic failure of an interposer or a cable.”

[0212] However, in many cases, non-deterministic failures are the target of diagnostics tests, such as intermittent failures. In these cases, it is difficult to define the failure or its cause. In this case, the model will be defined in terms of the activity that is being stimulated. An example of this type of Coverage is: “bombarding the switch with heavy traffic.” Another example would be a test that exercises specific data patterns to stimulate failures, for instance, the failures of a interposer.

[0213] In both cases of failures (deterministic and non-deterministic), the target of the Coverage property is to provide a lucid definition of what the test can do. After reporting a problem, the given set of diagnostics is required to identify what test to use. The association of a reported problem with a test is left to the expert who is using the test. The Coverage property serves as the basis for this association.

[0214] 2) Sensitivity

[0215] Some of the failures detected by the diagnostics, are non-deterministic. This present a dilemma for the developer of the test. The test can be written in such a way that it runs until a failure is detected, but using such a test may be impractical, as it could run forever.

[0216] As a result, we can assume that some tests, although having the potential of detecting a failure, may fail to stimulate the particular failure and therefore not detect it. The target of the Sensitivity property is to specify the probability of a test missing a failure.

[0217] In the case of a deterministic failure, it is good engineering practice to have a test that presents full sensitivity to the failure. For example, a failure caused by a disconnected cable is deterministic, so that the test that detects it should also be deterministic.

[0218] The Sensitivity property of the diagnostic, is used to characterize another probable difficulty. A test may report a failure in its absence, for example the problem of bad CRC. A packet passing through an adapter may have a bad CRC. This is a normal situation, unless the frequency of the bad packets is higher than an allowable threshold. It is possible that due to a stressful test, that threshold might be reached.

[0219] We will use the following terminology when defining the Sensitivity property of a test:

[0220] False-positive: There is no failure, but the test reported a failure.

[0221] False-negative: There is a failure, but the test missed it.

[0222] 3) Operability

[0223] The Operability properties of a diagnostics test encompass several characteristics, all related to the limitations of the test:

[0224] Destructive: During the execution of a destructive test, applications that are using switch network cannot be run. A destructive test can change the routing information of the nodes.

[0225] User intervention: Define user intervention during the execution of the test, like insertion of a wrap plug.

[0226] System constraints: Limitation in running the test, such as limited memory, or specific system topology.

[0227] When Bring-up, Maintenance, Installation, or Normal operation.

[0228] 4) Performance

[0229] The Performance property of a diagnostic identifies the time it takes to run the test. This is an important factor for a test that stresses the system, or tests that iterate through data patterns. Some tests may take a long time to run, or their time of execution can be defined by the user.

[0230] 5) Scalability

[0231] In the development of a diagnostic test for a parallel system, it is important to have tests which are scalable. A target of all tests will be to have a complexity no more than linear in N (number of nodes) The Scalability property describes this relationship.

[0232] Another area of complexity in diagnostics tests is that they are computer programs, part of the system management software. The constraints on running these programs is an important consideration, as they may have to be executed in presence of system failures.

[0233] 2.2. Existing Approaches

[0234] Today there are three main approaches known in the art to write switch network diagnostics tests:

[0235] 1. Writing different script that perform some kind of automation of system administration methods.

[0236] 2. Writing stand-alone user space programs that run on top of the operating system but separately from the switch management software.

[0237] 3. Writing programs that performs diagnostics but are non-compatible with switch management software.

[0238] Although these approaches may be used to implement the testing and diagnostic methods of the present invention, they have serious disadvantages:

[0239] 1. Scripts are system administrators' home-made utilities, which mostly perform some automation of trivial testing (like pinging some nodes in order to test the nodes' connectivity). These scripts can test some very simple deterministic situations (such as a node that consistently does not respond to the ping), but are useless for more sophisticated testing. Also, system administrators usually do not have real knowledge about underlying hardware at a sufficient level in order to create more clever scripts.

[0240] 2. User space programs designed to diagnose switch-related problems suffer from the fact that they cannot access privileged resources, such as adapters, switches, reported errors, etc. They are not able to create sufficient testing of the hardware or analyze the results of testing, since they have no integration or interfaces with switch management software that was created for such diagnostics purposes.

[0241] 3. Diagnostics programs that are non-compatible with switch management software cannot run concurrently with applications. One extreme example of such diagnostics program is how advanced diagnostics implemented in most Windows NT boxes: user have to exit Windows NT and reboot the machine to DOS, and from there run the advanced diagnostics test suite.

[0242] A client-server model for diagnostics architecture in a switch network, as described herein, allows easy creation of diagnostics tests that are able to access every resource of the network, are able to run concurrently with the user applications, and can be incorporated in the switch management software.

[0243] 2.3. SP Switch Diagnostics Architecture

[0244] SP diagnostics system (SPD) architecture is designed to meet several requirements:

[0245] 1. Tests must fit into the system management architecture.

[0246] 2. Tests must not disturb (or prevent) the system management software from doing its primary job.

[0247] 3. Tests must abstract system management software from its details, by providing common interface. The system management software should be unaware of test specifics.

[0248] 4. Test musts be loaded dynamically only when the user wants to run them, and be unloaded immediately after they are done, leaving the system in exactly same state as before test was started.

[0249] 5. Performance, reliability, serviceability and availability features of the system management software must be unchanged when diagnostics are not invoked.

[0250] FIG. 11 is a schematic block diagram that illustrates the SPD architecture, in accordance with a preferred embodiment of the present invention. The architecture includes two different parts: SPD layers that run on a CWS 300, and SPD layers that run on every node 310.

[0251] When the user invokes a test on the CWS, a Test Invocation Module (TIM) 304 creates a Test Definition File (TDF) 306, initializes a SPD Message Daemon Server 308 and invokes a SPD GUI program 302. After this whole environment was created on the CWS, the TIM remotely invokes an i_stub program 320 on the primary node. This program generates a diagnostics request to the system management module, called a fault-service daemon (FSD) 322.

[0252] FSD is an event-driven program that idles most of the time waiting for received events, such as error packets from any network component. When FSD receives a diagnostics request, it changes its run mode to the diagnostics mode. In this mode, all received events are first handled by the diagnostics models, and only after that might be forwarded to the FSD for further processing. First, FSD loads a dynamically-loadable library SPDLib 316. When SPDLib is loaded, FSD resolves all hook functions for diagnostic processing. There are several such functions that are called only when diagnostics are running.

[0253] The main responsibility of the SPDLib is to provide a general interface between tests and the FSD, abstract the FSD logic from the test logic, and to provide a common, general mechanism for communication with the user on the CWS. For this purpose, the SPDLib also includes a Message Daemon Client 318, which communicates with Message Daemon Server 308. When SPDLib is loaded and properly initialized, it loads a test 312 that is to be performed. Every test is implemented as a dynamically-loadable library, which is statically linked with a Test Object-Oriented Framework 314. From the TDF file on the CWS, SPDLib receives all necessary information about, the test—where on the node the test resides, what is the test name, what are the port numbers of receive and transmit sockets of the SPD Message Daemon Server, etc.

[0254] SPDLib loads the test library and resolves all test interface functions. Every test includes several modules that provide different coverage of the problem. The SPD Test Object-Oriented Framework, which lays the basis for the tests, provides a general design and implementation framework. The tests have many things in common in their logic, including: the way tests are handled in the initialization stage, the way tests are terminated, the way different models of tests are loaded and unloaded, the way request messages and service packets are handled and processed, the way the communication protocols are used (initialization of the protocols and use of protocol services), and more.

[0255] The SPD object-oriented framework architecture provides a common framework that addresses all of these issues, providing a high degree of code reuse. The SPD framework has two aspects: the first is the framework itself, which provides all design and implementation patterns for the tests; and the second is helper utilities, which create a general interface to common resources, such as adapter use, protocol use, AIX resources use, etc.

[0256] 2.4. Class Descriptions

[0257] FIG. 12 is a schematic block diagram illustrating a hierarchy 330 of classes used in the SPD architecture, in accordance with a preferred embodiment of the present invention. The classes shown in the figure are described hereinbelow:

[0258] Class ModelCoordinator 332

[0259] This class is an orchestrator of all SPD frameworks. It decides which models to run and when, handles interactions with SPD, including registering callbacks and providing handler functions. It creates Event objects from received packets or requests and invokes Event processing. It also supplies an interface for model/test termination.

[0260] Class AlgorithmPrototype 338

[0261] This class provide a general interface to an algorithm by which the model on the node is initialized or terminated. On every node type this algorithm is different, and this class hides specifics of the algorithm.

[0262] Two derived classes—PrimaryAlgorithm 340 and NonPrimaryAlgorithm 342 implement their logic in the proper way.

[0263] Class Model Prototype 336

[0264] This class represents model Finite State Machine (FSM), is responsible for state transitions, and holds state objects of the FSM states. Users derive their own classes from this class, representing different models of behavior on different types of nodes.

[0265] Class ModelFactoryPrototype 334

[0266] This class provides an interface for creating and enumerating test models. Its derived class holds knowledge of existing test-specific models.

[0267] Class State 346

[0268] This class represents the state of a Finite State Machine of the test model. The State object is used for registration of proper handlers for events that have to be processed in this state. The user might either derive his own states from this class, or use the provided class for State implementation.

[0269] Class Event 344

[0270] This class implements events in the framework and can process itself. Users can define callback data and store data in the event object, but, it is up to user to delete the callback data when necessary. (It is recommended that user detach the callback data from the event object prior to its deletion).

[0271] Event knows its type, subtype and kind, and requires EventToHandlerMap to retrieve the appropriate handler to process itself. The user cannot destroy Event. Rather, Event will take care of its deletion by itself, depending on how many references to it are still pending (i.e., how many instances of the event are in process)

[0272] Class HandlerPrototype 348

[0273] HandlerProtoype class is an abstract class that defines all necessary methods for event processing. All user-defined handlers should be derived from this class. Return code from the handler processing will be eventually returned to the SPDlib in order to decide what further actions should be taken according to this return code.

[0274] Class DefaultHandler 350: public HandlerPrototype

[0275] Class is used when the user did not provide any handler for event processing. The user can override this default processing.

[0276] In order to write a test, the test developer must define the test Finite State Machine, as well as events that are of interest to the test in every state, and must implement the handlers for these events. All other work is handled by the framework, minimizing the design, implementation and testing times of every test.

[0277] The user issues a request to run diagnostics on the CWS. The Test Invocation Module (TIM) parses the command line arguments and creates the Test Definition File (TDF). This file contains, in addition to test-specific parameters, several diagnostic sub-system common parameters, including the test to be invoked, the path to the loadable library to go with the test, models of the test to be run (optional), maximum time that the user allows test to run, nodes that are allowed or forbidden for the test to use, etc.

[0278] After the TIM invokes the Message Daemon Server and GUI program, and invokes the i_stub interface program on the primary node, it puts a diagnostics request into the Fault-Service Daemon (FSD) message queue.

[0279] When the FSD picks up this message from the queue, it realizes that this message was originated by the diagnostics subsystem (because of the message type) and loads the SPDLib. The SPDLib serves as an abstraction layer between the FSD and the Diagnostics Framework. It has a number of responsibilities, among them: to provide common interface to tests, and to provide a reliable communication protocol between nodes that participate in diagnostics activity.

[0280] After it is loaded, the SPDLib first resolves all hooks that exist in the FSD and are not active when there is no diagnostics activity. Every message, request or packet (all of which are termed events) received by the FSD will first be processed by these hook functions of the SPDLib. When the SPDLib finds that the particular test should continue with the processing, it forwards the event to the test processor.

[0281] Next, the SPDLib reads the TDF file and determines on this basis which library to load and via what path, and then loads this library. Each test library is made up of two parts: the statically-linked Test Object-Oriented Framework (T.O.O.F.), and test-specific logic on top of this Framework.

[0282] The T.O.O.F. creates a uniform environment for tests, depending on which node the test is running. The test environments of different nodes are different. For example, the primary node orchestrates all diagnostics activity, while non-primary nodes act as slaves.

[0283] The ModelCoordinator class of the T.O.O.F. is the class that actually orchestrates all test activity. First it decides which test models will run and in what fashion. For this purpose, it queries ModelFactoryPrototype (which is an abstract, user-implemented class) as to which models are available in the test and the estimated execution time for each model. With this information and data from the TDF in hand, the ModelCoordinator generates a list of models that are going to be executed and in what order. It then loads first test model from the list, executes it, unloads when the model is done, loads the next model, and so forth.

[0284] The test itself is implemented as a Finite State Machine (FSM). There are number of events in every state that are of interest for the test, and the test has different handlers for these events. When an event happens, the ModelCoordinator invokes the proper handler for the current state, in order to process this event.

[0285] Because the T.O.O.F. is used, the test developer need only decide what FSM to use for any particular test, which events are of interest for the FSM, and how to handle these events. The test developer is thus free to concentrate only on test-specific logic.

[0286] The switch diagnostics architecture described hereinabove allows the development of proactive diagnostics. Proactive diagnostics are diagnostic tests that run concurrently with user applications during underutilized system times, and test different hardware components during these times. The purpose of these proactive tests is to find out whether the tested hardware degrades in its operation, and to warn user about possible failures prior to the failure itself.

[0287] One of the facts that facilitates development of proactive diagnostics on top of the SP Switch Diagnostics Architecture is that this architecture does not disturb in any way the work of the management software layers and runs concurrently with them in the same process space, giving the diagnostics the ability to access any privileged resource. Because all modules are implemented as shared loadable libraries, they are loaded only when a node is to perform proactive diagnostic testing, and are immediately unloaded after the testing is done.

[0288] Diagnostic tests that are written in the present framework are preferably evaluated using an error injector, as described hereinabove.