Apparatus for testing communication equipment

[0002] This application claims the benefit of prior-filed copending U.S. Provisional Application No. 60/052,430 filed Jul. 14, 1997, the disclosure of which is incorporated herein.

[0003] The present invention relates in general to testing apparatus, and more particular to apparatus and methods for testing the operability of transaction systems to find errors and faults, as experienced by customers and users of the systems.

[0004] The networking of computers and the computerized equipment has lead to substantial enhancements in the accessibility and distribution of data and information. The networking of equipment can be accomplished by local area networks, wide area networks and other types of communication systems, such as the public switched telephone network (PSTN). While the integration of geographically remote equipment is substantially facilitated by networks of various types, the failures in such networks can and systems have substantial and wide-spread ramifications. Indeed, the troubleshooting of networked systems becomes more troublesome and complicated. There exist test operations and methods that can carry out periodic tests on a network system and report failures. In many instances, the test circuits are driven by software embedded in the application program. While this may be effective, a failure of the application system often results in the inability to report failures. As a result, the customers or users are the first to experience inoperability of the system, rather than a repair or technical staff.

[0005] From the foregoing, it can be seen that a need exists for a technique to carry out test on transaction equipment in a manner substantially the same as the procedures utilized by a customer or user. Another need exists for a procedure to test the equipment in the manner utilized by customers so that an accurate portray of the fault can be experienced by the test operation. Another need exists for a test technique for finding faults in the equipment before customers, thereby reducing customer dissatisfaction with the system. Another need exists for a test method for carrying out tests on systems at periodic time intervals to verify the operability of the system.

[0006] In accordance with the principles and concepts of the invention, the shortcomings and disadvantages of the prior art systems have been overcome. According to the preferred embodiment of the invention, the test system accesses a transaction-type system in a manner much like the users thereof. The responses of the transaction system are much like those received by the users of the system. Based on the test conducted, and the response thereto, it can be determined if the system is functional, and if not, which sub-system is at fault. The invention is integrated with conventional test systems so that the test can be carried out periodically, and malfunctions reported quickly to the appropriate technical staff or repair persons.

[0007] In accordance with an important feature of the invention, a traffic driver/checker is provided with probe points connected to the various subsystems and components of the system. In addition, the traffic driver/checker includes check points connected to various other equipment of the system to be monitored and tested to obtain inputs in response to tests carried out. The traffic driver/checker is programmed with a Management Information Base (MIB) that controls the flow of tests on the transaction system. The traffic driver/checker operates in conjunction with a conventional enterprise automation package to perform the surveillance, monitoring and testing of the transaction system.

[0008] Further features and advantages will become apparent from the following and more particular description of the preferred embodiment of the invention, as illustrated in the accompanying drawings in which like reference characters generally refer to the same parts, elements or functions throughout the views, and in which:

[0009] FIG. 1 is a block diagram illustrating the environment in which the invention can be advantageously practiced;

[0010] FIG. 2 is the management information base (MIB) structured according to the preferred embodiment of the invention;

[0011] FIG. 3 is a user screen display illustrating the number of areas selectable for use;

[0012] FIG. 4 is a user screen display illustrating a promotion window;

[0013] FIG. 5 is a user screen display showing a test case editor;

[0014] FIG. 6 is a user screen display illustrating a test executive window;

[0015] FIG. 7 illustrates a screen display showing a test list with corresponding test results; and

[0016] FIG. 8 is a blocked diagram showing the interrelationship between the MIB, the test case list and the individual test cases.

[0017] FIG. 1 illustrates a block diagram of the environment in which the preferred embodiment of the invention is adapted for operation. The invention is well adapted for use with transaction systems, such as automatic call distributors (ACDs). Conventional ACD equipment is accessed through the public switched telephone network (PSTN) 20. The PSTN 20 is coupled to an interactive voice response ACD unit 22 by way of telephone lines. The IVR ACD 22 is, in turn, coupled to one or more interactive voice response units that contain the IVR call flow sequences, one shown as reference 24. The IVR call flow retrieves data from legacy systems, associates data to the individual calls, and transfers calls to agent ACD queues. The IVR units 24 can be conventional Unix systems programmed to provide various types of voice responses. The IVR units 24 are coupled to an agent ACD system 26. The agent ACD system 26 is coupled to and delivers queued calls to plural agent telephone sets, one shown as reference 28. As is common, an agent or operator working with the telephone set 28 also has available a terminal or other computer-type equipment 30. As the agent is discussing with the calling party on the telephone set 28, the terminal 30 can be utilized to call up data related to the calling customer. A database concerning matters related to the various calling customers is stored in a mainframe computer 32. The mainframe computer is coupled to a computer telephony integration (CTI) unit 36 via a data gateway 34. The CTI provides screen displays or pops to the terminals 30. The CTI unit 36 is coupled to an application server 38 which, in turn, is coupled to the agent terminal 30.

[0018] The overall ACD system functions to receive incoming calls and distribute the calls to the various agent positions that are not then busy. The agents can then verbally discuss matters with the calling customer, call up data concerning the customer, and provide human assistance. As can be appreciated, the monitoring and testing of the system for operability is important so that any faults are detected early and repaired such that the system functions optimally. While the preferred embodiment is described in conjunction with an ACD system, other transaction systems, such as legacy systems, and other systems, are applicable to use by the invention.

[0019] In order to provide surveillance and monitoring for the ACD system, the invention employs conventional equipment adapted for managing local area networks. In the preferred embodiment, utilized in conjunction with the invention is an enterprise automatic package, such as the Hewlett-Packard IT operations center equipment 40. The operations center 40 is programmed therein the rules, policies and instructions for generating trouble tickets concerning any malfunctions of the system to be monitored. In addition, the operations center 40 functions in conjunction with Hewlett-Packard openview equipment, identified by reference 42. The openview equipment 42 includes a system map for providing SNMP control and a polling rate for carrying out tests.

[0020] In accordance with an important feature of the invention, provided is a traffic driver/checker 44 that includes plural output checkpoints 48 coupled to the system to be monitored and tested. The driver/checker 44 includes an SNMP proxy agent that operates under the control of the open view equipment 42, through a system MIB. The traffic driver/checker 44 can communicate with the openview equipment 42 by way of the Internet. In addition, the traffic driver/checker 44 includes a number of output probe points 46 connected to the various sub-systems of the ACD system. The open view equipment 42 continually polls the ACD system through the traffic driver/checker 44. Both the directory numbers of the PSTN 20 and the IVR individual numbers are polled. When a failure is detected, the operations center 40 probes the system, via the traffic driver/checker 44, to determine which component or sub-system failed. When the failed component is identified by the rules programmed in the operations center 40, a trouble ticket together with an alert is sent to the appropriate office for tracking and repair. The open view equipment 42 provides in a standard manner SNMP mechanisms to control the traffic driver/checker 44. The open view equipment 42 further provides polling control and obtains and sets variables in the driver/checker 44. The operations center 40 contains the policies and control to utilize the probe points 46 provided by the driver/checker 44 to complete surveillance and determine the failing component when outages are detected.

[0021] In the preferred form of the invention, the probe points 46 provided by the traffic driver/checker extend as directory numbers to the PSTN 20, as queue directory numbers to the interactive voice response ACD unit 22, as test ports and respective directory numbers to the individual voice response units 24, as queue directory numbers to the agent ACD unit 26, as transactions to the mainframe computer 32 and as a test program on each IVR to the data gateway 34. Various data or information can be input by the traffic driver/checker 44 to the ACD system to determine operability thereof. In the case of the PSTN 20, a telephone number can be dialed to access the ACD system. Responses from the ACD system, in the nature of check points 48 are obtained from the computer telephony integration unit 36, the agent terminal 30 and the agent telephone unit 28. Two mechanisms are utilized for system surveillance of the ACD system. First, polling the system (especially the individual IVRs) and, second, monitoring customer traffic. The first mechanism verifies the correct operation of the IVRs and the CTI components of the system. The second monitor skills requested by the customers, and statistically verifies that calls are routed to the correct agents for the skills requested. The openview equipment 42 continually polls the system through the traffic driver/checker 44. Both the PSTN 20 directory numbers and the IVR individual numbers are polled. When a failure is detected the operations center 40 probes the system to determine which component failed. A failure can be a situation when a call is not completed through the system to a virtual agent. As noted above, when the failed component is identified by the rules programmed in the operations center 40, a trouble ticket and alert notice are dispatched to the appropriate groups for tracking and repair.

[0022] In the preferred embodiment, the traffic driver/checker 44 includes a personal computer with a Pentium 166 MHz processor, 128 MB of RAM, 15 megabytes of hard disk drive, Windows NT 4.0, a Dialogic T1 board, and other line interface cards or circuits.

[0023] FIG. 2 illustrates the MIB structure programmed within the traffic driver/checker 44. The MIB structure 50 includes a traffic driver/check MIB 52, an active test MIB 54, a test case set MIB 56 and an individual test case MIB 58. The traffic driver/checker MIB 52 describes the traffic driver/checker and the voice ports forming a part of the checker 44. Each port section of the traffic driver/checker MIB 52 may contain the object identifier (OID) of the active test set MIB 54 when a test is utilizing the port. The active test set MIB 54 describes and controls active tests carried out to test the ACD system. This MIB is polled by the SNMP manager in order to run tests which monitor the system under surveillance. Each MIB indicates the set of test cases to run as a result of the SNMP manager “get” command.

[0024] The test case MIBs 56 describe individual test cases. These MIBs are defined so as to be extensible by the user, either by adding new test case MIBs in associated action routines, or by using an instance of the generic test case MIB that executes any user provided windows MT program as a test case.

[0025] The test case MIBs 56 have two sections, namely the test setup and test execution sections. In a set of test cases, all test set up sections of the MIBs are executed before the test execution sections. The test set up sections of the test case MIB is synchronous, meaning that it does not return a “success” value to the “SNMP get” command until the test set up is complete.

[0026] The openview equipment 42 communications with the traffic driver/checker 44 by using a system MIB. The system MIB is shown in Tables 1-6 below. The OID of the elements in the MIB is prefixed with the notation 1.3.6.1.4.1.[999], where the following relationship exists:

[0027] 1 equals ISO

[0028] 3 equals ORG

[0029] 6 equals DOD

[0030] 1 equals Internet

[0031] 4 equal private

[0032] [999] equals the business controlling the testing apparatus

[0033] 1 equals the traffic driver/checker MIB 44.

[0034] The traffic driver/checker static station of the MIB is set forth below. The OID for each MIB element must be unique. The “n” in the OID enables multiple agents. For example, n equal 1 for the first agent, n equal 2 for the second agent, and so on.

[0035] The traffic driver/checker test set or dynamic section of the MIB is as follows. In this section of the MIB, the “n” in the OID enables the traffic driver/checker to execute multiple active tests. For example, n equal 1 for the first active test, n equal 2 for the second active test, and so on.

[0036] In this section of the MIB, the “n” in the OID enables the traffic driver/checker 44 to contain an unlimited number of sequences of test cases. For example, n equal 1 for the first sequence of test cases, n equal 2 for the second sequence of test cases, and so on.

[0037] In this section of the MIB, the “m” in the OID enables the traffic driver/checker to execute a sequence of test to complete an active test. For example, m equal 1 for the first test case, m equal 2 for the second test case, and so on.

[0038] The test case section of the MIB is as follows. In this section of the MIB, the “n” in the OID gives each test a unique OID. For example, n equal 1 for the first test case in a section, n equal 2 for the second test case in a section, and so on.

[0039] The generic test case section of the MIB is illustrated below. This shows the test case set up, test case execution structure to the test case.

[0040] For the test case set up and execution, the “string” is a command line, with input fields, that executes a windows MT program.

[0041] The voice test case section of the MIB is as follows. Expected DNs called are:

[0042] a) call sender,

[0043] b) input ACD queue,

[0044] c) individual IVR, and

[0045] d) agent ACD queue.

[0046] Table 5 below illustrates some generic voice test cases.

[0047] Table 6 shows an MIB for an agent call. This MIB provides information to test a case where an IVR input string results in a call to an agent and some data function, such as a screen pop, in addition to the call.

[0048] The transaction section of the MIB is set forth below in Table 7.

[0049] For a more thorough understanding of the utilization of SNMP and MIBs, reference is made to the text TOTAL SNMP: “Exploring The Simple Network Management Protocol”, by Sean J. Harnedy, Prentice Hall 1997, ISBN 0-13-6469949; and “Understanding MIBs”, by David Perkins, Evan McGinnis, Prentice Hall, ISBN 0-13-437708-7, the disclosures of which are incorporated herein by reference.

[0050] To ensure that customers of the ACD system are satisfied with the service level they are receiving, the monitoring system tests the services in the same way that a customer will use the service.

[0051] The ACD system is monitored in a 6-step process:

[0052] 1) Define exactly what services are to be to monitored,

[0053] 2) List all the components, both hardware and software that are needed to provide the services,

[0054] 3) List the number of ICONs needed to represent the components on the Graphical Display of the service,

[0055] 4) Define the points and the components that should be tested,

[0056] 5) Define the tests that should be run in order to verify that the components are working properly, and

[0057] 6) Write the test routines using MIBs and test cases which the MIBs will use to verify that the systems are working properly.

[0058] These steps are discussed in more detail below.

[0059] The first part of the monitoring procedure is a planning process. A list is made of the services that are to be monitored. It may be desirable to monitor only a subset of the overall service package, or all the services can be monitored.

[0060] Once the services to be monitored have been listed, it is necessary to identify the steps a customer performs to use the service (dial a certain telephone number, enter password, identification, et cetera). A definition is also needed of what constitutes successful service delivery—who is the customer, and what service is expected.

[0061] Next, the various subsystems used in providing each service are identified. Systems interdependencies should be identified as well. Each node should be identified, along with the test cases and each test case's parameters, plus details as to what constitutes a successful test.

[0062] By identifying the services and the various system elements that are necessary to provide each service, information is compiled to simulate a customer transaction. What constitutes a passing indication for each subsystem and for the overall service transaction is also identified.

[0063] For example, consider a service which provides customers with voice mail when the access number is dialed. To test this service, it is necessary to dial through the telephone network to the voice mail system, and attempt to retrieve a voice mail message. The information required to write the test procedure includes the telephone number to dial, a method of confirming that the telephone was answered, the voice mail mailbox number, the password, and a message stored at the test mailbox. The test case list would include two tests: (1) dialing into the system and (2) message retrieval. If either test fails, subsequent tests are needed. Did the dialup portion of the test fail? Why? Some failures may be line busy, no dial tone, or ringing existed but the distant end failed to answer the call.

[0064] Additional tests will be needed to discover why voice mail retrieval failed. Did the system accept the password? If the voice mail system accepted the password, did it fail only once (perhaps due to a bad connection) or did it fail repeatedly (possibly the voice mail program failed, or was the test equipment unable to access the mailbox because the access database table was changed.

[0065] A passing indication is obtained when the test equipment dials into the system and successfully retrieves the test voice mail message from the voice mail system.

[0066] The probe points 46 may be placed at subsystem boundary points—for example, a gateway, a network access point, or a network transport system. Probe points may also be placed within an application system as the transactions or remote procedure calls (RPCs).

[0067] Boundary probe points define the subsystem which has failed. If tests to a particular probe point fail, but network subsystems up to that point have been proven to be working, the problem is likely to be found in the subsystem which failed the test.

[0068] Internal probe points define the part of an application or subsystem which has failed. These probe points are useful after a test to a boundary probe point has failed, because they provide additional diagnostic information. As one example, if boundary probe points show a mainframe application gateway is not returning requested information, running a test to an internal probe point could help determine that the problem is not the gateway itself, but rather one or more application end points which are failing to report data received.

[0069] Service assurance tests can verify that the systems are delivering the services expected by the customers. Defining these tests requires use of the information already gathered—in particular, the definition of what constitutes a successful customer transaction, and what a customer needs to do in order to obtain the service.

[0070] Diagnostic tests are required after a transaction failure has been detected. Using the information identified above, each boundary point is tested to determine which subsystem has failed. If the system is tested in a sequential manner, from beginning to end, the first boundary point which fails the diagnostic test must contain the faulty subsystem.

[0071] Once the faulty subsystem has been identified, running tests to various sub-elements within it will identify which parts or components have failed.

[0072] The diagnostic rules are applied at the enterprise automation level. For example, if a network data path has failed, it is likely there will be several other tests which fail. The function of the enterprise automation layer is analyze the results of the test, and isolate the most likely causes of the failure.

[0073] Consider the example in FIG. 1, where an application on the server 38 has failed. If a sequence of tests is initiated via the PSTN 20, dialing into the IVR 24, all subsystems up to and including the agent position telephone 28 and data screen 30 will pass the test. However, all systems from the application server 38 to the mainframe 32 will fail the tests, because the test requests cannot pass through the failed application on the server 38. The test equipment in conjunction with an enterprise automation package will identify the failed component and facilitate the repair process by eliminating guesswork in identifying the point of failure.

[0074] Once a service assurance test sequence shows a system component has failed, the test equipment in conjunction with the enterprise automation equipment begins a series of diagnostic tests to determine exactly where the failure has occurred. The diagnostic rules associated with the enterprise automation equipment determine the source of the trouble.

[0075] As an example, suppose that tests have just been completed on the application server 38, and the CTI server 36 has passed its tests. While tests are being performed on the CTI server 36, the application server fails—hence, the CTI tests report a failure. The diagnostic rules then run a series of diagnostic tests to determine the source of the failure, and optionally route a trouble ticket to the appropriate operations group.

[0076] As noted above, the traffic driver/checker 44 is designed to work with an SNMP network management system, such as HP OpenView 40. After a fault condition has been identified, the enterprise automation package has the option to follow a predefined rule to issue a trouble ticket to the operations group best able to resolve the problem. When the trouble has been resolved, the technical personnel can clear the ticket. By enforcing a policy requiring the technical staff to clear the ticket with operations personnel, a complete record of the trouble and its resolution becomes available at the management level.

[0077] The MIB (Management Information Base) is a primary component of the monitoring system. The MIB allows one to define precisely what tests that are to be run, in which order, and what response is expected from the test. The test equipment will execute the MIB step by step, and will return a pass or a fail condition to the monitoring system, such as HP OpenView 40.

[0078] The MIB structure is essentially a collection of linked lists and databases, which provide information about the tests to be run, the order in which they are to run, and the circumstances under which they are to run. The first MIB identifies the test case lists according to ASN1 numeric names by the module name, otherwise also known as object identifiers (OID). Table 8 below shows an assortment of different modules which are contained in the MIB, including several different test cases, lists of test cases to be run, as well as some management modules.

[0079] The test equipment provides a “User Test Case table” which is integrated into the MIB structure. Using this user-defined test case table, external programs can be defined and run under conditions set by the user.

[0080] The traffic driver/checker 44 has four sets of tables visible to the user.

[0081] The four sets of tables are:

[0082] 1. The Node Table

[0083] This contains a list of the node is the system, examples of components are: the PSTN, IVR pool, Call Center Agents in a location, and Mainframe Transactions.

[0084] 2. The Test Case Lists Table

[0085] This is a list (sequence) of test cases. Each test case in the list is to be run before the complete test list is passed. A sequence of test cases will stop on the first failure, or if a test case does not respond within the specified timeout.

[0086] A timeout of zero means the test case has no timeout. This is used for long running tests (for example and IVR port monitor), that may report a failure at a later point in time.

[0087] 3. The Test Case List Table

[0088] This contains the run time information for test cases for a specific test list. This includes test case result (pass, fail, or timeout), the test case result (a text string), and the test result information from all previously successful test cases in the test case list.

[0089] 4. Tables of test cases with similar parameter syntax, but different parameter values.

[0090] These are the tables of test cases. These contain the static information used to run a test. Static information includes, telephone number to dial, program to run, and timeout for the test.

[0091] Various user-selectable options are available by clicking on areas of the user screen interface of the traffic driver/checker 44, as follows:

[0092] Database:

[0093] Initialize

[0094] Expunge

[0095] Promote

[0096] Exit

[0097] Testing:

[0098] Components

[0099] Test Cases

[0100] Help:

[0101] File—Load

[0102] Testing—Nodes

[0103] Testing—Test Cases

[0104] Test Case Running

[0105] About Service Meter

[0106] “Initialize” is used to initialize the test system.

[0107] “Expunge” deletes system component entries which have been previously marked for deletion.

[0108] “Promote” moves the contents of the database from the “test” test apparatus to the production systems.

[0109] The “Exit” option terminates the test program.

[0110] At the “Testing” menu, one may select either the “Components” section or the “Test Cases” option.

[0111] When the program is first started, the only choice under either option is “Load.” This loads any existing database entries for either the system component list or the test case list.

[0112] Add Node allows the user to add a new test location of the Components section MIB.

[0113] Add List permits the addition of a new test case list to the sequence of tests to be run. A new test case list can be added to either a newly added node or to a node which already has test cases defined. Of course, the list may also be added by clicking on the “Add List” menu choice.

[0114] Clicking on “Load” loads information from the database for use by traffic driver/checker 44.

[0115] “Delete” erases any test case or node which has been previously marked for deletion.

[0116] “Undelete” allows restoral of a previously-deleted test node, in case the node was deleted by mistake.

[0117] The first three functions have the same uses as previously discussed under “Components,” viz. deleting, loading, and undeleting test cases.

[0118] The fourth function, “Duplicate,” allows the user to copy a previously-defined test case, and make a new test case by changing some of its parameters. An example of this feature would be performing the same series of tests on a different IVR in another city. Duplicating the test case and changing the telephone number to be dialed will create a new test case without the need for re-typing all the other information.

[0119] The “HELP” files contain sections useful to the user in operating the traffic driver/checker 44.

[0120] FIG. 3 shows the administration screen display which illustrates the description of other clickable areas.

[0121] 1. An item is shown as selected by a color change of the first image in the item.

[0122] 2. The test cases in a list can be moved by drag and drop, by clicking and holding the left mouse button on a test case. In the upward direction, the test case inserts before the test case it is dropped on. In the downward direction it inserts after the test case it is dropped on.

[0123] 3. Clicking on the second colored bitmap invokes the test case editor for the test case. Because there is only one copy of a test case (a row in its test case table), changing the values in the test case editor change the values for this test case in every tree view leaf.

[0124] 4. The OID of an item is show by clicking with the right mouse button.

[0125] 5. By clicking on the plus or minus bitmap one can collapse or expand the tree.

[0126] Test Cases Tree View

[0127] 6. Clicking with the left mouse button on the second bitmap in a test case tree view leaf invokes the Test Case Editor. Because there is only one copy of a test case (a row in its test case table) changing the values in the test case editor change the values for this test case in every tree view leaf.

[0128] 7. Clicking with the right mouse button, or click and holding the left mouse button on the first test case leaf bitmap enables one to drag the test case and drop it into a Test List.

[0129] FIG. 4 shows the database promotion screen display. This screen display enables one to change the destination host and port at which the traffic driver/checker 44 is responsive. For correct operation, for example, the traffic driver/checker 44 is responsive on port 161 (the standard SNMP port). In the preferred embodiment, the traffic driver/checker 44 cannot run concurrently with other SNMP application in a single Windows NT machine.

[0130] The single step box is useful to trickle promotion traffic to the traffic driver/checker 44. If checked, another button labeled Step is made visible, and clicking on this button promotes the contents of one traffic driver/checker 44 table at a time.

[0131] The test case editor of FIG. 5 shows the parameters for a test case. There are two parameter data types, string and integer. For string data there are two sub types of string, internal which is less than 255 bytes long, and external string types that consists of a file and its associated file name.

[0132] The check box next to the string name indicates if the string is internal or external. When clicked to check the box, a browse menu is prompted to select the file name of the file that contains the external string.

[0133] For integer data the valid range defined in the MIB is shown. Values outside this range are not allowed.

[0134] The box labeled Test Case Mode sets the mode of the test case. Simulation modes are Timeout, Pass and Fail. The Run modes are Learn and Test.

[0135] Test Case modes are learn and test. The Learn mode is used when running a test case to discover how it reacts to a know good component. In learn mode the result should be recorded by the test case so that is can determine a pass or fail when running in the test mode. The Test mode is used to run a test case against a component to determine if the component is working.

[0136] One special result is possible when running in the test learn, or test mode. This is a fail case, and occurs when the test case event in the test case OCX is unassigned (no entry address). The traffic driver/checker 44 considers this is a test failure, and this result may be attributed to an implementation error.

[0137] FIG. 6 shows the Test Executive screen display of the traffic driver/checker 44.

[0138] On the left side of the screen display are a number of SNMP counters. In normal operation the error counters are preferably at zero.

[0139] The community field shows the last community name received. If there is no record of a community name, the SNMP request is not processed. To add a community name, either accept the name in the community field, or type the desired name and press the “add” button. Similarly, one can remove a name with the delete button.

[0140] The reminder of the fields show the last OID in and out, and the SNMP data packet. The format of these packets is described in the text “Understanding MIBs” identified above.

[0141] In addition to the foregoing information, a log file is prepared. Errors are recorded. All messages can be recorded by selecting the messages button. Log messages are written in either a hex version of SNMP packet format, or for errors, in plain text.

[0142] FIG. 7 shows the Program Control screen display. This screen display shows a Test List with a test result of “Test Complete: Timeout”. The test case with green bitmaps respond with a “pass”. The test case with the black bitmap timed out, and Program Control has stopped the test case. The test cases with the yellow bitmap are “ready to run”, but cannot run because of the test case timeout. Each test case is a separate windows task that communicates with the program manager through a UDP socket.

[0143] One can run a test case list from the Program Control Window. This is accomplished by clicking the right mouse button on the bit map before the Test Case List name. This feature is intended to test the Test Cases without having to poll the traffic driver/checker 44 run time from an SNMP manager.

[0144] A test case can be canceled by night clicking on the test case icon in a test Case List. This sends a stop message from program control to the Test Case. If the Test Case does not stop within 30 seconds, Program Control will timeout and destroy the Test Case process.

[0145] Test cases are built in the following manner. Before building the test cases for a system, a review of the particular system is needed. The goal for the review is to provide all the information needed to build the test cases. The review is preferably a four-step procedure, as follows:

[0146] 1) Analyze the services provided to customers. The analysis should clearly identify what constitutes a working system and how the customers determine whether or not the service is working.

[0147] 2) List the components of the technical systems used in providing the service, and what each system must do to successfully deliver service to the customer. This creates the basis for defining a series of pass/fail tests which will be run using the traffic driver/checker 44.

[0148] 3) For each system, list how it is known whether the system is working, from the customer's viewpoint. This will define the normal operation for the system.

[0149] 4) For each system, determine the conditions which characterize whether the system is succeeding or failing in its assigned task. These pass/fail conditions will inform the traffic driver/checker 44 whether or not the system is fulfilling its assigned role in providing service to the customer.

[0150] Once the foregoing information has been compiled, building actual test cases becomes a three-step process:

[0151] 1) Create a MB to define the test process and to tell the traffic driver/checker 44 the order in which tests are to proceed.

[0152] 2) Create a series of test cases (test case lists) which determine whether the system passes or fails the test. These test cases may include any combination of tests defined for the systems.

[0153] 3) Create a script which runs a single test, for each aspect of the system which is to be monitored. Collections of these test scripts are used to verify that the systems are operating properly.

[0154] FIG. 8 illustrates the interrelationship between the MIBs, the test case list and the individual test cases.

[0155] The following is an example of an MIB. A user of the test equipment can add additional MIBs and test cases.

[0156] Each test case (row in the test case table) represents a specific test case. Each test case in a test case table has the same syntax parameter list (each parameter is a column in the test case table). Each test case in the table may have different run time parameters, including the name that the program uses to run the test case.

[0157] Each test case table has a unique number, namely a part of the MIB OID that identifies the test case table. In the example below, the unique number is defined by the clause {sigModules 7} in the MODULE-IDENTITY section of the MIB definition.

[0158] To build a new test case, the names in this section are changed to the new test case name. (The text identified above and entitled “Understanding MIBs” can be utilized to obtain a explanation of the ASN syntax required to make this change).

[0159] This section of the MIB Is Preferably Required for all test cases.

[0160] The test case MIB is added to the compliance section of the MIBs included, and to the capabilities part of the MIB having the proper file. In addition the new MIB name is added to the include file (sys-mib.inc ).

[0161] The MIB is then complied with the m2.bat file provided in the MIB subdirectory. At a DOS prompt enter the command m2 (which contains the MIB compiler command line) is entered. When the MIB complies correctly, the output of the MIB compiler (sys-mib.imf) is loaded into the Database Load menu option.

[0162] Tests of systems can be carried over into multiple levels. System-wide service assurance tests can be built using two different methods: centralized and distributed testing. Each method has its advantages and disadvantages.

[0163] In the centralized model, every test is run from a single system, the centralized test unit tests every other node within the system. This method makes test control easy as each test is scheduled from a single subsystem; it is likely the best method for small system configurations, where all the components are in a single building, or on a single LAN segment. This configuration may be sufficient for networked systems with two or three interconnected LANs.

[0164] The disadvantage of the foregoing test implementation arises from the nature of networked systems. If a link between the test system and other networked computer systems fails, it is difficult to test whether or not the other LAN segments are functioning proper, because the target system cannot be reached. The link between the test system and the other subnetworks must be repaired before useful testing can continue. This disadvantage can be reduced significantly if more than one link between the test system and the computer subnets can be made available.

[0165] Distributed testing implementations require a test system in each major area of the network. A business with several distributed offices may want to be able to test within each major city, and aggregate the reports to a centralized office for trouble ticket handling. Subnets can be tested locally, reducing the likelihood of link failure between the test system and the applications to be tested. Redundancy is also provided, because more than one test unit is resident on the network. In the unlikely event of a test system failure, another system can resume testing for the failed unit.

[0166] Each individual application on each system node is tested on a pass/fail basis. When the application fails, the test apparatus of the invention performs diagnostic tests on the node to determine the location of the failure, and whether the fault is likely to be hardware or software related. Each node which is involved in providing customer service is fault tested, on a repeating basis to ensure that customer service is being provided in a satisfactory manner.

[0167] The test equipment provides node-by-node diagnostic tests when required, to analyze a reported test failure. The node-by-node testing is performed under command of the enterprise automation system.

[0168] MIBs are addressed by way of Object Identifiers (OIDs).

[0169] An example of an OID is: 1.3.6.1.4.1.2392. 2. 5. 2. 2. 1. The prefix, 1.3.6.1.4.1 refers to an SNMP MIB. The number 2392 is the test apparatus MIB enterprise number. The heart of the MIB numbering scheme is: 2. 5. 2. 2. 1.

[0170] The following Table 9 illustrates the OID table numbering. In general OIDs represent a static committed interface between MIBs users. Within the static definition of elements addressed by OIDs are dynamic tables. The dynamic tables have rows that are implementation, both for internal use by the test apparatus and the test case Components and Test Case Lists, defined above. The test apparatus makes extensive use of dynamic tables.

[0171] When the test apparatus of the invention first runs, it accesses the MIB, and performs an initialization routine which “walks” through the MIB and creates all the test cases necessary [as defined in the MIB]. The test apparatus does not run the test cases until commanded to do so; however, it is ready to run the tests due to the setup/initialization routine it performs. This process is referred to as MIB discovery.

[0172] As noted above, MIB has dynamic tables contained therein. The Object Ids of these tables row change as Components or Test Case Lists are deleted and the database expunged. When the database is expunged, and promoted from the Administration to the Executive routine, each item in the MIB must be rediscovered.

[0173] In addition, the OIDs of Components is dependent on their position in the test case list table. According to the preferred embodiment, new components and test lists are added at the end of the table.

[0174] The Component and Test Case List tables should be carefully prepared and built if automation is being utilized. The Component, Test Case List and Test Case run time OIDs are the way that automation package understands what an individual test case list is testing. These OIDs will change if there is a deletions in the MIB tables and the MIB database is expunged.

[0175] The functions of the test apparatus is operated under control of an enterprise automation package, such as HP OpenView 40. The enterprise automation system commands the test apparatus to perform one or more tests on a repeating basis. These commands, which cause the MIB to run a test case and provide a result to the automation system, are considered to “poll the MIB.” The traffic driver/checker 44 advises the automation system whether the target device passed or failed the test.

[0176] The SNMP Manager polls (issues and SNMP GET request) the Test Case List state element whose OID is: 1.3.6.1.4.1.2392.2.5.2.2.x.4, where x is the number of the Test Case List, and ranges from 1 to 999. The upper limit of 999 is a traffic driver/checker 44 restriction on the maximum number of rows in a table.

[0177] Polling the Test Case list result can return any one of seven states. When the SNMP manager reads the state, two actions are taken. The first is to return the state. The second is to advance the Test Case List to its next state.

[0178] The Test Case Lists contains a memory for diagnosing failed tests. When a test fails, the failing tests' run time OID is placed in the Test List at OID 1.3.6.1.4.1.2392.2.5.2.3.x.6, where x is the number of the Test Case List.

[0179] The table row referenced by this OID contains information from the failing test (if the test is well behaved) at the OID “failed_test_case.11. This table element is a 255 byte character string. The contents of this string are test case dependent, and are mostly defined by the test case designer.

[0180] There are three test failure results generated internally by the traffic driver/checker 44. These are:

[0181] 1. Test Fail: Timeout.

[0182] 2. Test Fail: Simulator.

[0183] 3. Test Fail: No test program assigned.

[0184] The states, the color of the state in the Program Control tree view, an explanation of the states meaning and the next action taken when the state is read is contained in the table, Table 10, below.

[0185] It is possible to change (write to) the MIB with the SNMP management system, with the exception of the OID 1.3.6.1.4.1.2392.2.99, which is used to restart the Executive.

[0186] Automation packages are rule-based systems; that is, they run tests and use a specified rule to determine whether the response passes or fails to meet the criteria. The rules embody the requirements passing the test, and identify the meaning of the result received.

[0187] An example of a rule affecting the interpretation of test results is “The ABC system will be out of service for maintenance every day between midnight and 2:00 a.m.” Clearly, any test which runs on this system during the period from midnight to 2:00 a.m., will fail every time the test is run. It may be decided that the way to implement this rule is not to run tests on applications dependent on the ABC system during the hours of ABC's scheduled downtime. Thus, in this case, the enterprise automation package may be defined so tests are run only during the remaining 22 hours of each day.

[0188] When a service assurance test fails a predefined number of times, the traffic driver/checker 44 uses information obtained from the test results to run a series of diagnostic tests that determine which subsystem failed. The information is passed back to the enterprise automation package. The enterprise automation package can then use the information to initiate a trouble ticket and route the ticket to the correct technical group for repair.

[0189] The following Visual Basic program is an example of a simple test case. The six events that must be consider are: Form Load, Form Terminate, OnEndTest, OnSet Parms, OnSetup and OnRunTest.

[0190] In the example shown, Form Load sets the Test Case Caption. The port number issues to distinguish one test case from another. The port number is in the 3000 range. The port number is built from adding 3000 to the unique test case value in the Test Case List MIB table (OID 1.3.6.1.4.1.2392.2.5.2.3.x), where x is the unique row value of the test case in the Test Case List table.

[0191] OnEndTest is a message sent from Program Control to the Test case. Both this event and Form Terminate mean end, free all of the resources and end. If the test case does not end after receiving an OnEndTest event, Program Control waits for a test complete event from the test case for 30 seconds. If this is not received, Program Control cancels the test case process.

[0192] The OnSetParms event occurs the same number of times there is a test case parameter in the MIB. In the preferred embodiment, there are a maximum of 16 parameters allowed for a Test Case (a Test Case Editor limitation). In the following sample the parameter strings are stored in a string array. MIB values with suffixes 8 to 16 are sent to a test case. Other MIB parameters (1 to 7) are used by the Program Control.

[0193] OID: is the unique test case number in the test case list.

[0194] Parm: is the parameter value, always a string.

[0195] ParmType: Internal for up to a 255 byte string. External is a file, whose file name is contained in Parm.

[0196] The OnSetupTest event occurs twice. This event occurs First time when all parameters have been sent to the test case. At this point the test case should set up the run time environment.

[0197] A second time of the event occurs with an internal string that contains information from previously successful test cases. This string is used to pass information between test cases in the Test Case list. It contains concatenated information from the:

[0198] TestDataType, TestData and TestResult

[0199] OnRunTest: Run the test case. There are no parameters.

[0200] OCX properties.

[0201] There are three test case properties, as discussed above.

[0202] TestDataType:

[0203] Internal_text only.

[0204] TestData:

[0205] A short string that contains information to be passed to subsequent test cases. The aggregate length of this string from all test cases in a test list is 255 bytes

[0206] TestResult

[0207] True if the test passed. False if the test failed.

[0208] The Program Control of the traffic driver/checker 44 can run and terminate a test case without the use of an SNMP Manager. The right mouse button click on the test list bitmap in the Program Control windows runs the test list. The right button clicked on the test case bitmap stops the test case.

[0209] The Test Case OCX provides a Writelog method. This, in conjunction with the logLevel property controls the OCX bases test case log. The OCX can log all messages between Program Control and the Test Case. The logLevel is set to all to collect all messages.

[0210] The logCount property controls how many messages are written into a log file. The logCount value default is 4096.

[0211] For example:

[0212] LogLevel=all

[0213] Call writeLog( info, “This is an information message”)

[0214] Or

[0215] LogLevel=error

[0216] Call writeLog( error, “This is an error message”)

[0217] The following is an example of the Visual Basic Test case.

[0218] In summary, the Management Information Base (MIB) is the basic building block for Enterprise Management. The MIB identifies the component applications in the system, associates each component with one or more test cases, and directs the flow of tests over the network or subnet to be tested. The MIBs are used to control the flow of testing, as well as to pass information from the SNMP agent back to the traffic driver/checker 44.

[0219] The MIB structure uses databases and lists; some lists are used as indexes; others are cross-references between the MIB and the “human names.” An example is the name “TestCaseListGroup” —an ASCII string which is more meaningful to humans than “1.3.6.1.4.1.2392. 3. 4. 1. 3”,the Object ID belonging to TestCaseListGroup.

[0220] While the preferred embodiment of the invention has been disclosed with reference to a specific test apparatus and methods of operation thereof, it is to be understood that many changes in detail may be made as a matter of engineering or software choices, without departing from the spirit and scope of the invention, as defined by the appended claims.