Expert System Tester

1. Field of the Invention

The present invention is directed to an expert system tester that allows extensive expert system rulebased testing without the test program having any knowledge of the particular rulebase being tested and, more particularly, provides a practical regression testing method for analyzing rulebase changes after a modification has been made which is cost effective and significantly raises rulebase quality.

2. Description of the Related Art

It is common for expert system rulebases to typically exceed a thousand rules and in such systems sensor inputs commonly number in the hundreds. If the sensors are digital sensors providing one of two input values, the number of possible combinations of input values is enormous and testing all combinations is practically impossible. When analog sensors are involved, which have an infinite number of possible input values, the testing problem becomes even greater. During normal development of new rules for a large rulebase, normal debug type of production rule testing by the knowledge engineer is performed. In such a situation when the knowledge engineer produces a new rule, the engineer exercises the system using simulated plant data having values over the expected range of sensor values for the sensors examined by the new rule to verify that the new rule is operating as expected, that is, producing the diagnosis expected when expected. In such a situation, sensor values other than the sensor values used by the new rule are kept at normal values. As a result, production type testing does not determine the effect of other sensors on the new rule or the interaction of the new rule with the remaining rules. A second type of testing is to place the expert system online with real, continuous plant data and have the knowledge engineer carefully examine the diagnoses made to determine whether the diagnoses were intended. It has become apparent that more effective techniques and tools are needed for operational verification of large rulebase and input number systems.

It is an object of this invention to provide diagnostic rulebase verification without having to utilize any knowledge inside the tested rulebase to initialize and exercise the test tool.

It is another object of the present invention to provide a realistic regression test system for an expert system that has the capability of exercising all combinations of input data types.

It is a further object of the present invention to provide an expert system tester which depends on specialized test cases to exercise sophisticated diagnoses.

It is also an object of the present invention to provide a test system which will determine the effect of all sensors on a new rule and the interaction of all rules with the new rule.

The above objects can be accomplished by an expert system test system which has the capability of exercising all combinations of data input from specialized data test sets. The system includes a test plan which is used by the testing system to guide the iterative application of the specialized test data to the expert system to produce outputs for each iteration which are compared to expected results or to a baseline output produced with a normal data test set. If the expectations are not met, the system produces an exception report which allows the knowledge engineer to determine whether the exception is abnormal. An output log of the test is produced which allows regression between rulebase changes to be determined. The system includes a test language which allows the user to formulate a test plane without the need for complicated programming associated therewith.

These together with other objects and advantages which will be subsequently apparent, reside in the details of construction and operation as more fully hereinafter described and claimed, reference being had to the accompanying drawings forming a part hereof, wherein like numerals refer to like parts throughout.

FIG. 1 illustrates the components, inputs and outputs of the present invention;

FIG. 2 is a flowchart of the operation of a first order test in accordance with the present invention;

FIG. 3 illustrates how a first order test can be modified to produce a second order test;

FIG. 4 illustrates data flow when the present invention is used with a preferred expert system; and

FIG. 5 illustrates the execution sequence of the preferred expert system and the present invention.

Regression testing is a method for analyzing rulebase changes after a modification has been made. Two kinds of changes are possible: (1) changes that were intended, and (2) inadvertent changes that regress other unrelated aspects of the rulebase. Regression testing is important because changes and error corrections may introduce more errors than were created in the original coding of the schemata of the rulebase.

A complete regression test exhaustively exercises the rulebase through all the possible combinations of data input and diagnosis scenarios. From a practical viewpoint, this is not possible because of the time needed to perform a complete test. A more realistic regression test exercises all combinations of data input types and depends on specialized test cases representing each type (a subset of an exhaustive test) to exercise sophisticated diagnoses.

The testing system or regression analyzer of the present invention addresses these practical test requirements. The present invention accepts as input a production quality rulebase, a sensor data set, and a test plan. For each data combination a normal diagnosis is run using "normal" data which will not cause any diagnosis to indicate an abnormality such as an alarm. The normal diagnosis is followed by an iteration with perturbed data. The normal and perturbed test cycle can continue until all combinations have been tested. The system produces a test log file and a separate exception list of possible inadequacies, as defined by the test plan. The analyzer can be run interactively to perform simple tests, or in batch mode overnight to run extensive tests. The log and exceptions can be compared between rule changes to determine if regression has occurred.

The regression tester 10 of the present invention, as illustrated in FIG. 1, interacts with an expert system 12 which includes a production rulebase 14 and an expert system inference engine 16. It is preferred that the expert system inference engine be the Processor Diagnostic System (PDS) available from Westinghhouse and described in U.S. Pat. Nos. 4,644,479 and 4,649,515. A description of the operation of the preferred expert system can be found in Kemper and Harper, The PDS Manual, PDS version 5.1, Diagnostics, Westinghouse Electric Corporation (1987) incorporated by reference herein. The regression analyzer includes a test plan 18, which is created using a test plan language, to be described in more detail later, and which is used by the test system 20 to access normal and test data sets 22 that are applied to the inference engine 16. The inference engine performs an analysis (diagnoses) using the production rules in the rulebase 14 producing outputs which are stored in a log file 24 by the test system and compared to expected results to produce an exception report 26. The system also allows the output to be displayed on the CRT screen 28. It is preferred that the present invention be implemented on a Digital Equipment VAX 8000 Series computer running a VMS operating system. It is also preferred that the present invention be implemented in a language such as "C" suitable for a structured design approach as discussed later herein.

As will be discussed in more detail later, the present invention is capable of performing first through sixth order specialized expert system tests. One of skill will of course recognize that as processors get faster even higher order tests will become practical. FIG. 2 illustrates the general operation of the present invention during a first order test. The present invention starts 40 by reading 42 the test plan 18. The test plan 18 is created using a specialized test plan language which will be discussed in more detail later. The test plan language produces a regression test plan 18 which is stored in a text file and read 42 into the test system 20. Next the production rulebase 14 is loaded 44 followed by loading 46 of the test data. Next a pointer to the sensor whose value is to be changed is set 48 and then all sensors are set to a normal value and all are updated 50. The setting of all sensor values to a normal value applies a complete normal data test set to the expert system. The update operation causes the inference engine 16 of the expert system to recognize that a new sensor value is available. In this way in step 50 all the rules are flagged for firing or testing.

Next, the rules in the expert system which have new (updated) sensor values are fired and continue firing until no further changes (firings) occur. If this is the first cycle 54 of the test, the results of the normal sensor data expert system diagnoses are saved 56 as a baseline. Next one of the sensors is set to a test value and updated 58, followed by firing 60 of the supported rules, that is, the rules which use the new sensor value. The output of the expert system (updated malfunctions, preferably with a positive confidence) is then written to a log file 24. A complete output data set is produced because all input values are applied to the expert system. The output is next compared 64 to the test plan exceptions and if there is a match, an exception report is written 66. The pointer is then updated 68 and tested 70 to determine if the end of the sensors has been reached. If not, the cycle is repeated, otherwise, the system stops. As can be seen from FIG. 2, the system cyclically applies the normal test data and the specialized test data and iteratively (one sensor value change at a time) applies the specialized test data between cycles. That is, a test value is provided for all the sensors, thereby providing a complete specialized test set.

FIG. 3 illustrates the general operation of the present invention when a second order test is being performed. This requires the addition of a new pointer 80 so that two sensor values can be changed from a normal value to a perturbed value. To determine whether this is the first cycle of the test, both pointers must be tested 82 before the results are saved 56. Because two sensor values are now being changed both sensor values must be updated 84 before the rules are fired 60. Because an additional pointer is included, the system must update 86 this pointer and test it to determine whether it has reached the end of the sensors. As can be seen by comparing FIGS. 2 and 3 to provide additional test order capability, the present invention merely needs to provide additional sensor pointers, appropriate sensors update steps and pointer increment and test loops. Appendix IV illustrates how such can be provided for a sixth order test capability in a structured design approach.

The sensor data or data sets 22 input to the regression analyzer or system 20 of the present invention are preferably in standard PDS form. Seven data sets, or in PDS terminology, timesets, are required for sixth order test capability. These timesets can be created using PDS. The sensor values can be logical or numerical as needed.

The first timeset should be a normal set of readings. This set should produce confidence factors in the PDS system between zero and minus one. This data is used as a benchmark or baseline to compare with all other conditions.

The next six timesets represent the data produced from abnormal equipment operation. Generally the abnormal timesets or data sets should be values taken on both sides of and at the boundaries for different condition levels of the expert system. That is, the test set should provide a test value for each diagnostic level of the system to be tested, for example, in a power plant the levels are predicted, diagnostic, warning, shutdown and sensor failure. For a power plant these values can be obtained from the alarm levels of the plant. The examples used int his description will be for a power plant even though other types of data sets, such as economic data could be used.

The second timeset has readings that are expected when each sensor is under range or failed with a low value. These readings should generate at least one sensor diagnosis for each failed sensor and the confidence factor for the sensor rules in the PDS system should be positive while all others are negative. The third timeset has readings that are expected when each sensor is over range or failed with a high value. These readings should generate at least one sensor diagnosis for each failed sensor with one confidence factor condition in the PDS system positive and all others negative. The fourth timeset is used to define readings that have a predictive value in the rulebase design. That is, the predictive level sensor values fire rules which predict likely consequences of continued operation under such conditions. This category is intended to provide test data at a level before diagnostic alarms are activated. The fifth timeset has readings that trigger diagnostic PDS alarms. These readings should generate low level, positive confidence in at least one malfunction diagnosis. The sixth timeset has readings that trigger protection I PDS alarms. These readings should generate moderate level, positive confidence in at least one malfunction diagnosis. The seventh timeset has readings that trigger protection II PDS alarms. These readings should generate high level, positive confidence in at least one malfunction diagnosis.

The regression test plan 18 in the present invention is defined in a text file and read into the test system 20. The plan is composed of tests that exercise the rulebase with one, two, three, or up to six combinations of sensor values changed with each cycle. As previously illustrated, for example, a first order test cycle runs a normal reading through the rulebase, followed by using the same set of normal readings with one sensor value changed to a trip reading. The entire first order test repeats the normal/perturbed execution for every sensor. The plan definition can also include exception report specifications for each test. An exception is a request to record any instance of a condition. For example, a useful exception to include in the above first order test is to report if there are no malfunctions with confidence greater than 0.5 resulting from one of the test cycles. Exceptions should be designed in a way to signal regression test failure by the existence of exception reports.

The first line of the test plan file preferably contains the directory specification for where the rulebase files reside. Each of the following lines preferably has either a test definition or exception definition. Exceptions following a test definition apply only to that test.

A single regression test is preferably specified by a set of sensor categories surrounded by parentheses. For example, a first order test of the trip data, followed by a second order test of the warning and trip data is given by:

(Trip)

(Warn Trip)

A plan with this test sequence designates a first order test which will cause the rulebase to be executed for every sensor trip value, and the log file will record all the updated malfunctions and procedure results. This is followed a second order test performed by executing the rulebase for every trip value, while each sensor is set in turn to the warning value. Because there are no exceptions defined, no entries are made in the exception report 26.

Exceptions are divided into three categories providing three basic filters: 1. Update exceptions; 2. Level exceptions; and 3. Change exceptions. Update exceptions test to see if appropriate malfunctions and procedures are updated by a diagnosis cycle. For example, the exception to report any updated malfunction is:

Exception: Malf update.

Level exceptions compare the resulting malfunction or procedure parameter with a fixed value. For example, the exception to report when a malfunction confidence is greater than 0.5 is:

Exception: Malf CF>0.5.

Change exceptions compare the diagnosis percent change with a constant. The percent change is computed using a conventional formula: (newvalue-oldvalue)/oldvalue. Oldvalue is the malfunction or procedure parameter of the normal diagnosis saved in step 56 of FIG. 2. Newvalue is the same parameter that results from the second diagnosis (58 and 60) of the cycle. The effect is that a positive change is away from zero and a negative change is toward zero. For example, the exception to report at least a positive ten percent change in a procedure confidence is:

Exception: % Proc>10.0

There are three classes of exceptions providing three variations on the three basic filters: 1. Simple exceptions; 2. Set exceptions; and 3. Sufficiency exceptions. Simple exceptions are like the category examples shown above, where the exception is composed of a parameter, an operator, and constant. Each time a simple exception tests true, the test definition, perturbed sensors, malfunction or procedure that triggered the exception and parameter values are logged 66 in the report. Set exceptions are simple exceptions preceded with the qualifier ALL, SOME, or NO. In this case a report is generated if the set composed of the exception parameter matches the exception definition. If the exception tests true for the entire test, the test definition and perturbed sensors are reported 66 in the exception report. With the SOME set qualifier the number of elements in the set is also entered 66 in the exception report. For example, the exception to report when some of the malfunction confidences in a regression test are greater than zero is:

Exception: Some Malf CF>0.0

Sufficiency exceptions are simple exceptions preceded with the qualifier LTN (less than) or GTN (greater than). In this case a report is generated if the number of exception matches is less than (greater than) the order of the regression test. If the exception tests true for the entire test, the test definition, preturned sensors, and number of matches is reported 66. For example, the exception to report when less than one malfunction severity in a trip regression test is greater than 3.0 is:

(Trip)

Exception: LTN Malf Severity>3.0

Each exception is of the form:

Exception: <qualifier> <parameter>

<operator> <constant>

The parameter specifies what result of the diagnosis is to be examined. A parameter is defined by:

<object><attribute>

Valid objects are malfunctions or procedures. Omitting the object portion of the parameter indicates that both malfunction and procedures should be checked. Attributes, used by all categories except update exceptions, when the preferred expert system is used, are confidence (CF), severity (SEV), importance (IMP), or priority (PR).

Operators used in exceptions are:

Updated, -Updated, ==, -=, >, >=, <, <=

The first two operators are used only with update exceptions and indicate if any rule supported by the updated value fired. The rest define a comparison between the parameter and a floating point constant with the third and fourth operators compare for equality and inequality.

The requirements of the test plan can be implemented in a test plan language as defined in Appendix I. This language definition can be used by a person of ordinary skill in the art to create a parser generated by the YACC and LEX unix third generation language development tools available which will accept the test plan text file and output an appropriate internal representation of the plan. An example of a test plan is illustrated below:

Rulebase directory:

ps:[harper.pds.code.regress.vbl]

TEST 1. (Normal)

Exception 1: - Updated

TEST 2. (Trip)

Exception 1: CF>0.800000

Exception 2: LTN CF>0.800000

Two results files are preferably produced by the regression analyzer of the present invention. If the analyzer is run in batch mode there also is an output file from the session. The output file shows the progress of the tests. The log file 24 is usually quite lengthy because it lists all the sensors, their descriptions, and the test data, all the malfunctions and procedures and their descriptions, the test plan, and the individual test result listings. The exception results or report file 20 is more manageable and lists all the exception reports from the tests.

The screen output 28 from the present invention preferably shows informational and error messages generated during the rerunning of PDS. Each test is identified and timestamped along with the diagnostic cycles that are run. This output is similar to the file produced by the diagnostic version of PDS. An example of screen output can be found in Appendix II.

The log file 24 is a record of the entire test. This record can be achieved in a conventional source code library and used to later compare the complete test results with complete test results of a run with new rulebase changes to perform a regression comparison. A conventional difference comparison between complete log files will show exactly what changed between the time the tests were run. In this way, a regression analysis between rulebase changes can be performed. In addition, the log defines explicitly what the inputs and outputs to the rulebase are, which can later be used as training examples for other types of expert systems, for example, a neural net expert system implementation. There are preferably three sections in the log file as illustrated in Appendix II. The first section details the test parameters: the sensors and sensor data, malfunctions and procedures, and all their descriptions. The second section is a listing of the test plan. The third section contains the test cases. For each case the malfunctions and procedures that were updated are listed in alphabetical order, along with their confidence, severity, importance, and priority.

The exceptions file 26 is a report of each instance that matches with the exceptions defined in the test plan. The exceptions file is essentially a filtered log file where the filter characteristics are defined by the exceptions specified by the user. As illustrated in Appendix II, each report is of the form:

(<timeset>/<sensors> . . .)<object><message>

As previously mentioned, timesets are sensor data categories, for example trip or warning. Sensors are the names of the sensors that were assigned values other than normal for this diagnostic cycle. Objects are either malfunction or procedure names. Messages are text that identify the type of exception report.

The present invention is preferably implemented using a structured design approach as discussed is:

Structured Design: Fundamentals of a Discipline of Computer Program Design, Yourdon and Constantine, Yourdan Press, 1979; Structural Analysis And System Specification, DeMarco, Yourdon Press, 1979; Software Engineering: A Practitioner's Approach, Pressman, McGraw Hill Book Company, 1982; and Tutorial on Software Design Techniques, Freemen and Wasserman, 4th Ed, IEEE Computer Society Press, 1983.

More particularly the present invention is preferably implemented using a systems development tool such as CASE Analyst/RT Users Manual for VAX station/VMS Hosts VO.O 4.1, Mentor Graphics, 1988 which implements a methodology described in Structured Analysis And System Specification, De Marco, Yourdon Press, 1979. This approach or tool allows the designer to create and maintain control and data flow diagrams which can be implemented by standard "C" routines and custom "C" routines quickly and efficiently. FIGS. 4 and 5 illustrate the present invention using data flow and control flow diagrams in this methodology using the above-mentioned tool. The data defintions used when the present invention is implemented using this methodology is attached as appendix III. One of ordinary skill in the art would be able to create the present invention from the diagrams, the information previously described and the appendices. FIG. 4 illustrates the data flow into and out of the test system 20 in addition to the information previously mentioned with respect to FIG. 1. FIG. 4 illustrates that the production environment 100, that is plant information must be included in the system.

As illustrated in FIGS. 4 and 5, the first step in the present invention, is to read 200 the test input in the test plan language by the user, parse the test plan input and create the internal test plan representation 18. The next step is to initialize 202 all global variables. This step is part of the PDS expert system inference engine 16 and it creates and initializes the true, false and other contents, and creates and initializes the PDS variables that manage sensor time steps. The system is next recovered 204 which is also part of the PDS inference engine 16. This routine accepts the text file knowledge base definition and loads the corresponding schemata into memory 14. The next step is also part of the PDS inference engine 16 and it reads 206 the unit information text file. This file is part of the PDS production environment 100 that associates a knowledge base with a specific customer application and, as a side effect, the present invention uses the unit name in all output file headers. Next the system reads 208 the sensors values, a step which is also a part of the PDS inference engine 16. This step reads the sensor data text file in standard format and loads the sensor values and time stamps into an internal reading list. The PDS inference engine 16 then restores 210 the histories. To perform this step the inference engine reads a history text file and loads text history lists and event records into the corresponding schemata. The history text file is part of the PDS production environment 100 that preserves time-based analysis results across inference engine restarts. Then a step is performed by the test system 20 ti initiate 212 the data timesets. This step merely uses the reading list created in the read sensor step 208 to fill a seven column timesets array with each sensors normal, fail low, fail high, predict, diagnostic, warning and trip values. The use of an array rather than a text list enhances the speed of the test system. The next step, which is also part of the present invention, is to create 214 regress logs. This step summarizes and records the complete specification of the analysis by writing to the log file: the time and customer application for this test plan, a description of all sensors and each of their seven data values, and a description of all the possible malfunctions. The step 214 opens and initializes the exception report and log files by writing the time and customer application of this text plan to the file 26. The next step 216 is also part of the test system 20, it writes the test plan read in by the read plan step 200 to the log file. It appends a copy of the test plan to the log file. The last step 218 is the execute step which calls the PDS inference engine 16 repeatedly to execute the diagnoses specified by the test plan. The detailed algorithm for this step is set forth in appendix IV.

As discussed above the present invention will provide an efficient, standard test tool which will verify diagnostic rulebases both during development and after modifications and upgrades are made during commercial use. The present invention allows programmable exceptions, provides a language to describe them along with a log file which can be used for training and testing all types of expert systems including neural networks and provides a programmable test agenda with its own language.

The many features and advantages of the invention are apparent from the detailed specification and thus it is intended by the appended claims to cover all such features of the invention which fall within the true spirit and scope thereof. Further, since numerous modifications and changes will readily occur to those skilled in the art, it is not desired to limit the invention to the exact construction and operation illustrated and described, and accordingly all suitable modifications and equivalents may be resorted to, falling within the scope of the invention. For example, some expert systems have the capability of unfiring rules and PDS is one such system. Rather than fire all rules with normal values at steps 50 and 52 to effectively reset the system to the baseline, it is possible to unfire all the fired rules. ##SPC1##