Diagnosis of faults in a complex system

[0002] This patent application claims the benefit of U.S. Provisional Application No. 60/162,045 filed on Oct. 28, 1999.

[0003] The present invention relates generally to machine diagnostics, and more specifically to a system and method that improves diagnostic accuracy by presenting fault and operational data in a chronologically ordered file.

[0004] A machine, such as a locomotive or other complex system used in industrial processes, medical imaging, telecommunications, aerospace applications, and power generation may include controls and sensors for monitoring the various systems and subsystems of the machine recording certain operational parameters and generating a fault indication when an anomalous operating condition occurs. Certain of these anomalous conditions may require the imposition of operational restrictions on the machine, without requiring a complete and total shutdown. In any case, because restricted operation can be costly, it is essential to accurately diagnose and quickly repair the machine.

[0005] Such complex machines may generate an error log (or fault log), containing information related to a malfunction. The field engineer called to diagnose and repair the machine will first consult the error log to assist with the diagnosis process. The error log presents a “signature” of the machine's operation and can be used by the repair technician to identify specific malfunctions and the operational parameters of the machine before, during and after the fault occurred. Using her accumulated experiences at solving machine malfunctions, the field engineer reviews the error log, identifies the root cause of the fault and then repairs the machine to correct the problem. If the diagnosis was accurate, the repair will correct the machine malfunction. When the error log contains only a small amount of information and the machine is relatively simple, this manual process will work fairly well. However, if the error log is voluminous and the machine is complex, some entries may have an uncertain relationship or perhaps no relationship to the malfunction. It will therefore be difficult for the field engineer to properly review and analyze all the information and successfully diagnose the fault.

[0006] To overcome the problems associated with evaluating large amounts of data in error logs, computer-based diagnostic expert systems have been developed and put to use. These diagnostic expert systems are developed by interviewing field engineers to determine how they proceed to diagnose and fix a machine malfunction. The interview results are then translated into rules and procedures that are stored in a repository, which forms a either a rule base or a knowledge base for machine repairs. The rule or knowledge base operates in conjunction with a rule interpreter or a knowledge processor to form the diagnostic expert system. Based on information input by the technician, the rule interpreter or knowledge processor can quickly parse information in the rule or knowledge base to evaluate the operation of the malfunctioning machine and provide guidance to the field engineer. One disadvantage associated with such conventional diagnostic expert systems is the limited scope of the rules or knowledge stored in the repository. The process of knowledge extraction from experts is time consuming, error prone and expensive. Finally, the rules are brittle and cannot be updated easily. To update the diagnostic expert system, the field engineers have to be frequently interviewed so that the rules and knowledge base can be reformulated.

[0007] Another class of diagnostic systems use artificial neural networks to correlate operational and fault data with potential root causes. An artificial neural network typically includes a number of input nodes, a layer of output nodes, and one or more “hidden” layer of nodes between the input and output nodes. Each node in each layer is connected to one or more nodes in the preceding and the following layer. The connections are via adjustable-weight links analogous to variable coupling-strength neurons. Before being placed in operation, the artificial neural network must be trained by iteratively adjusting the connection weights, using pairs of known input and output data, until the errors between the actual and known outputs, based on a consistent set of inputs, are acceptably small. A problem with using an artificial neural network for diagnosing machine malfunctions, is that the neural network does not produce explicit fault correlations that can be verified by experts and adjusted if desired. In addition, the conventional steps of training an artificial neural network do not provide a measure of its effectiveness so that more data can be added if necessary. Also, the effectiveness of the neural network is limited and does not work well for a large number of variables.

[0008] Case-based reasoning diagnostic expert systems can also be used to diagnose faults associated with malfunctioning machines. Case-based diagnostic systems use a collection of data, known as historical cases, and compare it to a new set of data, a new case, to diagnose faults. In this context, a case refers to a problem/solution pair that represents the diagnosis of a problem and the identification of an appropriate repair (i.e., solution). Case-based reasoning (CBR) is based on the observation that experiential knowledge (i.e., memory of past experiences) can be applied to solving current problems or determining the cause of current faults. The case-based reasoning process relies relatively little on pre-processing of raw input information or knowledge, but focuses instead on indexing, retrieving, reusing, comparing and archiving cases. Case-based reasoning approaches assume that each case is described by a fixed, known number of descriptive attributes and use a corpus of valid historical cases against which new incoming cases can be matched for the determination of the root cause of the fault and the generation of a repair recommendation.

[0009] Commonly assigned U.S. Pat. No. 5,463,768 discloses an approach to fault identification using fault or error log data from one or more malfunctioning machines using a CBR approach. Each of the historical error logs contains data representative of fault events occurring within the malfunctioning machines. In particular, a plurality of historical error logs are grouped into case sets of common malfunctions. Common patterns, i.e., identical consecutive rows or strings of error data in the case sets are used for comparison with new error log data. In this comparison process, sections of data in the new error log that are common to sections of data in each of the historical case sets (the historical error logs) are identified. Since the historical error logs have been correlated with a specific repair having a high probability of resolving the fault, the common sections of data in the historical error logs and the new error log can lead to a recommended repair with a high probability of resolving the fault.

[0010] U.S. Pat. No. 6,415,395, entitled “Method and System for Processing Repair Data and Fault Log Data to Facilitate Diagnostics”, assigned to the assignee of the present invention and herein incorporated by reference, discloses a system and method for processing historical repair data and historical fault log data, where this data is not restricted to sequential occurrences of fault log entries, as in the commonly owned patent described above. This system includes means for generating a plurality of cases from the repair data and the fault log data. Each case comprises a repair and a plurality of related, but distinct faults. For each case, at least one repair and distinct fault cluster combination is generated and then a weight is assigned thereto. This weight value indicates the likelihood that the repair will resolve any of the faults included within that fault cluster. The weight is calculated by dividing the number of times the fault cluster combination occurs in cases comprising related repairs by the number of times the combination occurs in all cases. New fault log data is entered into the system and compared with the plurality of fault log clusters. The repair associated with the matching fault log cluster represents a candidate repair to resolve the new fault. The report output from this system lists the candidate repairs in sequential order according to the calculated weights.

[0011] Further, U.S. Pat. No. 6,343.236, entitled “Method and System for Analyzing Fault Log Data for Diagnostics”, assigned to the same assignee of the present invention and herein incorporated by reference, discloses a system and method for analyzing new fault log data from a malfunctioning machine, again where the system and method are not restricted to sequential occurrences of fault log entries. The fault log data is clustered based on related faults and then compared with historical fault clusters. Each historic fault cluster has associated with it a repair wherein the correlation between the fault cluster and the repair is indicated by a repair weight. Upon locating a match between the current fault cluster and one or more of the historical fault clusters, a repair action is identified for the current fault cluster based on the repair associated with the matching historical fault cluster.

[0012] One particular type of fault that can be advantageously analyzed by certain fault analysis and diagnostic tools is the so-called “no trouble found” fault. Failure conditions that are difficult to diagnose within a complex system may result in such a declaration of no trouble found. Typically, the system experiences intermittent failures and when the system is taken out of service for diagnosis, there is no evidence of a fault or failure, i.e., the fault is intermittent. In this situation, the repair technician declares that the system is failure free and ready for return to service, i.e., no trouble found. Later, the system may experience a repeat failure due to the same problem.

[0013] It is believed that the fault and repair analysis tools disclosed in the patent applications described above provide certain advantages and advancements in the art of the diagnostics of complex machines. It would be desirable, however, to provide a system and method to improve the evaluation and identification of faults by undertaking the analysis on a chronologically ordered set of fault data and machine operational information.

[0014] The search for an effective tool to diagnose failures occurring in a complex system, such as a railroad locomotive, has been an elusive one. Several such tools are discussed above. The present invention describes a process that systematically combines fault log and machine operational information (sensor readings of various operational parameters commonly referred to as data packs) with maintenance and repair data and fault declaration information into a single chronologically ordered file. With the data ordered chronologically, the present invention offers a more effective tool to diagnose the exact nature and cause of a fault. Combining all of this information, as taught by the present invention, presents the repair analyst with a single file from which the state of the machine can be determined at any time prior to and after the occurrence or recording of a specific fault. By identifying anomalies and analyzing operational parameters occurring immediately prior to the fault (and in some cases, after the fault), the analyst can identify the most probable cause. Additionally, by reviewing the data pack information recorded immediately after a repair or other maintenance action, the analyst can determine the effectiveness of that repair or maintenance activity. The chronologically ordered data presentation also allows the detection of incipient faults based on anomalous conditions and operational parameters. Analysis rules can then be established for later use in recognizing potential fault conditions in the operational data. The present invention can also help to create data filters so that only the information identified as pertinent to a particular fault is analyzed. The filtered out data can be ignored due to its apparent lack of correlation with a specific fault. Using the teachings of the present invention increases the accuracy of failure diagnosis, resulting in more effective (e.g., more timely and less costly) troubleshooting and repair actions. Fewer repeat failures will occur because there will be fewer misdiagnosed cases and fewer no trouble found faults.

[0015] The features and advantages of the present invention will become apparent from the following detailed description of the invention when read together with the accompanying drawings in which:

[0016] FIG. 1 is a block diagram of one embodiment of a system of the present invention using a processor for processing operation parametric data and fault log data;

[0017] FIG. 2 is an illustration of exemplary repair log data;

[0018] FIG. 3 is an illustration of exemplary fault log data;

[0019] FIG. 4 is a flowchart illustrating operation of the present invention; and

[0020] FIGS. 5 and 6 illustrate exemplary data ordered in accordance with the teachings of the present invention.

[0021] Before describing in detail the particular fault diagnosis system in accordance with the present invention, it should be observed that the present invention resides primarily in a novel combination of processing steps and hardware elements related to a system for diagnosing faults on a machine. Accordingly, these processing steps and hardware components have been represented by conventional processes and elements in the drawings, showing only those specific details that are pertinent to the present invention, so as not to obscure the disclosure with structural details that will be readily apparent to those skilled in the art having the benefit of the description herein.

[0022] FIG. 1 diagrammatically illustrates one embodiment of the diagnostic system 10 of the present invention for analyzing faults for the purpose of identifying root causes, fault patterns, and correlations of these patterns with operational data. The diagnostic system 10 provides a process for automatically harvesting or mining repair data describing related and unrelated repairs, fault log data describing faults or anomalous events, and parametric operational data from one or more machines, such as locomotives. The diagnostic system 10 generates a chronological list of the repair information, fault data, and operational parametric information for analysis by a locomotive repair expert. With the disparate databases combined in a chronologically ordered database, the process of analysis is easier as the potential correlation between various data items is easier to identify.

[0023] Although the present invention is described with reference to a locomotive, the diagnostic system 10 can be used in conjunction with any machine where the operational parameters of the machine are monitored. Exemplary applications include complex machinery used in the medical diagnostic field, telecommunications field, and aircraft engine field.

[0024] The exemplary diagnostic system 10 illustrated in FIG. 1 includes a processor 12 such as a computer (e.g., a UNIX workstation) having a hard drive, input devices such as a keyboard or a mouse, magnetic storage media (e.g., tape cartridges or disks), optical storage media (e.g., CD-ROM's), and output devices such as a display and a printer. The processor 12 is connected to and processes data contained in a repair database 20 and a fault log database 22.

[0025] The repair database 20 includes repair data or records describing repairs or maintenance actions for one or more locomotives. FIG. 2 shows an exemplary portion 30 of the repair data contained in the repair data database 20. The repair data includes a customer identifier in a column 32, a unique locomotive identifier or locomotive road number in a column 33, the repair date in a column 34, a repair code in a column 35, a prose description of the repair in a column 36, a description of the actual repair performed in a column 37, a description of the failure mode that the repair was intended to resolve (if any) in a column 38, the sub-assembly on which the repair was conducted in a column 39, and the primary system to which the repair relates in acolumn 40. The information in the repair database 20 is collected from field support personnel located at the locomotive maintenance facilities. Locomotives are brought into the repair facility as required to undergo repair, and also on a predetermined fixed schedule for regular preventative maintenance and minor repairs. Entries in the repair database are made at that time.

[0026] The fault log database 22 includes fault log data indicating faults on the locomotive, as recorded by an on-board monitor system at the locomotive. Further details of an exemplary on-board monitor system can be found in commonly-owned United States patent application bearing the title “On-Board Monitor for a Railroad Locomotive”, filed on Oct. 25, 2000 with application Ser. No. 09/696,368.

[0027] FIG. 3 shows an exemplary portion 40 of the fault log data stored in the fault log database 22. The fault log data may include a customer identifier in a column 42, a unique locomotive identifier or road number in a column 44, the date that the fault occurred in a column 45, a fault code in a column 46, and a prose description of the fault in a column 50. The time at which the fault occurred is shown in a column 47 and the fault reset time is set forth in a column 48. The values in the columns 47 and 48 are the number of counted intervals from a predetermined start time. These count intervals can be converted to eastern standard time using a well known software decoder and given the start date and time and the count interval. Resetting of a fault can be accomplished automatically or manually. For those faults that do not endanger locomotive operation, after the fault occurs, it will be automatically reset. The recording of this fault serves as a flag indicating the occurrence of an anomalous event (i.e., a fault), but it was only minor or did not prevail for an extended duration. For example, a traction motor current in excess of a threshold value will cause the recording of a fault, but if the current value returns to a normal rating immediately after the spike, the fault will be reset and the locomotive can continue to operate as normal. More serious faults require analysis by human intervention to determine whether the locomotive can continue operation. The occurrence of such faults is recorded and also disables operation of the locomotive subsystem to which it relates. If the locomotive operator or technician determines that continued operation during this fault condition is acceptable, he can manually reset the fault, causing the subsystem to be reactivated. If the nature of the fault is such that operation cannot continue, the locomotive is removed from service. Additional information in the form of operating parametric values when the fault occurred are shown in the fault log 40 within that area designated by a reference character 49. This operational information may include temperature sensor readings, pressure sensor readings, electrical voltage and current readings, and power output readings. Operational states may also be shown within the area designated by the reference character 49. Such operational states include whether the locomotive is in a motoring or a dynamic braking mode, whether any particular subsystem within the locomotive is undergoing a self test, whether the locomotive is stationary, and whether the engine is operating under maximum load conditions.

[0028] As taught by the application above entitled “On-Board Monitor for a Railroad Locomotive”, the information in the fault log 40 is downloaded a predetermined number of times each day from the locomotive to a remote monitoring and diagnostic center. Certain faults occurring on the locomotive generate an immediate call from the locomotive back to the remote monitoring and diagnostic center, after which the remote monitoring and diagnostic center calls the locomotive for downloading the information shown in the fault log 40. In one embodiment, the remote monitoring and diagnostic center calls the locomotive three times per day to download the fault log data and operational parametric data.

[0029] A customer database 25 in FIG. 1 contains additional fault and operational information as recorded by the railroad operating the locomotive. Exemplary information included in each of these data files, which is generally entered by a customer repair technician or the locomotive operator, includes certain faults and anomalous conditions and the time and date of their occurrence. A unique locomotive identifier, for example the locomotive road number, is also included in each data file.

[0030] A statistics database 26 includes a plurality of files identifying certain attributes of each locomotive and operational parametric data. For example, a locomotive file in this database includes the version number of significant software applications running on the locomotive, the model number associated with key systems, and other locomotive features (for example, number of engine cylinders and maximum horsepower rating). This file also contains certain cumulative operational statistics such as the cumulative current supplied to the traction motors (in megawatt hours), the current generated by the dynamic braking system (in megawatt hours), the number of engine operational hours, the total number of miles traveled by the locomotive, and the total number of engine hours. Finally, this database includes operational parametric information collected at regular intervals or collected immediately following the occurrence of a fault. The data to be collected and the collection frequency can be either automatically or manually set. Also, in lieu of including the operational parametric data related to a fault in the statistics database, this information can be collected when the fault occurs and made a part of the fault file. In such a case, the operational parametric data would be stored in the fault log database 22, rather than the statistics database 26.

[0031] FIG. 4 illustrates the process executed by the processor 12 (see FIG. 1) for creating the chronological listing of repairs, faults, and operational data. At a step 80, repair data is read from the repair database 20. At a step 82, the fault data is read from the fault log database 22. At a step 84, information input by the railroad customer into its own database (referred to as the customer database 25 in FIG. 1) is read by the processor 12. Statistics data from the statistics database 26 is read at a step 86. At a step 88, the processor 12 chronologically orders the various input data and creates a report, as indicated at a step 90.

[0032] A representative sample of the chronologically ordered report is illustrated in FIG. 5. Organizing the information from the various databases into a time line structure facilitates analysis of locomotive road failures and identification of fault patterns that can be used for early detection and more robust diagnosis of faults. A column 100 identifies the locomotive based on its unit number or road number. The date and time of the occurrence (where the “occurrence” can be a fault, a repair, or simply the reading of operational data) is listed in a column 102. A column 104 identifies a fault by number and a column 106 identifies the fault with a label. A more complete description of the fault is set forth in a column 108. A column 110 identifies the occurrence time using a sequential counter that begins counting at a predetermined time, where the counting start point is under either automatic or manual control. If the fault is reset, that time, again using the counter sequence, is set forth in a column 112. The remaining columns show data related to certain locomotive parameters. The locomotive speed is set forth in a column 114, the direction in a column 116, the notch setting in a column 118, the engine speed in a column 120, the volts across the traction motors in a column 122, and the current input to the traction motors in a column 124. A column 126 lists a sub-identifying number for the fault shown in the column 104.

[0033] FIG. 6 is another chronological listing of various fault events in accord with the present invention, including an overlay of a repair performed on Feb. 27, 1999, in Morrill, Nebr. The repair was intended to resolve the fault represented by fault code 7324. Note that in fact 7324 faults do not appear after the repair on February 27, but faults identified by code 731A continue to appear both before and after the repair.

[0034] Additional information that can be included in the time line includes: the cab version, the cumulative number of motor megawatt hours and dynamic brake megawatt hours, the total power produced at the time when the fault occurred, the cumulative number of miles logged by the locomotive, the cumulative engine hours, and the latitude and longitude of the locomotive at the time of the occurrence.

[0035] CAB is one of many controller cards on the locomotive and includes a processor with executable software. Periodically, improvements and updates are made to this software and bugs in the software code are fixed, but the modified version of the software is not applied to all locomotives simultaneously. Therefore, it is important to know which version of the software is running on a particular locomotive as this information may help in determining the cause for certain faults or in understanding the behavior of out-of-spec operational parameters in the datapack information.

[0036] The first fault listed in the time line of FIG. 5 bears number 731A. Turning to the fault description column 108, this fault relates to the phase modulator associated with inverter 3. As is generally known, a diesel engine of a locomotive is drivingly coupled to an alternator for supplying three phase alternating current, which is then rectified to direct current and then inverted back to frequency controllable alternating current for driving traction motors associated with each locomotive axle. Generally, there is one inverter providing controllable alternating current to each traction motor. The first fault listed involves the phase modulator operative on the positive going portion of the B phase alternating current of the third inverter. The fault may also be caused by a problem in the inverter motor controller/traction motor controller (or IMC 2-3, 4, 7/TMC-3, 0). Certain locomotives include an inverter motor controller while others use a traction motor controller. It should be realized that the potential fault causes listed in the column 108 do not necessarily represent the only conditions leading to fault 731A. Those listed are simply the most likely causes. The fault sub-identification in the column 126 (in this case sub-identification number 4) provides the user with more specific information relative to fault 731A.

[0037] Note that within a bubble 130, the fault 731A has occurred twice. Once on Feb. 15, 1999, and again on Feb. 17, 1999. Recognizing that the time line shown in FIG. 5 is merely an excerpt, the analysis of the complete time line may indicate that the fault 731A has occurred on one or more occasions prior to Feb. 15, 1999. The locomotive expert analyzing the time line can identify the first occurrence of fault 731A and the various repairs and operational parameters recorded since that first occurrence. By combining the fault information, repair information, and operational information from separate databases and ordering the information chronologically, the locomotive expert can more easily determine causes and effects as related to the recorded data. The analysis can also identify concurrent faults and perhaps point to a first solution when the faults occur jointly, while a second solution is indicated when only one of the faults occurs. Fault solutions can also be identified dependent upon whether the fault occurs while the locomotive is in the motoring or the dynamic braking mode. Certain faults may occur on one or more of the inverters, indicating a particular solution. But, if all of the inverters manifest a fault, then a different root cause is indicated. By chronologically ordering the data as shown in FIG. 5, the locomotive expert can more easily and efficiently identify related and independent faults so that a more likely successful fault solution can be recommended. Clearly, such a process of careful analysis and identification of joint causes could not be undertaken if the various databases were not combined and chronologically ordered. Further, the root causes identified through the analysis set forth above can be used to create a rule in the expert analysis systems discussed above in the Background section.

[0038] One particular fault situation that can be advantageously analyzed using a time line created in accord with the present invention is a so-called “no trouble found” fault. Failure conditions that are difficult to diagnose within a complex system may result in the repair technician's declaration of a no trouble found event because he is unable to either recreate or diagnose the source of the problem. Typically, these no trouble found events occur due to intermittent failures. Or the complexity of the system may obscure the fault condition to a repair technician whose skills are deficient in some area relevant to system performance. No trouble found events are especially troublesome because the locomotive is placed back into service without a resolution of the fault and later may experience a repeat failure due to this same problem. These no trouble found events can also be advantageously analyzed and resolved by the present invention. Recall that certain information in the time line is made available from a customer database. Since the customer is frequently responsible for maintenance and repair of the locomotive, it is the customer repair technician who usually declares a fault to be a no trouble found event. However, when the time line analysis is undertaken by a locomotive repair expert, she may be able to discern the specific nature of the fault and its root cause using the data presented in the time line. As was mentioned above, the relevant time line data that led to the successful resolution of what heretofore had been declared a no trouble found event can be used in the expert systems discussed above to create rules that will be used to successfully identify and solve such faults.

[0039] The information contained within a rectangle 140 of FIG. 5 is the customer's record and related description of a locomotive road failure. The noted initial cause of the road failure, based on the customer's recorded comment in the time line, was that one or more traction motors were cut-out, leading to a loss of motive power. The unit was subsequently released as a no trouble found event, because when the technician examined the unit, the traction motors were operating properly. Reviewing the time line of FIG. 5, the locomotive expert can point to the 731A fault as a potential cause of the traction motor cut-out problem. If this was in fact a true no trouble found event (e.g., an intermittent fault), one would not expect to see the reappearance of the 731A fault. In fact, in a bubble 144, the 731A fault continues to log, indicating that the road failure was not a true no trouble found event and that the problem still exists. In fact, it was not until later that the problem causing the 731A fault was finally determined and repaired.

[0040] Generally, the invention has been described in conjunction with faults and repair for a single locomotive. Those skilled in the art will recognize that it is advantageous to chronologically combine fault, repair, and statistics information collected from a plurality of locomotives to identify those faults occurring on similar locomotives.

[0041] While the invention has been described with reference to a preferred embodiment, it will be understood by those skilled in the art that various changes may be made and equivalent elements may be substituted for elements thereof without departing from the scope of the present invention. In addition, modifications may be made to adapt a particular situation more material to the teachings of the present invention without departing from the essential scope thereof. Therefore, it is intended that the invention not be limited to the particular embodiment disclosed as the best mode contemplated for carrying out this invention, but that the invention will include all embodiments falling within the scope of the appended claims.