Error analysis fed from a knowledge base

[0001] The present application is related to co-pending applications entitled First Failure Data Capture, application Ser. No. 10/195,181, filed on even date herewith. These co-pending applications are assigned to the assignee of the present application, and herein incorporated by reference. A portion of the disclosure of this patent document contains material which is subjected to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever.

[0002] The present invention relates generally to information handling, and more particularly to error handling, recovery, and problem solving, for software and information—handling systems.

[0003] Various approaches have been proposed for handling errors or failures in computers. Examples include U.S. Pat. No. 6,170,067, System for Automatically Reporting a System Failure in a Server (Liu et al., Jan. 2, 2001); it involves monitoring functions such as cooling fan speed, processor operating temperature, and power supply. However, this example does not address software errors. Another example is U.S. Pat. No. 5,423,025 (Goldman et al., Jun. 6, 1995); it involves an error—handling mechanism for a controller, in a large-scale computer using the IBM ESA/390 architecture. In the above-mentioned examples, error—handling is not flexible; error—handling is not separated from hardware, and there is no dynamic tuning.

[0004] Generally, if a software product has any ability to handle errors, that ability is limited and inflexible. Conventional software fixes can be time-consuming to develop, and difficult to apply. Conventional software error messages often are not unique and not informative. Thus there is a need for flexible solutions that lead to a useful response; at the same time, the burden of reprogramming needs to be reduced, and the destabilizing effects of major code revisions need to be avoided.

[0005] An example of a solution to problems mentioned above comprises analyzing an incident, responding to said incident, and customizing the response, based on the result of the analysis. Another example of a solution comprises accessing a knowledge base, comparing a current incident to known incidents in said knowledge base, and if a match is found, retrieving information from said knowledge base, said information being relevant to the current incident.

[0011] The examples that follow involve the use of one or more computers and may involve the use of one or more communications networks. The present invention is not limited as to the type of computer on which it runs, and not limited as to the type of network used. Some of the examples that follow have been implemented by using object-oriented programming with the JAVA programming language; however, the invention could be implemented with another programming language. Some of the examples that follow have been implemented for handling errors in web application server software; however, the invention could be implemented for handling errors in any kind of software product or software component.

[0012] The following are definitions of terms used in the description of the present invention and in the claims:

[0013] “Comparing” means bringing together for the purpose of finding any likeness or difference, including a quantitative or qualitative likeness or difference. “Comparing” may involve answering questions including but not limited to: “Does a given item match any element of a set of known items?” Or “Is a measured value greater than a threshold value?”

[0014] “Component” means any element or part, and may include elements consisting of hardware or software or both.

[0015] “Computer-usable medium” means any carrier wave, signal or transmission facility for communication with computers, and any kind of computer memory, such as floppy disks, hard disks, Random Access Memory (RAM), Read Only Memory (ROM), CD-ROM, flash ROM, non-volatile ROM, and non-volatile memory.

[0016] “Customize” means to adapt, adjust or tune.

[0017] “Error” means any event that may be unexpected, undesirable, or incorrect.

[0018] “First failure data capture” (FFDC) refers to automated solutions that are typically “on” and ready to work the first time an error or failure occurs; it also refers to reducing the burdens of problem reproduction and repetitive data capture.

[0019] “Incident” means any event that may be perceived as a cause of past, present, or future trouble.

[0020] “Outputting” means producing, transmitting, or turning out in some manner, including but not limited to writing to disk, printing on paper, or displaying on a screen, or using an audio device.

[0021] “Storing” data or information, using a computer, means placing the data or information, for any length of time, in any kind of computer memory, such as floppy disks, hard disks, Random Access Memory (RAM), Read Only Memory (ROM), CD-ROM, flash ROM, non-volatile ROM, and non-volatile memory.

[0022] FIG. 1 illustrates a simplified example of an information handling system that may be used to practice the present invention. The invention may be implemented on a variety of hardware platforms, including embedded systems, personal computers, workstations, servers, and mainframes. The computer system of FIG. 1 has at least one processor 110. Processor 110 is interconnected via system bus 112 to random access memory (RAM) 116, read only memory (ROM) 114, and input/output (I/O) adapter 118 for connecting peripheral devices such as disk unit 120 and tape drive 140 to bus 112. The system has user interface adapter 122 for connecting keyboard 124, mouse 126, or other user interface devices such as audio output device 166 and audio input device 168 to bus 112. The system has communication adapter 134 for connecting the information handling system to a data processing network 150, and display adapter 136 for connecting bus 112 to display device 138. Communication adapter 134 may link the system depicted in FIG. 1 with hundreds or even thousands of similar systems, or other devices, such as remote printers, remote servers, or remote storage units. The system depicted in FIG. 1 may be linked to both local area networks (sometimes referred to as intranets) and wide area networks, such as the Internet.

[0023] While the computer system described in FIG. 1 is capable of executing the processes described herein, this computer system is simply one example of a computer system. Those skilled in the art will appreciate that many other computer system designs are capable of performing the processes described herein.

[0024] FIG. 2 is a block diagram illustrating an example of a method and system for handling errors according to the present invention. To begin with an overview, FIG. 2 shows filter 220 selecting an incident that requires further processing, and one or more diagnostic modules at 240 capturing data on a thread that encounters said incident, from runtime 210. Features for data capture were provided in runtime 210. An example like this may also involve outputting captured data (not shown in FIG. 2; see FIGS. 3 and 4). Arrows 225 and 255 symbolize flow of selected incidents in FIG. 2.

[0025] To begin a more detailed description of the example in FIG. 2, consider how an application or software component (symbolized by runtime 210) might utilize such a method and system for handling errors. One or more diagnostic modules 240 per software component were provided. Data was made available to diagnostic modules 240, at key places in the code, to help collect volatile data if an incident occurred. An example implementation involved providing runtime features (in runtime 210) for data capture. These features included transferring control to at least one data-capture engine when an incident occurred, creating diagnostic module(s) 240 (one or more diagnostic modules per component), registering diagnostic module(s) 240 with a diagnostic engine 230 (one diagnostic engine per thread), and making data available to diagnostic module(s) 240. An example implementation involved minimizing in-line code for data capture, or keeping code for data capture substantially separate from in-line paths, symbolized by showing one or more diagnostic modules at 240 separate from runtime 210 in FIG. 2.

[0026] Data capture (symbolized by double-headed arrow 245) involved bundling an incident in a persistent form that gathered data from multiple components. Data capture 245 comprised at least one action chosen from: producing a thread dump; producing a process dump; and producing a core dump. Data capture 245 in some cases comprised parsing exceptions (arrow 215 symbolizes exceptions, errors, or events), and based on said parsing exceptions, selecting data for capture. This was one way of accomplishing customized data capture, in addition to basic, generalized data capture.

[0027] There was another way in which data capture was customized. This customization was implemented by using analysis engine 250 and knowledge base 260. These two components, along with arrows 255, 265, and 275 (arrows showing communication with other components), are drawn in broken lines in FIG. 2, to symbolize that these are optional features. Using these optional features was a way of customizing data capture, based on a particular incident's characteristics. In an example implementation, analysis engine 250 compared a current incident to known incidents in knowledge base 260. If a match was found in knowledge base 260, information that was relevant to the current incident was retrieved from knowledge base 260. This retrieved information was used to customize data capture 245. Arrow 265 shows communication between analysis engine 250 and knowledge base 260.

[0028] An example implementation isolated one or more FFDC entities (such as analysis engine 250) from the specific features of any particular computing platform or product. On the other hand, some functions were associated with a specific computing platform or product, to utilize First Failure Data Capture in that platform or product. A platform-specific diagnostic engine 230 was utilized. A “Utils” class or interface represented platform-specific functions. Platform-specific functions comprised:

[0029] initiating boundaries for an incident;

[0030] capturing general data for said incident;

[0031] logging data from components that were involved with said incident; and

[0032] bundling output when processing was complete.

[0033] Initiating boundaries further comprised actions such as associating an incident with an output stream, or creating an output stream for said incident. Capturing general data further comprised capturing items such as date, timestamp, classpath, class loader, and configuration settings.

[0034] An example implementation suppressed data capture during certain states. Our prototype demonstrated that many benign errors could occur before steady state operation was attained. For that reason, suppressing data capture during certain states may be desirable. Some of these states may be: process initialization, process termination or shutdown, application initialization and others. It was the responsibility of the Utils implementation to define states where data capture could be suppressed.

[0035] In an example implementation, the Utils class provided a service to facilitate temporarily logging data that could be retrieved later if an incident occurred. The programming model was such that the data was logged during normal runtime processing, and the data was retrieved by the diagnostic module 240 that was interested. This facility allowed for automatic stack data capture.

[0036] Filter 220 was a first recipient of incidents, errors, or events; it was a means for selecting incidents that required further processing. Filtering was done according to criteria such as the severity of an error, or the type of exception. An Incident object was created by the filter and passed to other core FFDC entities. In an example implementation, filter 220 was also known as a “High Performance Filter” (HPF). Since all errors were routed to this filter, the filtration process was considered a high performance path. Functions of filter 220 included maintaining a list of incident types that required further processing, and preventing multiple invocation of data capture for the same incident. Filter 220 had an associated configuration file to list incidents that required further processing. It was also responsible for preventing multiple invocation of the remaining FFDC engines for the same incident by accessing the list of recent incidents processed by the diagnostic engine 230. Since filter 220 was the first FFDC engine that was called for incidents, it was the gate for FFDC. Disabling filter 220 essentially disabled FFDC.

[0037] Filter 220 filtered errors based on configuration settings. The filter 220 could be tuned, enabled, and disabled dynamically by updating its configuration settings. With rapid parsing techniques, incidents could be checked against the configuration settings for filtering. A simple implementation could be used for a tactical solution. Additionally, the diagnostic engine (DE) 230 was responsible for providing a set of previously handled incidents so that cascaded incidents were not subject to further processing. Cascaded incident analysis could be enabled for severe conditions. This option could be specified in the configuration settings. FFDC could be disabled at server initialization or during other states (e.g. shutdown, etc.). If a server would not start or encountered start up problems, the persistent form of the configuration setting could be updated prior to server startup, to enable processing of all incidents.

[0038] Calls from the runtime 210 were made to a method to forward exceptions to the filter 220. This was a method that was injected into runtime code of various components to participate in FFDC. These calls were made on error paths that were candidates for FFDC.

[0039] The filter 220 located a diagnostic engine 230 (DE) for the thread. The DE 230 provided a cache of recent incidents that played a role in the decision for continuing FFDC processing, or simply returning control to the runtime 210 without further processing. If FFDC processing was to be continued, an Incident object was created. An analysis engine (AE) 250 was located, if one was configured. A method to analyze the incident (e.g. AE.analyze(Incident)) was called and directives might be found. These along with the Incident and the object pointer were passed to the DE 230.

[0040] Analysis engine 250 was a means for comparing a current incident to known incidents, and a means for customizing responses. In an example implementation, analysis engine 250 was also known as a “Log Analysis Engine.” Analysis engine 250, an optional component, helped provide dynamic tuning information for incident handling. This information was referred to as “directives.” For increased flexibility, a diagnostic module 240 could be implemented to follow directives when available. Directives were stored in knowledge base 260. Analysis engine 250 matched an incident against a set of known incidents and provided additional information, directive(s), that could assist a diagnostic module 240 in customizing the data that was captured and logged.

[0041] A knowledge base 260 was a repository of information such as symptoms, directives, suggested workarounds and explanations. A knowledge base 260 provided input to analysis engine 250. When an Incident was sent to analysis engine 250 [via a method call to analyze the incident, e.g. AE.analyze(Incident)], this incident was compared to the repository, and if a match was found the associated directives were returned as a string array. The last entry in the array was the message or associated text that was normally displayed by the analysis engine 250. If no match is found, a null was returned.

[0042] An example implementation provided access to a knowledge base 260 via a web site. Knowledge base 260 was updated, and updated information was provided to users, utilizing familiar techniques for providing dynamic content through a web site. Thus any user could obtain updated information and benefit from other users' experience. Knowledge base 260 could be implemented by using IBM's DB2 technology, or other database management software could be used, such as ORACLE, INFORMIX, SYBASE, MYSQL, Microsoft Corporation's SQL SERVER, or similar software.

[0043] Control flow for incidents was orchestrated by the diagnostic engine 230 (DE). The diagnostic engine 230 was responsible for the distribution of an incident to the corresponding component. The DE 230 controlled the flow among diagnostic modules 240 for data collection. The diagnostic engine 230 was a dynamic class loader, parent, and invoker of diagnostic modules 240. The diagnostic engine 230 also provided platform-specific utilities for diagnostic modules 240, such as thread dump, core dump, etc., that diagnostic modules 240 could call for global data collection. The diagnostic engine 230 provided common methods for diagnostic modules 240. One instance of a diagnostic engine 230 was created per thread.

[0044] Diagnostic module(s) 240 (DM) were a means for capturing data; they were a data-gathering class for a component or sub-component. Diagnostic module(s) 240 were organized into a single class, rather than implementing disparate routines in various places of component code to handle incidents. DM 240 code was kept separate from in-line paths. Diagnostic module(s) 240 were developed and maintained by components participating in FFDC. Diagnostic module(s) 240 provided data that was beneficial to diagnose a problem. Diagnostic module(s) 240 assisted the DE 230 in control flow for the current Incident.

[0045] Here is a list of some functions performed by diagnostic module(s) 240: calling DE 230's services when appropriate to request thread dumps, core dumps etc.; parsing and processing directives specified by symptom files (supplied by DE 230); pulling and logging stacked data that was pushed for a component; performing generic data capture in the absence of directives (this would be a typical or common set of data that was gathered when diagnosing a problem in an area); parsing exceptions and collecting specific data that pertained to the situation; providing descriptions for the data that was dumped, so that data was easy for service personnel to understand.

[0046] FIG. 3 is a block diagram illustrating a method and system for handling errors, with a hypothetical online banking example. To begin with an overview of this example, a set of applications such as security function 310, bank application 311, and bank application 312 are designed to allow bank customers to conduct financial transactions via the Internet. Normally, a customer would be authenticated before completing a transaction. Successful authentication requires security function 310 to communicate with directory 300. However, in this example broken connection 305 prevents authentication. FIG. 3 shows analysis engine 250 receiving an incident (arrow 255). In this example, the incident involves broken connection 305 to directory 300. FIG. 3 shows security diagnostic module 340 capturing data associated with said incident, from security function 310. For example, security diagnostic module 340 may probe the state of security function 310. The example in FIG. 3 also involves outputting captured data, and outputting a description for said captured data (arrow 315) whereby problem-solving is promoted. FIG. 3 is a simplified diagram that (unlike FIG. 2) does not show filter 220, selecting an incident that requires further processing.

[0047] As indicated by the dashed line, the components analysis engine 250 and knowledge base 260 may be incorporated into one computer system 302, or these components may be incorporated into separate computer systems independent of, but accessible to, one another. Arrow 265 shows communication between analysis engine 250 and knowledge base 260.

[0048] For example, analysis engine 250 may get directives, appropriate for this incident, from knowledge base 260. Directives from knowledge base 260 may provide a means for customizing data capture, based on an incident's characteristics. For example, a directive from knowledge base 260 may instruct security diagnostic module 340 to probe for network status at the time of the incident.

[0049] Diagnostic modules (e.g. security diagnostic module 340) would be a means for providing output 315. Providing output at 315 may comprise providing a notification message (e.g. via an administration console or other forms of centralized monitoring) to indicate that data capture for an Incident has occurred, and that data is available at that time. Providing output at 315 may comprise describing captured data, and describing a context for said captured data, whereby problem-solving is promoted. For example, output at 315 may include a message to an operator, such as: “User cannot be authenticated. Security function 310 cannot connect to directory 300.” The following is another security-related example of output at 315:

[0050] Security Component FFDC Data:

[0051] Unexpected Exception:

• [0052] <exception name>
• [0053] <call stack>

[0054] A Security Exception was caught during com.ibm.websphere.security.passwordVerify(“******”) for user “joe”.

[0055] The credentials of the user appear to have expired. The captured data shows:

[0056] Current Time: 12:21 pm, Credentials creation time: 12:01, credential expiration interval: 20 minutes.

[0057] Query issued to retrieve user from LTPA on dummy.ldap.server.com:389 is: “cn= . . . ”

• [0058] LDAP Specific error= . . .
• [0059] Cache value of data from Security Vault is: . . .

[0060] Dump of active user data from SecurityLTPACache:

• [0061] John Smith, cn=,ou=,cred created at: 11:17 am, expiration 11:37 am, etc.

[0062] The above example is hypothetical; however, it demonstrates how incident data can be made easier to comprehend, whereby problem-solving is promoted.

[0063] An example like the one in FIG. 3 may involve temporarily logging data that could be retrieved later if an incident occurred. The following is another security-related example: if a login is occurring, the security runtime (security function 310) may opt to push the user name or CN in case an exception happens. In security diagnostic module 340, the user identity can be retrieved and logged. The data can also be used to capture artifacts that are related to it (e.g. uid=‘joe’). Security diagnostic module 340 may choose to extract cache state for this user (e.g. credential timeout, expiration, age).

[0064] An example like the one in FIG. 3 may involve taking corrective action, or recovery action, in response to an incident Corrective action could be automatic, or could involve some human intervention. In the example of FIG. 3, involving a broken connection 305 to directory 300, a directive such as “try to reconnect to directory 300 ” might be retrieved from knowledge base 260. Thus, by automatically following this directive, security function 310 could be reconnected to directory 300. Obtaining a proper directive and taking corrective action could be accomplished through security diagnostic module 340. Consider other examples of corrective action: issuing a warning to the Java virtual machine or operating system about a condition (e.g. resource shortage) that contributed to the incident, or issuing an action message to the operator.

[0065] In another example of corrective action or self-healing, a DM (e.g. security diagnostic module 340) may request an automated download and application of a fix, that could be made effective instantly or at the next restart. Self tuning could also be performed using this conduit. For example, if a value exceeds a certain threshold, an intentional Incident could be created and handed to the filter (FIG. 3 is a simplified diagram that, unlike FIG. 2, does not show filter 220). The Diagnostic Module that handles the incident could analyze the Incident and perform self-tuning (e.g. increase or decrease a thread pool size). A set of known conditions may only occur in some environments and impact system throughput. Feeding some of these conditions, if they occur, to the FFDC engines and providing tuning parameters through directives would be another example of corrective action.

[0066] An example like the one in FIG. 3 may involve performing operations on multiple servers. FIG. 3 is a simplified diagram; electronic banking or other electronic transactions may actually involve at least one network and two or more servers. In an example of multiple servers, a first transaction diagnostic module on a first server may determine that an error originated on a second server. The first transaction diagnostic module may send a request to invoke a second, remote, transaction diagnostic module for data capture on a second server.

[0067] FIG. 4 is a flow chart illustrating an example of a method for handling errors according to the present invention. This example begins at block 410, selecting an incident that requires further processing. At this point, for example, calls from the runtime are made to a method to forward exceptions to a filter. With rapid parsing techniques, errors can be checked against configuration settings for filtering. Filtering may be done according to criteria such as the severity of an error, or the type of exception. The DE provides a cache of recent incidents that play a role in the decision to continue processing, or simply returning control to the runtime without further processing. If processing is to be continued, an Incident object is created.

[0068] At decision 420, the path depends on whether an analysis function or analysis engine is enabled. In the example of FIG. 2, optional analysis engine 250 and knowledge base 260 were described. Table 1 summarizes options that may be utilized for analysis and diagnostics. Also see FIG. 2 and the description of analysis engine 250 and diagnostic engine 230.

[0069]

TABLE 1
SETTINGS FOR ANALYSIS ENGINE (AE)
AND DIAGNOSTIC ENGINE (DE)
	DE
	AE	OFF	ON
	OFF	No problem	Non-customized,
		analysis. No data	basic data
		capture.	capture.
	ON	Problem analysis.	Problem analysis
		No data capture.	and customized
		Good for software	data capture (full
		development.	FFDC function).

[0070] If analysis is not enabled, the “No” branch is taken at decision 420, and the next action is at block 460, basic response. Non-customized, basic data capture is an example of a basic response. On the other hand, if analysis is enabled, the “Yes” branch is taken at decision 420. For example, an AE is located, if one is configured. The next action is comparing a current incident to known incidents, at decision 430: does the current incident match a known incident? A method to analyze the incident [e.g. AE.analyze(Incident)] is called and directives may be found. If the current incident does not match a known incident, the “No” branch is taken at decision 430, and the next action is at block 460, a basic response. On the other hand, if the current incident matches a known incident, the “Yes” branch is taken at decision 430, and the next action is at block 440, returning or retrieving information that is relevant to the current incident. For example, this may involve returning or retrieving one or more items such as directives, solutions, work-arounds, and explanations.

[0071] This leads to a customized response at block 450. For example, this may involve one or more responses such as taking corrective action, providing a message, and capturing data. Consider data capture as a response. An incident may be bundled into a single persistent form that gathers data from multiple components. For efficiency, the data can be persisted as its size grows or at the completion of data gathering. Consider customized data capture as an example: a DM for a connection manager may be implemented to handle the directive strings: “dump_sql_statement”, “dump_result_set”, etc. Assuming that the default action for an SQL error 972 results in capturing the SQL statement but not the result set, a directive to dump the result set in the case of SQL error 972 could be easily added. The preceding example shows directives in the form of string arrays. To extend the example, consider how directives could give a troubleshooter the flexibility to request additional data under certain conditions. This could be achieved with no code changes in a DM operating in a production environment. Directive strings could be added to a knowledge base.

[0072] The last action in this example is outputting captured data, at block 460. The incident may be post processed to produce output that is easy to comprehend. Diagnostic Modules (DM's) may provide output data that is as self descriptive as possible to eliminate post processing, or provide tools for post processing. The diagnostic engine may request additional data to be provided by several components, through their respective DM's, to form a complete state or snapshot at the time of the incident. The additional components may be those present on the call stack or those called directly. The captured data should be written in a format that permits easy navigation.

[0073] One example is making the incident data readable by a servlet that is provided for FFDC. Customers could then grant service personnel and developers secure online access to FFDC data via this servlet. The servlet would allow development and support teams to browse and search incident data. Utilizing this servlet and granting access to service personnel would enable instant, remote, problem investigation.

[0074] A convention could be established for naming incident data files. An incident would generate a separate persistent file with a unique name. The following is an example:

• [0075] <log location>%incident%<processid>%<threadid>%<timestamp>
• [0076] where “%” may be a “/”, “\”, “.” etc.

[0077] Referring back to decision 430 in FIG. 4, consider some solutions to problems of inflexible error handling and recovery. The path along the left edge of the flow chart in FIG. 4 could be summarized as follows: analyzing an incident (decision 430 and block 440) then responding to said incident (block 450) while customizing the response, based on a result obtained by the analysis.

[0078] To look at the left path in more detail, a solution may involve accessing a knowledge base, and comparing a current incident to known incidents in said knowledge base. If a match is found, at decision 430 the “Yes” branch is taken to block 440, returning or retrieving information from said knowledge base, said information being relevant to said current incident. A knowledge base may comprise data elements such as: symptoms; directives; solutions; work-arounds; and explanations.

[0079] To consider symptoms in more detail, as a means for comparing a current incident to known incidents, there are various ways to identify or categorize an incident. For example, an incident may occur, where component A calls component B, which in turn calls component C, which encounters error E. Analyzing this incident typically would include looking for the sequence A, B, C, E in a knowledge base. Call stacks are a good way to identify incidents. Here are some examples of items that could be used as symptoms: message number, SQL code, method names extracted from a call stack, exception class names, component state data.

[0080] Directives, and messages conveying solutions, work-arounds, or explanations, may be retrieved from a knowledge base. Messages were described above in connection with FIG. 3. Here is another example of a message to an operator: “Exception X occurs when host name is typed in lower case, while host name in name tree is in mixed case.” Directives were described above in connection with FIGS. 2, 3 and 4. Dynamic tuning information for incident handling are referred to as “directives.” For increased flexibility, directives can assist a diagnostic module in customizing the response.

[0081] Achieving a customized response (FIG. 4, block 450) may involve storing customization information, retrieving said customization information, and automatically keeping said customization information up to date. An enhanced version of an analysis engine could be a means for keeping customization information up to date (i.e. a means for keeping a knowledge base up to date). This could be realized through the use of public File Transfer Protocol (FTP) sites to provide current customization information, for example. A utility could be used to add, delete or modify elements in a knowledge base. A monitor thread may handle dynamic updating of a knowledge base. After the latest version of a knowledge base is downloaded, a monitor thread may detect the change and reload a knowledge base.

[0082] A knowledge base architecture may be open for use by multiple components. The above-mentioned use of FTP is a way of making a knowledge base available to a plurality of software components. Providing a knowledge base through the use of a web site and database management software was described above in connection with FIG. 2. Extensible markup language (XML) or some other markup language could be used. Consider a hypothetical example involving two software products, the software products sold under the trademarks WEBSPHERE and COMMERCESUITE by IBM. A resource property file could be used as input to the analysis engine to specify a persistent location of a knowledge base (perhaps implemented as symptom and directive files), an FTP site for live updates (optional), and a software component, software product, or subsystem registration name. A configuration file for the analysis engine could be modeled as follows:

• [0083] ! List of products
• [0084] Subsystems=WebSphere CommerceSuite etc.
• [0085] ! List of symptom files and FTP update sites.
• [0086] WebSphereSymptomFile=/usr/IBMWebSphere/ . . . /symptomdb.xml
• [0087] WebSphereSymptomFileUpdateSite=ftp://ftp.software.com/appservers/websphere/ . . . /symptomdb.xml
• [0088] CommerceSuiteSymptomFile=/usr/IBMCommerceSuite/ . . . /symptomdb.xml
• [0089] CommerceSuiteSymptomFileUpdateSite=ftp://ftp.software.com/CommerceSuite/ . . . /symptomdb.xml

[0090] To extend this example, consider adding a third software component. The third component's installation process could call an analysis engine utility to add the third component's symptom and directive file into the resource list. The third component may also participate in FFDC by registering its diagnostic module(s) into the diagnostic engine.

[0091] Those skilled in the art will recognize that blocks in the above-mentioned flow chart could be arranged in a somewhat different order, but still describe the invention. Blocks could be added to the above-mentioned flow chart to describe details, or optional features; some blocks could be subtracted to show a simplified example.

[0092] In conclusion, we have shown examples of solutions for flexible error—handling, recovery, and problem-solving.

[0093] One of the possible implementations of the invention is an application, namely a set of instructions (program code) executed by a processor of a computer from a computer-usable medium such as a memory of a computer. Until required by the computer, the set of instructions may be stored in another computer memory, for example, in a hard disk drive, or in a removable memory such as an optical disk (for eventual use in a CD ROM) or floppy disk (for eventual use in a floppy disk drive), or downloaded via the Internet or other computer network. Thus, the present invention may be implemented as a computer-usable medium having computer-executable instructions for use in a computer. In addition, although the various methods described are conveniently implemented in a general-purpose computer selectively activated or reconfigured by software, one of ordinary skill in the art would also recognize that such methods may be carried out in hardware, in firmware, or in more specialized apparatus constructed to perform the required method steps.

[0094] While the invention has been shown and described with reference to particular embodiments thereof, it will be understood by those skilled in the art that the foregoing and other changes in form and detail may be made therein without departing from the spirit and scope of the invention. The appended claims are to encompass within their scope all such changes and modifications as are within the true spirit and scope of this invention. Furthermore, it is to be understood that the invention is solely defined by the appended claims. It will be understood by those with skill in the art that if a specific number of an introduced claim element is intended, such intent will be explicitly recited in the claim, and in the absence of such recitation no such limitation is present. For non-limiting example, as an aid to understanding, the appended claims may contain the introductory phrases “at least one” or “one or more” to introduce claim elements. However, the use of such phrases should not be construed to imply that the introduction of a claim element by indefinite articles such as “a” or “an” limits any particular claim containing such introduced claim element to inventions containing only one such element, even when the same claim includes the introductory phrases “at least one” or “one or more” and indefinite articles such as “a” or “an;” the same holds true for the use in the claims of definite articles.