Diagnosis of system health with event logs

[0001] The complexity of current computing systems and applications provided therein is quickly outgrowing the human ability to manage at an economic cost. For example, it is common to find data centers with thousands of host computing systems servicing hundreds to thousands of applications and components that provide web, computations and other services. In such distributed environments, diagnosis of failures and performance problems is an extremely difficult task for human operators. To facilitate diagnosis, commercial and open source management tools have been developed to measure and collect data from systems, networks and applications in the form of system metrics (i.e., data measurements), application metrics, and system and application event logs. However, with the large amounts of data collected, the operator is faced with the daunting task of manually going through the data, which is becoming unmanageable. These challenges have led researchers to propose the use of automated machine learning and statistical learning theory methods to aid with the detection, diagnosis and repair efforts of distributed systems and applications.

[0002] As referred herein, system and application event logs (hereinafter, “event logs” or “logs”) are records of system (both hardware and software) and application (software) events that have taken place in a system. Examples of event logs include but are not limited to failures to start a component or complete an action, system or application performance reaching predetermined thresholds, system or application errors, security events, network connection events. Each event entry typically includes a date stamp, a time stamp, and a message detailing the event. Unlike system metrics and application metrics, which contain structured numeric data, event logs are semi-structured and typically contain free text information. Event logs are essentially text messages written by the developers of the system and application. There are potentially many different messages. For example, it was found that there were more than 280,000 distinct event messages (after removing timestamps and fields containing numerical symbols only) in the event logs collected on one instance of an Information Technology (IT) system in a 9-month period.

[0003] Some prior solutions for diagnosing and repairing distributed systems and applications involve the use of search engines (e.g., as available from the Splunk Company of San Francisco, Splunk.com) or analysis modules (e.g., as available from LogLogic, Inc. of San Jose, Calif., loglogic.com) to perform indexing and parsing of the logs, whereby users have to provide adequate search queries to find desired information about the system or application health in the logs. Other prior solutions simply provide analyses of logs without correlating them with defined application or system health and typically require knowledge of the log structures and types of log messages a-priori. This leads to a finding of many types of data patterns in the logs that may not be important for diagnosing or forecasting a system or application behavior.

[0009] For simplicity and illustrative purposes, the principles of the embodiments are described by referring mainly to examples thereof. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the embodiments. It will be apparent however, to one of ordinary skill in the art, that the embodiments may be practiced without limitation to these specific details. In other instances, well known methods and structures have not been described in detail so as not to unnecessarily obscure the embodiments.

[0010] Because of the sheer size and number of types of event logs that may be generated in an IT system, there is a need for a systematic approach to distill a smaller set of “prototypical” or exemplary feature messages or clusters (hereinafter, “PM set”) from the event logs to simplify the monitoring of such logs. Once the PM set is defined, it is used to transduce the text event logs into numeric-based metrics for input into learning probabilistic classifier models (hereinafter, “classifier models” or “models”) that capture the correlation between the numeric-based metrics and predefined system health indicators. One type of such classifier models is described in U.S. Patent Application Publication No. 2006/0188011(hereinafter, “Publication PAP-011”), with publication date of Aug. 24, 2006 of U.S. patent application Ser. No. 10/987,611, filed Nov. 12, 2004, which is herein incorporated by reference in its entirety. Alternative embodiments are contemplated wherein other models for correlating the numeric-based metrics and the system health states are applicable as well. As referred herein, numeric-based metrics of a system are numeric (as opposed to textual messages in event logs) data measurements indicating conditions of the system and applications operating therein. As also referred herein, a system health indicator provides a status or state of the system in accordance with predefined acceptable thresholds. A system health indicator may have one or more states to indicate different levels of system health. An example of a system health indicator is a service level objective (SLO) typically found in a service level agreement (SLA). A system SLO may have two states, compliance or violation to indicate whether such a SLO has been complied or violated in the system. It also may have more than two states to indicate different levels of system health, such violation, 50% compliance, full compliance, etc.

[0011] Described herein are methods and systems that provide an efficient representation of event logs of an IT system and applications therein that is amenable to modeling techniques that produce diagnosis or forecasting of the system health. As referred herein, and as understood in the art, information technology, or IT, encompasses all forms of technology, including but not limited to the design, development, installation, and implementation of hardware and software information or computing systems and software applications, used to create, store, exchange and utilize information in its various forms including but not limited to business data, conversations, still images, motion pictures and multimedia presentations technology and with the design, development, installation, and implementation of information systems and applications. IT distributed environments may be employed, for example, by Internet Service Providers (ISP), web merchants, and web search engines to provide IT applications and services to users.

[0012] System

[0013] FIG. 1 illustrates a block diagram of an environment 100 wherein an embodiment may be practiced. The environment 100 includes an IT system 110, a metrics transducer module 120, and a model building engine or module 130. The IT system 110 includes one or more computing machines, each computing machine hosting one or more software applications or components. Examples of computing systems include but are not limited to personal computers, workstations, mainframe computers, web servers, application servers, database servers.

[0014] The IT system 110 is instrumented to generate system event logs, which includes event logs of both the system hardware and software applications therein, and monitored values for the predefined system health indicators in any manner known in the art. For example, commercially available data collection tools such as OpenView application by Hewlett Packard® Company and Microsoft NT 4.0 Performance Counters by Microsoft may be used to monitor the IT system 110.

[0015] The metrics transducer module 120 is operable to receive the event logs generated by the system 110 and transduce the event logs into metrics. In one embodiment, the metrics transducer module 120 distills a PM set from the received event logs by performing text clustering. The metrics transducer module 120 performs text clustering by combining similar event messages in the event logs to form a cluster. For example, messages generated by the same fprintf statements with slightly different parameters may be organized or classified into a single cluster. In effect, message clustering reverse engineers the “templates” that were used to generate the event messages and ignore the minor differences. In one embodiment, the message clustering is sequentially performed in an incremental fashion because over the lifetime of the system 110, code changes may be pushed into production that result in new messages appearing. Alternatively, it is possible to wait until all possible event messages are found in the collected events logs before they are batch clustered. The sequential clustering methodology is now described. However, it should be understood by one skilled in the art, based on the present disclosure herein, that minor modification may be done in order to apply such a methodology to batch clustering.

[0016] According to one embodiment, the similarity between two text messages found in the event logs is measured with a cosine distance function:

[0017] $\begin{array}{cc}{D}_{\cos }\left(A,B\right)=\frac{\sum _i^{}\mathrm{match}\left(a_i,b_i\right)}{\sqrt{A·B}},\mathrm{match}\left(a_i,b_i\right)=\left\{\begin{array}{cc}1& \mathrm{if}{a}_i=b_i\\ 0& \mathrm{otherwise}\end{array}\right\}& \mathrm{Equation}1\end{array}$
where A and B are the messages, |•| represents the number of words in a message, and a_i is the i'th word in message A. The cosine distance is a number between 0 and 1. When D_cos=1, the two messages A and B are identical, and when D_cos=0, the two messages are completely different. Upon seeing a new message, the clustering method compares the new message with the existing clusters (each cluster representing a prototypical feature message). If there exists a cluster to which the cosine distance is larger than a predefined threshold (e.g., 0.85), then the message is added to the existing cluster count. Otherwise, a new cluster is created with the new message. For example, the following event messages:

[0018] java.net.connectexception: db server connection refused; error host001; and

[0019] java.net.connectexception: db server connection refused; error code

are clustered together because their cosine distance is 0.857 (>0.85).

[0020] The metrics transducer module 120 then counts a number of times each prototypical feature message appears in a given time interval (set to match the interval of the predefined system health indicators) and use these counts as the input metrics for classifier models. It should be noted that the statistical properties of these feature-message-based metrics is different compared to system metrics or application metrics. In one embodiment, a different distribution for these input metrics is used in the classifier models. For system metrics, the normal distribution is used; whereas, for feature-message-based metrics, a modified Gamma distribution is used, which the inventors have observed to fit better than the normal and other distributions. Formally, the modified Gamma distribution follows:

[0021] $\begin{array}{cc}P\left(x=X\right)=\{\begin{array}{cc}{p}_z& \mathrm{if}X=0\\ \left(1-p_z\right)X^{k-1}\frac{e^{-X/\theta }}{\Gamma \left(k\right){\theta }^k}& \mathrm{otherwise}\end{array}& \mathrm{Equation}2\end{array}$
The value of x is always a non-negative integer. The modified Gamma distribution fits the feature message counts better because these counts exhibit a heavy tail with an additional large concentration of 0 counts.

[0022] The model building engine 130 is operable to receive the input metrics from the metrics transducer module 120 and the monitored values for the predefined system health indicators from the system 110 (directly from the system 110 or through the metrics transducer module 120). It then derives or generates classifier models that correlate the input metrics as transduced from the event logs to the monitored system health indicators, as described in the Publication PAP-011.

[0023] FIG. 2 illustrates a block diagram of a computerized system 200 that is operable to be used as a platform for implementing the IT system 110, the metrics transducer module 120, or the model building engine 130. Alternatively, the computerized system 200 is operable as a platform for implementing the entire environment 100 or any combination of the components 110-130 therein.

[0024] The computer system 200 includes one or more processors, such as processor 202, providing an execution platform for executing software. Thus, the computerized system 200 includes one or more single-core or multi-core processors of any of a number of computer processors, such as processors from Intel, AMD, and Cyrix. As referred herein, a computer processor may be a general-purpose processor, such as a central processing unit (CPU) or any other multi-purpose processor or microprocessor. A computer processor also may be a special-purpose processor, such as a graphics processing unit (GPU), an audio processor, a digital signal processor, or another processor dedicated for one or more processing purposes. Commands and data from the processor 202 are communicated over a communication bus 204 or through point-to-point links with other components in the computer system 200.

[0025] The computer system 200 also includes a main memory 206 where software is resident during runtime, and a secondary memory 208. The secondary memory 208 may also be a computer-readable medium (CRM) that may be used to store software programs, applications, or modules that implement the method 300 (as described later), or parts thereof. The main memory 206 and secondary memory 208 (and an optional removable storage unit 214) each includes, for example, a hard disk drive and/or a removable storage drive 212 representing a floppy diskette drive, a magnetic tape drive, a compact disk drive, etc., or a nonvolatile memory where a copy of the software is stored. In one example, the secondary memory 408 also includes ROM (read only memory), EPROM (erasable, programmable ROM), EEPROM (electrically erasable, programmable ROM), or any other electronic, optical, magnetic, or other storage or transmission device capable of providing a processor or processing unit with computer-readable instructions. The computer system 200 includes a display 220 connected via a display adapter 222, user interfaces comprising one or more input devices 218, such as a keyboard, a mouse, a stylus, and the like. However, the input devices 218 and the display 220 are optional. A network interface 230 is provided for communicating with other computer systems via, for example, a network.

[0026] Process

[0027] FIG. 3 illustrates a flowchart diagram of a method 300 for providing an efficient representation of event logs of an IT system and applications therein that is amenable to modeling techniques that produce diagnosis or forecasting of the system health, in accordance with one embodiment. For illustrative purposes only and not to be limiting thereof, the method 300 is discussed in the context of the system environment 100 illustrated in FIG. 1. Also for simplification purposes, the method 300 is discussed wherein system health indicators are represented by SLO binary states of compliance and violation. However, it should be understood that the method 300 is modifiable to accommodate other types of system health indicators and different number of health states without deviating from the scope of the present disclosure.

[0028] At 310, the text event logs are generated and values of predefined system health indicators, e.g., SLO states (compliance or violation), of the system 110 are monitored by the system 110, using any commercially available data collection tools, such as OpenView software available from Hewlett Packard® Company and Microsoft NT 4.0 Performance Counters available from Microsoft®. In one embodiment, for each predefined time period, window, or epoch (e.g., 5-minute intervals), there is one or more generated event logs, with a plurality of event messages therein, and a corresponding SLO state Sε{s⁺,s⁻} of the system 110. The generated event logs and monitored SLO states are received or obtained by the metrics transducer module 120.

[0029] At 320, the transducer module 120 transduces the event logs into numeric-based metrics by distilling a PM set from the event logs. It then counts the number of times each prototypical feature message appears in a given time interval and use these counts as the input metrics for the learning probability classifier models.

[0030] At 330, the model building engine 130 receives the transduced numeric-based metrics from the metrics transducer module 120 and the monitored values for the predefined system health indicators from the system 110 (directly from the system 110 or through the metrics transducer module 120). It then computes or derives classifier models that correlate the numeric-based metrics, and thus the generated event logs, to the monitored system health indicators. In one embodiment, the model building engine 130 builds a classifier model, as a Naïve Bayes model based on the transduced numeric-based metrics and a corresponding system health indicator state for each predefined time period, as described in the Publication PAP-011.

[0031] FIGS. 4A-B illustrate a flowchart diagram to provide further details of 320 in FIG. 3 above for transducing the event logs into numeric-based metrics, in accordance with one embodiment.

[0032] At 321, a distance function for sequentially clustering the event logs into a PM set is set or provided in the metrics transducer module 120. This distance function may be defined as desired by a user of the environment 100 or any component therein. An example of the distance function is as described earlier in Equation 1.

[0033] At 322, a threshold for identifying a similarity between event messages based on the distance function is set or provided in the metrics transducer module 120. As with the distance function, the similarity threshold may be defined as desired by a user. For example, if the cosine distance function as described in Equation 1 is employed, the similarity threshold may be a value near the maximum value of 1 (which indicates the two compared messages are identical). Thus, whenever the calculated distance between two messages is equal to or greater than such a threshold value, the two messages are deemed similar for clustering.

[0034] At 323, the PM set is initialized to empty.

[0035] At 324, for each predefined time window or period (e.g., each 5-minute interval), the metrics transducer module 120 employs the predefined distance function to compute, in order, a distance between each of the event messages found in the event logs received for such a predefined time window and a prototypical feature message found in the PM set.

[0036] At 325, if the computed distance between an event message and all prototypical feature messages is smaller than (or either equal to or smaller than) the predefined similarity threshold, the metrics transducer module 120 designates such an event message as a prototypical feature message in the PM set, i.e., a new member in the PM set, for comparison with other event messages. It should be noted that the first event message is automatically designated as a prototypical feature message because there is initially no other prototypical feature message for the first event message to compare. Accordingly, the PM set is dynamically created for each predefined time window.

[0037] At 326, if the computed distance between an event message and a prototypical feature message is greater than or equal to (or just greater than) the predefined similarity threshold, the metrics transducer module 120 maps such an event message to the particular prototypical feature message and increment by one a count of the particular prototypical feature message.

[0038] At 327, the metrics transducer module 120 aggregates or counts the number of times each prototypical feature message appears in each predefined period and use these counts as the input metrics for the learning probability classifier models.

[0039] Accordingly, for each predefined time window, the metrics transducer module 120 generates a pair of a vector {right arrow over (M)} of values of the transduced numeric-based metrics and a corresponding SLO state Sε{s⁺,s⁻} (compliance or violation, respectively) of the system 110. Each element m_i of the vector {right arrow over (M)} contains a value indicating the total number of a particular prototypical feature message that is found in the received event logs for each predefined time window. Thus, for multiple predefined time windows, there are multiple pairs of <{right arrow over (M)},S>. These pairs are input to the model building engine 130 to create a model for each SLO state relating each state to different values and patterns of metrics that are collected and received by the metrics transducer module 120 from the measured system 110.

[0040] In recap, the systems and methods as described herein are operable to provide compact representations of raw textual data in system event logs and transform such representations into numeric-based metrics for system modeling techniques that can produce diagnosis or forecasting of the system health.

[0041] What has been described and illustrated herein is an embodiment along with some of its variations. The terms, descriptions and figures used herein are set forth by way of illustration only and are not meant as limitations. Those skilled in the art will recognize that many variations are possible within the spirit and scope of the subject matter, which is intended to be defined by the following claims—and their equivalents—in which all terms are meant in their broadest reasonable sense unless otherwise indicated.