Detecting anomalies in time series data

[0001] Many computing system generate metrics during system operation. Such metrics may describe various aspects of the system such as system performance and errors encountered by the system. Such metrics may be collected and analyzed to better understand the behavior of the system.

[0008] The various embodiments described herein relate to processing of time series data, and more specifically, to automated and unsupervised detection of anomalies within time series data. Various anomaly detection embodiments described herein can automatically monitor large numbers of distinct time series at different time resolutions (e.g., minutes, hours, days, weeks, etc.) and detect anomalies therein. Various embodiments described herein can detect anomalies in time series that are non-stationary, non-ergodic, heteroskedastic, and/or contain cyclical or seasonal effects. Various types of anomalies can be detected, such as slopes, step functions (i.e., discontinuities), spikes (e.g., V shapes), combinations of spikes and step functions (e.g., V shapes in which the trailing edge of the V has a step function relative to the leading edge of the V), and shaking (e.g., transitions between low variance and high variance).

[0009] Some of the anomaly detection embodiments described herein detect anomalies that are within variance, yet statistically significant. An example of such an anomaly is a small but persistent 20 ms step function in a metric with 50 ms of standard deviation.

[0010] In some of the embodiments described herein, multiple co-occurring anomalies are grouped together as “quakes.” The term “quake” alludes to seismologic time series, in which physically separated seismographs record individual tremors at different locations that are then combined and reported as a single earthquake.

[0011] The examples discussed herein relate to metrics generated during operating of a computing system or environment. However, the anomaly detection techniques disclosed herein also apply to time series that represent other types of data. In the following discussion, a general description of the system and its components is provided, followed by a discussion of the operation of the same.

[0012] With reference to FIG. 1, shown is a networked environment 100 according to various embodiments. The networked environment 100 includes a computing environment 103 and one or more client devices 106, in data communication by way of a network 109. The network 109 includes, for example, the Internet, intranets, extranets, wide area networks (WANs), local area networks (LANs), wired networks, wireless networks, or other suitable networks, or any combination of two or more such networks.

[0013] The computing environment 103 may comprise, for example, a server computer or any other system providing computing capacity. Alternatively, a plurality of computing devices may be employed that are arranged, for example, in one or more server banks or computer banks or other arrangements. For example, a plurality of computing devices together may comprise a cloud computing resource, a grid computing resource, and/or any other distributed computing arrangement. Such computing devices may be located in a single installation or may be distributed among many different geographical locations.

[0014] The computing environment 103 may be operated by a hosting provider to host network sites for various entities, or may be controlled and managed by a provider entity that provides access to resources of the computing environment 103 to entities for a fee under a utility computing model. For example, entities may arrange for access to applications executing in the computing environment 103, for storage in the computing environment 103, or for messaging between components in the computing environment 103. Such a system is sometimes referred to as “cloud computing system,” but the term “cloud computing” is not intended to be limiting.

[0015] The computing environment 103 may utilize differing hardware platforms and may have differing hardware configurations of resources, for example, of central processing units (CPUs) that provide general-purpose processing resources, graphics processing units (GPUs) that provide graphics processing resources, system memory, data storage characteristics such as capacity, storage bandwidth, and storage input/output operations per second (IOPS), network bandwidth, and/or other hardware characteristics. The computing environment 103 may be executing instances of virtual machines used to provide general computing capability to entities. Such entities may be capable of terminating machine instances or launching machine instances whenever they desire, thereby providing an elastic computing resource.

[0016] Various applications and/or other functionality may be executed in the computing environment 103 according to various embodiments. The components executed on the computing environment 103 include, for example, a metrics collector 112, a system event logger 113, an anomaly detector 115, an anomaly portal 118, computing services 121, and other applications, services, processes, systems, engines, or functionality not discussed in detail herein. Various data is stored in a data store 124 that is accessible to the computing environment 103. The data store 124 may be representative of a plurality of data stores as can be appreciated. The data stored in the data store 124, for example, is associated with the operation of the various applications and/or functional entities of the computing environment 103 as described herein. The data stored in the data store 124 includes, for example, metrics 127, time series 130, system events 133, anomalies 136, anomaly groups 139, and potentially other data.

[0017] A computing service 121 is executed to provide a particular functionality to the computing environment 103 as a whole, and in doing so, may export one or more functions through a programmatic interface. A computing service 121 may be implemented, for example, as a web service. Various computing services 121 may (singly or in combination) implement features such as a network storage service, a cloud computing service, a search service, an advertising service, a network page server, an electronic commerce marketplace, or components thereof.

[0018] In providing their respective functions, the various computing services 121 may generate one or more metrics 127. For example, a network page server may track metrics 127 such as latency in generating network pages, the number of pages generated in a time period, the number of customers requesting a particular network page, etc. As another example, a network storage service may track metrics 127 such as the latency in fulfilling a storage request, the number of customers creating a storage object, etc. As yet another example, a search service may track metrics 127 such as the latency in fulfilling a search query, the number of search queries fulfilled in a particular time period, etc.

[0019] The system event logger 113 is executed to record system events 133 associated with the computing environment 103. Examples of such system events 133 include installation or deployment of new computing services 121 in the computing environment 103, updates to existing computing services 121 in the computing environment 103, diagnostics tests running in the computing environment 103, changes to the hardware configuration of the computing environment 103, and/or changes to the configuration of the network 109.

[0020] The metrics collector 112 is executed to collect the various metrics 127 produced by the computing services 121 and to store the metrics 127 in the data store 124. The metrics collector 112 may also aggregate the collected metrics 127 in various ways. For example, the metrics collector 112 may aggregate all instances of a metric 127 from a particular computing service 121 (e.g., the search service) that reside in a particular data center, that reside in a particular geographic region, etc.

[0021] Each particular metric 127 has a particular value at a particular time, and these metric values may be collected into a time series 130 for the metric 127. That is, latency metrics 127 produced by the network page server may be gathered into a network page latency time series 130, which can be viewed as a series of {time, value} pairs. Each time series 130 has a particular resolution or periodicity, e.g., hourly, daily, weekly, monthly, quarterly, etc.

[0022] The anomaly detector 115 is executed to detect anomalies in one or more of these time series 130. The functionality of the anomaly detector 115 will described in further detail below, but a brief description will now be provided. In summary, the anomaly detector 115 processes the time series 130 to identify unexpected changes in the time series 130 and designate these occurrences as anomalies 136. A representative non-limiting list of anomalies 136 includes unexpected changes in the mean of a time series 130, unexpected changes in the variance of a time series 130, outlier values of a time series 130, and missing and/or unexpectedly appearing data in a time series 130. The anomaly detector 115 may assign a magnitude or score to the detected anomaly 136.

[0023] The anomaly detector 115 may further determine whether any other anomalies 136 are present in the same time series 130 or a different time series 130, and if so designate the co-occurring anomalies 136 as a co-occurring anomaly group 139. The anomaly detector 115 may further determine whether one or more system events 133 occur during the same time period as the detected anomalies 136.

[0024] The anomaly detector 115 may store a description of the anomalies 136 in the data store 124. Such a description may include, for example, the anomaly type, the time series 130 in which the anomaly 136 was detected, the time at which the anomaly 136 occurred, and the existence and/or identity of a co-occurring anomaly group 139.

[0025] The anomaly portal 118 is executed to provide information about detected anomalies 136 to client devices 106. In some embodiments, the anomaly portal 118 is integrated with the anomaly detector 115. In some embodiments, the anomaly portal 118 is integrated into a comprehensive analytics tool that also provides other types of analytics data to developers of the computing environment 103.

[0026] Having discussed the computing environment 103, the client device 106 will now be considered. The client device 106 is representative of a plurality of client devices that may be coupled to the network 109. The client device 106 may comprise, for example, a processor-based system such as a computer system. Such a computer system may be embodied in the form of a desktop computer, a laptop computer, a personal digital assistant, a cellular telephone, a smart phone, a set-top box, a television, a music player, a video player, a media player, a web pad, a tablet computer system, a game console, an electronic book reader, or other devices with like capability. The client device 106 may be configured to execute various applications such as a client application 142. The client device 106 may be configured to execute applications beyond the client application 142 such as, for example, email applications, messaging applications, and/or other applications.

[0027] The client application 142 may be executed in the client device 106, for example, to receive information about detected anomalies 136 from the anomaly detector 115. Examples of such information include the anomaly type, the magnitude, the time series 130, any co-occurring anomaly groups 139, any co-occurring system events 133, and so on. The client application 142 may then render information about the detected anomalies 136 as provided by the anomaly detector 115 for display to a user. The display may be graphical in nature. The client application 142 may also be executed to query the anomaly detector 115 for particular reports, for example, a list of all anomalies 136 detected within a time period, a list of anomalies 136 associated with particular metrics 127, a list of anomalies 136 associated with particular computing services 121, etc. The client application 142 may be execute in the context of a browser (not shown), or may be implemented as a standalone application.

[0028] Next, a general description of the operation of the various components of the computing environment 103 is provided. To begin, the various computing services 121 execute and, in doing so, generate metrics 127. These metrics 127 are periodically collected by the metrics collector 112 and stored in the data store 124. The metrics 127 are processed to generate a time series 130 for each metric 127. The anomaly detector 115 processes one or more of the time series 130 to detect anomalies 136 within the time series 130 and to assign a magnitude to each anomaly 136. Some embodiments of the anomaly detector 115 may perform additional processing, for example, to identify groups of co-occurring anomalies 136 as well as system events 133 that co-occur with identified anomalies 136. Information about the detected anomalies 136, anomaly groups 139, and system events 133 is stored in the data store 124.

[0029] A user of the client device 106 obtains information about anomalies 136 in the computing environment 103 by interacting with the client application 142. The client application 142 generates a request for anomaly information which is serviced by the anomaly portal 118. The client application 142 may then display information about the detected anomalies 136, anomaly groups 139, and/or system events 133, for example, in a graphical format.

[0030] Moving on to FIG. 2, shown is a flowchart that provides one example of the operation of portion(s) of the anomaly detector 115 according to various embodiments. It is understood that the flowchart of FIG. 2 provides merely an example of the many different types of functional arrangements that may be employed to implement the operation of portion(s) of the anomaly detector 115 as described herein. As an alternative, the flowchart of FIG. 2 may be viewed as depicting an example of steps of a method implemented in the computing environment 103 (FIG. 1) according to one or more embodiments.

[0031] Beginning at box 203, the anomaly detector 115 obtains one or more metrics 127 associated with the computing environment 103. As explained above, these metrics 127 are generated by the various computing services 121 during the operation of the computing environment 103 and are stored in the data store 124. In some embodiments, the metrics 127 are periodically collected by the metrics collector 112, and the anomaly detector 115 requests the collected metrics 127 from the metrics collector 112. In other embodiments, the anomaly detector 115 retrieves the metrics 127 from the data store 124.

[0032] Next, at box 206, the anomaly detector 115 processes collected values for each metric 127 to produce a time series 130 for each metric 127. For example, if at box 203 the anomaly detector 115 obtains values for a search query latency metric 127 and a page load frequency metric 127, then at box 206 the anomaly detector 115 generates a search query latency time series 130 and a page load frequency time series 130. The time series 130 may be generated by, for example, adding to the series 130 each particular value of the metric 127, along with a timestamp stored with the value. In some embodiments, the metrics 127 may already be grouped in a time series, in which case the processing at box 206 may be omitted.

[0033] At box 209, the anomaly detector 115 looks for one or more anomalies 136 in the one or more time series 130 generated at box 206. This anomaly detection process will be discussed in further detail in connection with FIG. 3, but it should be noted that the process may include classifying identified anomalies 136 into anomaly types as well as determining a time at which each anomaly 136 occurred within the time series 130.

[0034] Next, at box 212, the anomaly detector 115 examines any anomalies 136 detected at box 209 to identify any of the anomalies 136 that occur at the same time or in the same time period. To do so, the anomaly detector 115 may partition the time series 130 into various groups or buckets according to the time of occurrence of the anomaly 136. The duration of this co-occurrence time period may be predefined, for example, through the anomaly portal 118. The co-occurring anomalies 136 may be designated as a co-occurring anomaly group 139, which may be referred to herein as a “quake.”

[0035] Next, at box 215, the anomaly detector 115 looks for a correlation between the detected anomalies 136 and one or more system events 133 stored in the data store 124. To do so, the anomaly detector 115 may examine timestamps for the detected anomalies 136 or the anomaly groups 139, along with timestamps for the system events 133, and look for system events 133 that overlap in time with a detected anomaly 136 or with a co-occurring anomaly group 139. The system events 133 may include, for example, changes to the computing services 121 that execute in the computing environment 103, changes to the configuration of the computing environment 103, and/or changes to the network 109.

[0036] At box 218, the anomaly detector 115 assigns a magnitude or score to the detected anomaly, or to the group of co-occurring anomalies 136. Various factors may be used to compute this score as should be appreciated. A non-limiting list of such factors includes a measure of significance or importance of the metric 127 from which the anomaly 136 was derived, the number of time series 130 involved in the anomaly 136, and the magnitudes of the individual anomaly 136 in a group of co-occurring anomalies 136.

[0037] Next, at box 221, the anomaly detector 115 reports the detected anomalies 136, the assigned magnitude, and any correlated external events. For example, the anomaly detector 115 may provide this information through the anomaly portal 118. The anomaly detector 115 may encode this information for display to a client device 106, or may provide the information to the anomaly portal 118 for such encoding. The process of FIG. 2 is then complete.

[0038] In the embodiment described above in connection with FIG. 2, the anomaly detector 115 assigns scores to individual anomalies 136 during the same process that assigns a magnitude to co-occurring anomaly groups 139. In other embodiments, the anomaly detector 115 assigns scores to individual anomalies 136 as those anomalies 136 are detected at box 209.

[0039] In the embodiment described above in connection with FIG. 2, the anomaly detector 115 processes all the collected metrics 127 in order to identify anomalies 136. In other embodiments, only a subset of the collected metrics 127 is processed at box 206, where the subset is configurable through the anomaly portal 118. In still other embodiments, anomalies 136 are identified in the all the collected metrics 127, but the processing is staggered rather than being performed all at once.

[0040] Turning now to FIG. 3, shown is a flowchart that another example of the operation of portion(s) of the anomaly detector 115 according to various embodiments. Specifically, the flowchart of FIG. 3 provides further details of the operation of box 209 from FIG. 2, which detects anomalies 136 in one or more time series 130. Although the process of FIG. 3 is described in terms of operations on a particular time series 130, some embodiments apply the operation of FIG. 3 to each time series 130. It is understood that the flowchart of FIG. 3 provides merely an example of the many different types of functional arrangements that may be employed to implement the operation of portion(s) of the anomaly detector 115 as described herein. As an alternative, the flowchart of FIG. 3 may be viewed as depicting an example of steps of a method implemented in the anomaly detector 115 (FIG. 1) according to one or more embodiments.

[0041] Beginning at box 303, the anomaly detector 115 decomposes the time series 130 into a smoothed component and a noise component. To this end, the anomaly detector 115 may compute a moving median of the time series 130 to filter out the noise component, leaving the smoothed component. The window size for the moving median is based on the sample rate of the time series 130. Various techniques may be used to handle edges and missing values in the time series 130 as may be appreciated. In some embodiments, a wavelet filter is applied to the computed medians.

[0042] The point wise sum of the two components produced at box 303 is the original time series 130. Thus, the noise component can be viewed as the point wise difference between the time series 130 and the smoothed component. Conversely, the smoothed component can be viewed as the point wise difference between the original time series 130 and the noise component.

[0043] Next, at box 306, the anomaly detector 115 examines the noise component extracted at box 303 to identify any outliers present in the component, using a moving window with a predefined window size. To this end, the anomaly detector 115 may compute a central value and a moving interquartile range (mIQR) of the noise component. Data points that are more than a predefined multiple of the mIQR distance from the mean may be designated as anomalies 136, and more specifically, as spike anomalies 136. In one embodiment, the predefined multiple is three. Using an interquartile range as the moving window has the advantage of moving with the variance over time while not being too sensitive to outlier effects. In some embodiments, symmetric quartiles are used (e.g., 25th and 75th percentiles, 10th and 90th percentiles, etc.), while asymmetric quartiles are used in other embodiments. Various types of central values may be used, including but not limited to the mean and the median.

[0044] At box 309, the anomaly detector 115 examines the smoothed component extracted at box 303 to identify any step functions present in in the component. In some embodiments, a step function corresponds to a change in the mean of the smoothed component. The step function detection process will be discussed in further detail in connection with FIG. 4.

[0045] Next, at box 312, the anomaly detector 115 examines the time series 130 to identify any changes in the variance of the time series 130. To this end, the anomaly detector 115 may compute the moving variance in the time series 130 and designate, as an anomaly 136, a change in variance above a predefined threshold. More specifically, such an anomaly 136 may be designated as a variance anomaly 136. In some embodiments, the anomaly detector 115 may compute the moving variance of the smooth component extracted at box 303. In other embodiments, the anomaly detector 115 may compute the moving variance for the original time series 130.

[0046] At box 315, the anomaly detector 115 examines the time series 130 to identify any changes in density in the values of the time series 130. Such changes in density may represent data points missing from the time series 130 or data points present in the time series 130 that were not previously present. A change in density above a predefined threshold is then designated as an anomaly 136, and more specifically, as a density change anomaly 136. In some embodiments, the anomaly detector 115 may compute the moving density of the original time series 130. In other embodiments, the anomaly detector 115 may separately compute moving densities for the noise component and smooth component extracted at box 303. The process of FIG. 3 is complete.

[0047] With reference now to FIG. 4, shown is a flowchart that another example of the operation of portion(s) of the anomaly detector 115 according to various embodiments. Specifically, the flowchart of FIG. 4 provides further details of the operation of box 309 from FIG. 3, which detects step functions in a smoothed component of a time series 130. Although the process of FIG. 4 is described in terms of operations on a particular time series 130, some embodiments apply the operation of FIG. 4 to each time series 130. It is understood that the flowchart of FIG. 4 provides merely an example of the many different types of functional arrangements that may be employed to implement the operation of portion(s) of the anomaly detector 115 as described herein. As an alternative, the flowchart of FIG. 4 may be viewed as depicting an example of steps of a method implemented in the anomaly detector 115 (FIG. 1) according to one or more embodiments.

[0048] Beginning at box 403, the anomaly detector 115 obtains the periodicity of the smoothed component of the time series 130. In some embodiments, the periodicity is known a priori and is retrieved from the data store 124. In other embodiments, the periodicity is computed by the anomaly detector 115 using a technique such as spectral analysis. A non-limiting list of example periods includes hourly, daily, weekly, monthly, quarterly, and yearly.

[0049] Next, at box 406, the anomaly detector 115 partitions the smoothed component into a set of adjacent “stripes,” ending at the most recently collected time. Each stripe is one period in length or duration. In some embodiments, the anomaly detector 115 discards the oldest stripe if it does not contain data for an entire period.

[0050] At box 409, the anomaly detector 115 takes a corresponding point from each stripe, separated by the period, and creates N time series 130 from these points, where N is the periodicity obtained at box 403. For example, if the period is weekly, seven new time series 130 are created. The first generated time series 130 includes the Sunday data point from the oldest stripe, the Sunday data point from the second oldest stripe, and so on, finishing with the Sunday data point from the current stripe. Similarly, the seventh and last time series 130 generated at box 409 includes the Saturday data point from the oldest stripe and finishes with the Saturday data point from the current stripe.

[0051] Next, at box 412, the anomaly detector 115 computes a central value for each periodic time series 130 generated at box 409, producing a plurality of central values. A central value may include, but is not limited to, the mean, the median, or the exponentially weighted mean. The anomaly detector 115 creates another time series 130 that includes these central values. The anomaly detector 115 designates the central value time series 130 as the normal time series 130.

[0052] At box 415, the anomaly detector 115 processes the most recent stripe to generate another time series 130 that includes the preceding N points in the most recent stripe, where N is a predefined value between one and the period or stripe length. The anomaly detector 115 designates the preceding time series 130 as the current trend time series 130.

[0053] Next, at box 418, the anomaly detector 115 predicts the next M points in the smoothed component from the normal time series 130 generated at box 412 and the current trend time series 130 generated at box 415. The anomaly detector 115 identifies any points in the smoothed component that deviate significantly from the predicted M next points, and designates such deviant points as anomalies 136. More specifically, the anomaly detector 115 designates any points identified at box 418 as mean anomalies 136 or step function anomalies 136. The measure of significance used at box 418 can be configured, for example, through the anomaly portal 118. The process of FIG. 4 is complete.

[0054] Moving on to FIG. 5, shown is a schematic block diagram of a computing device 500 according to an embodiment of the present disclosure. The computing device 500 corresponds to a representative computing device which may be employed in the computing environment 103 (FIG. 1). The computing device 500 includes at least one processor circuit, for example, having a processor 503 and a memory 506, both of which are coupled to a local interface 509. To this end, the computing device 500 may comprise, for example, at least one server computer or like device. The local interface 509 may comprise, for example, a data bus with an accompanying address/control bus or other bus structure as can be appreciated.

[0055] Stored in the memory 506 are both data and several components that are executable by the processor 503. In particular, stored in the memory 506 and executable by the processor 503 are the metrics collector 112, the system event logger 113, the anomaly detector 115, the anomaly portal 118, computing services 121, and potentially other applications. In addition, an operating system may be stored in the memory 506 and executable by the processor 503. While not illustrated, the client device 106 (FIG. 1) also includes components like those shown in FIG. 5, whereby the client application 142 (FIG. 1) is stored in a memory and executable by a processor.

[0056] It is understood that there may be other applications that are stored in the memory 506 and are executable by the processor 503 as can be appreciated. Where any component discussed herein is implemented in the form of software, any one of a number of programming languages may be employed such as, for example, C, C++, C#, Objective C, Java, JavaScript, Perl, PHP, Visual Basic, Python, Ruby, Delphi, Flash, or other programming languages.

[0057] A number of software components are stored in the memory 506 and are executable by the processor 503. In this respect, the term “executable” means a program file that is in a form that can ultimately be run by the processor 503. Examples of executable programs may be, for example, a compiled program that can be translated into machine code in a format that can be loaded into a random access portion of the memory 506 and executed by the processor 503, source code that may be expressed in proper format such as object code that is capable of being loaded into a random access portion of the memory 506 and executed by the processor 503, or source code that may be interpreted by another executable program to generate instructions in a random access portion of the memory 506 and executed by the processor 503, etc. An executable program may be stored in any portion or component of the memory 506 including, for example, random access memory (RAM), read-only memory (ROM), hard drive, solid-state drive, USB flash drive, memory card, optical disc such as compact disc (CD) or digital versatile disc (DVD), floppy disk, magnetic tape, or other memory components.

[0058] The memory 506 is defined herein as including both volatile and nonvolatile memory and data storage components. Volatile components are those that do not retain data values upon loss of power. Nonvolatile components are those that retain data upon a loss of power. Thus, the memory 506 may comprise, for example, random access memory (RAM), read-only memory (ROM), hard disk drives, solid-state drives, USB flash drives, memory cards accessed via a memory card reader, floppy disks accessed via an associated floppy disk drive, optical discs accessed via an optical disc drive, magnetic tapes accessed via an appropriate tape drive, and/or other memory components, or a combination of any two or more of these memory components. In addition, the RAM may comprise, for example, static random access memory (SRAM), dynamic random access memory (DRAM), or magnetic random access memory (MRAM) and other such devices. The ROM may comprise, for example, a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or other like memory device.

[0059] Also, the processor 503 may represent multiple processors and the memory 506 may represent multiple memories that operate in parallel processing circuits, respectively. In such a case, the local interface 509 may be an appropriate network 109 (FIG. 1) that facilitates communication between any two of the multiple processors 503, between any of the processors 503 and any of the memories 506, or between any two of the memories 506, etc. The local interface 509 may comprise additional systems designed to coordinate this communication, including, for example, performing load balancing. The processor 503 may be of electrical or of some other available construction.

[0060] Although the metrics collector 112, the system event logger 113, the he anomaly detector 115, the anomaly portal 118, computing services 121 and other various systems described herein may be embodied in software or code executed by general purpose hardware as discussed above, as an alternative the same may also be embodied in dedicated hardware or a combination of software/general purpose hardware and dedicated hardware. If embodied in dedicated hardware, each can be implemented as a circuit or state machine that employs any one of or a combination of a number of technologies. These technologies may include, but are not limited to, discrete logic circuits having logic gates for implementing various logic functions upon an application of one or more data signals, application specific integrated circuits having appropriate logic gates, or other components, etc. Such technologies are generally well known by those skilled in the art and, consequently, are not described in detail herein.

[0061] The flowcharts of FIGS. 2, 3, and 4 show the functionality and operation of an implementation of portions of the anomaly detector 115. If embodied in software, each block may represent a module, segment, or portion of code that comprises program instructions to implement the specified logical function(s). The program instructions may be embodied in the form of source code that comprises human-readable statements written in a programming language or machine code that comprises numerical instructions recognizable by a suitable execution system such as one of the processors 503 in a computer system or other system. The machine code may be converted from the source code, etc. If embodied in hardware, each block may represent a circuit or a number of interconnected circuits to implement the specified logical function(s).

[0062] Although the flowcharts of FIGS. 2, 3, and 4 show a specific order of execution, it is understood that the order of execution may differ from that which is depicted. For example, the order of execution of two or more blocks may be scrambled relative to the order shown. Also, two or more blocks shown in succession in the flowcharts of FIGS. 2, 3, and 4 may be executed concurrently or with partial concurrence. Further, in some embodiments, one or more of the blocks shown in the flowcharts of FIGS. 2, 3, and 4 may be skipped or omitted. In addition, any number of counters, state variables, warning semaphores, or messages might be added to the logical flow described herein, for purposes of enhanced utility, accounting, performance measurement, or providing troubleshooting aids, etc. It is understood that all such variations are within the scope of the present disclosure.

[0063] Also, any logic or application described herein (including the metrics collector 112, the system event logger 113, the he anomaly detector 115, the anomaly portal 118, and computing services 121) that comprises software or code can be embodied in any non-transitory computer-readable medium for use by or in connection with an instruction execution system such as, for example, the processor 503 in a computer system or other system. In this sense, the logic may comprise, for example, statements including instructions and declarations that can be fetched from the computer-readable medium and executed by the instruction execution system. In the context of the present disclosure, a “computer-readable medium” can be any medium that can contain, store, or maintain the logic or application described herein for use by or in connection with the instruction execution system. The computer-readable medium can comprise any one of many physical media such as, for example, magnetic, optical, or semiconductor media. More specific examples of a suitable computer-readable medium would include, but are not limited to, magnetic tapes, magnetic floppy diskettes, magnetic hard drives, memory cards, solid-state drives, USB flash drives, or optical discs. Also, the computer-readable medium may be a random access memory (RAM) including, for example, static random access memory (SRAM) and dynamic random access memory (DRAM), or magnetic random access memory (MRAM). In addition, the computer-readable medium may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or other type of memory device.

[0064] It should be emphasized that the above-described embodiments of the present disclosure are merely possible examples of implementations set forth for a clear understanding of the principles of the disclosure. Many variations and modifications may be made to the above-described embodiment(s) without departing substantially from the spirit and principles of the disclosure. All such modifications and variations are intended to be included herein within the scope of this disclosure and protected by the following claims.