Method for detecting outlier measures of activity

[0002] This invention relates to the field of measurement of activity of various kinds, such as (but is not restricted to) traffic in a data communication network, and more particularly to a method of measuring the activity to an improved accuracy.

[0003] In measuring activity of various kinds, such as electrical, fluid, information, object, etc. flow, and conditions, performance, and other activities, measurements are taken at particular times. These measurements often result in the determination of outlier points, particularly in conditions where the distribution of points is unpredictable. This problem is very severe in data communication networks.

[0004] Measurements are usually made with error; a set of measurements of the same value is often normally distributed about an exact point. However, some of these measurements will have enormous errors associated with them, due to some gross experimental mistake such as recording millimeters rather than meters. The detection and rejection of these outliers can be readily made where the expected distribution of the measurements is known and has a finite variance. However, in self similar distributions the variance is infinite (or is only limited by the maximum capacity for activity of the object). Outlier points in self similar distributions cannot therefore be recognized and rejected by using the expected distribution.

[0005] Leland et al, as described in the publication “On the Self Similar Nature of Ethernet Traffic” by W. E. Leland, W. Willinger, M. S. Taqqu, D. V. Wilson: ACM SIFCOMM, Computer Communication Review: pp 204-213, January 1995, discovered that the distribution of data network traffic is self similar, so that traffic measurements made on data networks suffer from the problem of outlier rejection.

[0006] Moreover, the frequency of outliers in data networks is extremely high. When measured over a wide variety of data networks the average outlier rate was 1%, practically all of these outliers being high. These outliers make the detection of alarm levels of activity in data networks very prone to error and also makes the forecasting of activity in data network very unreliable.

[0007] The only previously known general outlier detection method is one which rejects outlier measurements if the measurement is greater than or less than a possible range of values. All other specific methods rely on knowledge of the distribution of possible values.

[0008] A complex model allows the use of more than one variable in forecasting the distribution of a single variable, where feasible. In the absence of any model, the previous distribution of a variable is the best forecaster of the future distribution of that variable. Data communications network managers very much want to know when their networks or parts of their networks will run out of capacity. It was believed for many years that models of network behaviour could predict the future; since this used to be true for voice networks, it was assumed it would be true for data. Therefore models were developed which tried to predict future peaks based on previous means and peaks.

[0009] However, the work of Leland et al referred to above, and many others subsequently, have now shown that data network traffic is self similar (fractal). This implies that the variance of data network traffic is not only infinite but also is not even related to the mean. Therefore attempting to predict future peaks using any model that includes the mean is clearly wrong.

[0010] Moreover, if the variance is truly infinite, future peaks cannot even be predicted from previous peaks. In other words, self similar distributions may have a lower limit but do not have an upper limit.

[0011] However, it had not previously been observed that since communications lines do have upper limits in capacity, therefore the distributions in them cannot be truly self similar and their variances are definitely finite. Under these conditions the previous peak values can be used to predict future peak values, but the relationship between the mean and the peak remains indeterminate.

[0012] The idea of using linear fits to prior peaks to forecast future peaks in data communications networks had been previously invented by N. W. Dawes. Once tested, however, the problem of the peaks being heavily contaminated by invalid data points was noticed. The problem rate was found by experiment to be very high, with most forecasts being significantly faulty. Moreover, attempting to report the peak activity of any port in a network (the top talker) over even the last 24 hours was found to be routinely wrong, as the following will illustrate.

[0013] Activity values recorded by data communications devices about their own activity has been found in practice to be astonishingly error prone, with an average outlier rate of 1%. For example, attempting to determine the daily peak traffic rate on a single interface by measuring the rate every minute requires measuring 1,440 points per day, but on average 14 of these would be outliers, almost all being high. The daily peak point under these conditions would be 14 times more likely to be an outlier than a genuine value. The outliers were observed to be randomly distributed, so a simple filter that rejected activity levels outside the physical capacity of the interface was added. This rejected 10,000 outliers for every 1 accepted.

[0014] However, in monitoring even moderate sized networks of 1,000 devices and 10,000 communications interfaces, about 10 outliers still passed through this filter every day (scattered over these 10,000 interfaces). This left about 4% of all forecasts seriously in error. Moreover, analyses such as finding the busiest interface even just over the last day were routinely wrong. Analysis of the immediately previous year on such a network (a not uncommon requirement) would require 3.65×10⁹points to be cleared of outliers. A far better outlier rejection method is clearly required to enable both accurate historical analysis and accurate forecasting in data communications networks.

[0015] The present invention provides a method that rejected in a successful prototype approximately 10¹⁵outliers for every 1 accepted (in Ethernet networks), while rejecting effectively no genuine points. The invention provides similar performance on ATM, Frame Relay and other protocol based data communications networks. The present invention therefore renders practical and effective the linear forecasting method mentioned above, surprisingly only requiring use of peak data. The method can be used as a filter for the measured points.

[0016] It is an important aspect of the present invention that it does not rely on knowledge of the distribution of possible values. It provides very reliable detection and rejection of outliers and so enables very significant improvements in the accuracy of both alarm detection and activity forecasting.

[0017] The present invention has application to all fields that involve the measurement of self similar activity and all fields in which measurable activity flows from one object to another. The set of fields with self similar distributions to which the present invention has application is enormous. Therefore the small fraction of those given as examples in this specification are only some of those in which the present invention has applicability. Further, the set of fields which include measurable flows is similarly vast. The embodiments described herein should only be taken as representative of those applications, and the present invention is applicable to all such fields.

[0018] In accordance with an embodiment of the present invention, a method of detecting outliers measured during progression of an activity of an entity from one point to another, comprises measuring activity at a point in a first dimension, measuring the same activity at the same point in a second dimension at the same time as measuring the activity in the first dimension, and rejecting outliers which have values outside a maximum expected difference between the activity measured in the first and second dimensions.

[0019] The invention requires that a particular activity should be measured at the same time using different devices in different dimensions. If the two measurements disagree by more than a maximum experimental difference expected between these devices, an outlier is declared to have been detected. The maximum acceptable experimental difference is now not related to the variance with time of the measured activity. Two examples will now be given: the first uses the dimension of distance, the other uses other dimensions.

[0020] (a) Consider the traffic to be flowing from point A to point B, wherein the traffic leaving point A is the same as that arriving at point B. Therefore measuring the flow rate both at point A and at point B is the same as measuring the flow rate twice at point A, simultaneously, after adjusting for the time of flight.

[0021] This general method requires prior knowledge that A is connected to B, which can be determined by a general method, such as that described in the U.S. Pat. No. 5,926,462 issued Jul. 20, 1999 entitled Method of Determining Topology of a Network of Objects”, invented by N. W. Dawes, D. Schenkel and M. Slavitch.

[0022] (b) Consider the traffic flow from point A. The flow rate should be measured in two dimensions at once at point A. Adjusting for the ratio or difference in dimensions, this too is equivalent to measuring the flow rate twice. For example, in data networks a pair of such dimensions is bytes/second and frames/second.

[0023] In a given medium type, the maximum and minimum ratios of bytes per frame are defined by standards. In Ethernet media there can only be between 64 and 1500 bytes per frame. Therefore if the ratio of the measurements of flow in bytes per second to frames per second falls outside the range 64 to 1500, an outlier has been detected.

[0024] A novel aspect of the embodiment in which synchronized measurements are made in different dimensions is that of the requirement of different dimensionality.

[0025] Embodiments of the invention will be described below. Reference is made to the following definitions, in which:

[0026] e_i: the experimental error in measuring a single value of the variable W at object i.

[0027] _Ei: the experimental error in measuring a single value of the variable V at object i.

[0028] f_ij: the expected ratio of variable V from object i to object j such that V_i=V_jf_ij(e.g. in a sealed constant temperature pipe and measuring a flow of water: f_ij=1).

[0029] i: an index which describes which object is referred to (e.g. i=x).

[0030] Max: the maximum possible value for V_x(e.g. from the line capacity).

[0031] Min: the minimum possible value for V_x(often zero)

[0032] N: the number of standard deviations by which two measures of V_idisagree.

[0033] R: the ratio of V_ito W_i(e.g. the measured ratio of bytes to frames at object x).

[0034] RatioMax: the maximum ratio of V_ito W_i(if V_i>0). (e.g. 1500 is the maximum ratio of bytes to frames in Ethernet media).

[0035] RatioMin: the minimum ratio of V_ito W_i(if V_i>0); (e.g. 64 is the minimum ratio of bytes to frames in Ethernet media).

[0036] T_ij: the time of flight from object i to object j.

[0037] V_i: the value of the variable as measured at object i (e.g. V_ais the value to be checked).

[0038] W_i: the value of a variable W related to V is measured at object i. (e.g.: 64 V_i<=W_i<=1500 V_i)

[0039] x: the object whose measurement of V is to be checked.

[0040] y: an object connected to the object x by a medium that carries the activity which is measured by V. (e.g. if the activity was water flow, the medium could be a pipe).

[0041] The variables V and W are measured as follows.

[0042] In data communications networks, SNMP (Simple Network Management Protocol) is generally available. Some other method of accessing information within a device could be used: e.g. Telnet, HTML, CMIP, XML, CIM. The traffic counters for frames and for bytes are generally stored in the standard portion of the MIB (Management Information Base) tables of each port for each managed object in such a network. By requesting the counters for frames and the counters for bytes at the same time, and again requesting these counters synchronously at a later time, the change in frames per second and the change in bytes per second during the interval can be determined for this object. W would then represent frames/second and V would represent bytes/second.

[0043] In other applications W and V would represent other paired variables. For example, in an fluid system W could be the total volume that passed through a portion of pipe in a period of time and V the total mass that passed, with the density of the fluid constrained to a given range.

[0044] The expected errors E_iand e_ican be determined in advance by measuring the appropriate variable (V or W) using multiple instances of the equipment at the same time at object i. Alternatively the errors can be theoretically determined by examining the tolerances of the measurement equipment.

[0045] In data communication networks the errors were found to have a common value of 0.5, so long as careful synchronization of measurements was adhered to. Therefore a standard value of 0.5 could be used for Ei and ei. It is believed that this common value is used perhaps due to poor software libraries in common use.

[0046] A method of exploiting a connection between objects follows.

[0047] Let x and y be connected.

[0048] Let V_ybe measured T_xylater in time than V_x.

[0049] The expected variance in: (V_x−V_yf_xy) is (E_x²+E_y²f_xy²).

[0050] The number of standard deviations difference between (V_x−V_yf_xy) is therefore:

[0051] N=|V_x−V_y|/(E_x²+E_y²f_xy²)^½  1

[0052] By choosing a maximum value of N (e.g. 10 standard deviations), V_xcan be declared an outlier if N is greater than this maximum. Expressed in different terms: should N be found to be greater than a selected threshold, the pair of values V_xand V_yare declared to be outliers. Both are rejected as the expression is symmetric with respect to both.

[0053] The false negative rate of this method can be estimated. The false negative rate is the probability a correct point will be wrongly identified as an outlier.

[0054] Except for the outliers, V_xis usually normally distributed. This is due to the fact that V_xis being measured several ways at the same time. Even if V_xhas a non-normal distribution with time, the distribution of these several synchronized measurements of a single value will usually be normally distributed with a mean corresponding to the true value. This is where the synchronization of the measurements transforms a problem in which the distribution is unknown to one in which the distribution will usually be normal.

[0055] Therefore N above is directly related to the probability that a correct point will be rejected. Most statistical textbooks give the normal curve areas which can be used to determine this probability (e.g. as described in the publication “Statistical tables and Formulas” (Table I) by A. Hald, (New York: John Wiley 1952). However, by choosing a very large value of N indeed, the false negative rate can be made effectively zero. In trying to determine peak levels of activity, avoiding false negatives which would reject true peaks is very important.

[0056] For example:

[0057] N=3 gives a false negative rate of 0.002, but N=10 gives a false negative rate that can be ignored.

[0058] The false positive rate is failure to identify a true outlier. Assuming that the outliers are distributed uniformly (which appears to be the case in data communications networks) and that the basic rejection of outliers with values<Min and values>Max has been accomplished, the relationships are as follows.

[0059] The outlier is chosen from a value in the range Min . . . Max.

[0060] The acceptable values for V_xlie in the range

[0061] V_y−N (E_x²+E_y²f_xy²)^½<=V_x<=V_y+N (E_x²+E_y²f_xy²)^½  2

[0062] The probability P that V_xwill lie in this range by chance is:

[0063] P=2 N (E_x²+E_y²f_y²)^½/(Max−Min)  3

[0064] Since P is inversely proportional to Max, it is preferred that the speed of the network should be as high as possible.

[0065] For example, choosing values reasonable in a data communication network:

[0066] N=10

[0067] E_x²+E_y²f_xy²=1

[0068] Max=200,000 (frames/second)

[0069] Min=0 (frames/second)

[0070] then:

[0071] P=0.0001 (a 10,000:1 rejection rate).

[0072] This method can be combined with the others described herein (and otherwise) as described later with respect to use of filters in combination, which leads to a combined example value of

[0073] P=10⁻¹⁵

[0074] To exploit measurements in 2 different dimensions:

[0075] Let V_xand W_xbe measured at the same time on the object x.

[0076] The ratio R (V_x/W_x) is expected to be in the range Ratiomin to Ratiomax.

[0077] The number of standard deviations by which they disagree is N where:

[0078] if R is <Ratiomin:

[0079] N=|V_x−W_xRatiomin|/(E_x²+e_x²Ratiomin²)^½  4

[0080] if Ratiomin<=R<=Ratiomax

[0081] N=0  5

[0082] if R is >Ratiomax:

[0083] N=|V_x−W_xRatiomax|/(E_x²+e_x²Ratiomax²)^½  6

[0084] By choosing a maximum value of N (e.g. 10 standard deviations), V_xcan be declared an outlier if N is greater than this maximum. Expressed in different terms: should N be found to be greater than a selected threshold, the pair of values V_xand W_xare declared to be outliers. Both are rejected as the method is symmetric with respect to both.

[0085] The false negative rate of this method can be estimated as described earlier.

[0086] The false positive rate of this method is estimated as follows. As noted earlier, the false positive rate is failure to identify a true outlier. Assuming the outliers are assumed to be distributed uniformly (which appears to be the case in data communications networks) and that the basic rejection of outliers with values >Max has been accomplished, the relationships are as follows.

[0087] The acceptable ratios for V_x:W_xlie in the range Ratiomin . . . Ratiomax (since the errors E can usually be ignored as they extend this range only very slightly).

[0088] Given a value of V_x, the set of possible values for W_xwill be chosen at random from the range Min to Max.

[0089] The probability P that V_xwill lie in this permitted range by chance is approximately:

[0090] P=W_x(Ratiomax−Ratiomin)/(Max−Min)  7

[0091] Since P is inversely proportional to Max, it is preferred that the speed of the network should be as high as possible.

[0092] For example, choosing values for an Ethernet data communication network, and checking the value of frames/second (V_x) from the value of bytes/second (W_x):

[0093] This method can be combined with the others described herein (and otherwise) and as described below with regard to use of filters in combination, which leads to a combined example value of

[0094] P=10⁻¹⁵

[0095] With regard to measurements in m (>2) different dimensions.

[0096] Let V_xand W_x,1..mbe measured at the same time on the object x.

[0097] Let the following relationship hold:

[0098] V_x=Σ_i=1..ma_iW_x,i

[0099] Let N be the number of standard deviations by which this relationship is violated in the set of synchronous measurements of V and W made on x where:

[0100] N=|V_x−Σ_i=1..ma_iW_x,i|/(E_x²+Σ_i=1..ma_ie_x,i²)^½  8

[0101] By choosing a maximum value of N (e.g. 10 standard deviations), V_xcan be declared an outlier if N is greater than this maximum. Expressed in different terms: should N be found to be greater than a selected threshold, the set of values V_xand W_x,iare declared to be outliers. The set is rejected as the method is symmetric with respect to all members of the set.

[0102] In Ethernet networks, the following relationship holds:

[0103] frames=broadcasts+unicasts−errors.

[0104] Therefore simultaneously measuring the frame rate, the broadcast rate, the unicast rate and the error rate allows their substitution into relationship 8 above.

[0105] The false negative rate of this embodiment is determined as described above.

[0106] As noted earlier, the false positive rate is failure to identify a true outlier. Assuming that the outliers are distributed uniformly (which appears to be the case in data communications networks) and that the basic rejection of outliers with values >Max has been accomplished, these relationships follow.

[0107] The outlier is chosen from a value in the range Min . . . Max.

[0108] Letting:

[0109] Q=Σ_i=1..ma_iW_x,i

[0110] The acceptable values for V_xlie in the range

[0111] Q−N (E_x²+Σ_i=1..ma_ie_x,i²)^½<=V_x<=Q+N (E_x²+Σ_i=1..ma_ie_x,i²)^½  9

[0112] The probability P that V_xwill lie in this range by chance is:

[0113] P=2N (E_x²+Σ_i=1..ma_ie_x,i²)^½/(Max−Min)  10

[0114] Since P is inversely proportional to Max, it is preferred that the network should have the highest possible speed.

[0115] For example, choosing values reasonable in a data communication network: $N=10$ ${\left(E_x^2+\sum _{i=1\dots m}{a}_i{e}_{x,i}^2\right)}^{1/2}=1$ $\begin{array}{cc}\mathrm{Max}=200,000& \left(\mathrm{frames}/\mathrm{second}\right)\\ \mathrm{Min}=0& \left(\mathrm{frames}/\mathrm{second}\right)\\ \mathrm{then}\text{:}& \\ P=0.0001& \left(a10,000\text{:}1\mathrm{rejection}\mathrm{rate}\right).\end{array}$

[0116] This embodiment can be combined with the others described herein (and otherwise) and as described below using filters in combination, which leads to a combined example value of

[0117] P=10⁻¹⁵

[0118] Filter methods can be used in various combinations, with the combined rates being the product of the individual rates. Methods can be combined so long as the combinations of variables does not repeat combinations that have already been applied in combination. For example, suppose the frame rate (F) and byte rate (B) are simultaneously measured at two ends of a line (x and y). The following three cases can be used in filters, using the methods described above for exploiting measurements in two different dimensions in the first case, and exploiting a connection between objects in the second and third cases:

[0119] F_xagainst B_x

[0120] F_xagainst F_y

[0121] B_xagainst B_y

[0122] But now F_ycannot be used to further filter B_ysince this is already implied by the combination above.

[0123] The following definitions are used in the description below:

[0124] r_i=false negative rate from method i.

[0125] R=false negative rate from methods 1..m combined.

[0126] S_i=false positive rate from method i.

[0127] S=false positive rate from methods 1..m combined. then:

[0128] R=Π_i=1..mr_i  11

[0129] S=Σ_i=1..ms_i  12

[0130] For example, should only the methods described above exploiting the measurements in two and in more dimensions be applied to an Ethernet interface (not using the topological method perhaps because the connection was just changed):

[0131] R=0.001×0.0001=10⁻⁷

[0132] If the topological method for exploiting a connection between objects were used as well:

[0133] R=0.001×0.0001×0.0001=10⁻¹¹

[0134] If coupled to the simple filter, that rejects points outside the capacity of the interface:

[0135] R=10⁻¹¹×10⁻⁴=10⁻¹⁵

[0136] Since in testing the outliers appeared to be scattered uniformly across the entire range of values permitted by the number of bits in the counter, the larger the ratio of the maximum value storable in the counter to the maximum physical value possible, the better these filter methods were found to work. For example, in SNMP, 32 bit counters are used which can represent numbers with a range of 0 to 4×10⁹. In SNMP v2 64 bit counters are used which provides a range of 0 to 1.6×10¹⁹. Operating on the same problem conditions, the larger counters would therefore provide far better filtering performance than the smaller ones.

[0137] While the description above is to a general method for discovering outliers, it should be noted that the objects which have measurements made upon them could be physically discrete (e.g. communications devices), conceptually discrete (e.g. arbitrarily chosen volumes in a solid) or merely points of reference (e.g. locations in a pipe).

[0138] The following are some examples of what can be measured. If the topologies are known, this also enables the topological dependent filter described above to be used. Dawes et al, in the patent referred to earlier, describe a general method for topological determination which covers these and other examples. The determination of activity can then be used for model discovery, for diagnosis, description or for other purposes.

[0139] Embodiments of the invention can be applied to measurement of:

[0140] a: Electrical activity in neurons or neuronal regions of the brain.

[0141] b: Electrical signals and information transfers in communications systems: data, voice and mixed forms in static, mobile, satellite and hybrid networks.

[0142] c: Volume flow of fluids: for plumbing, heating, cooling, nuclear reactors, oil refineries, chemical plants, sewage networks, weather forecasting, flows in and from aquifers, blood circulation (especially in the heart); other biological fluids, sub intra and supra tectonic flows of lava, semisolids and solids.

[0143] d: Flow of information or rates of use in software systems and mixed software/hardware systems.

[0144] e: Object flows: fish, bird and animal migration paths, tracks and routes of vehicles.

[0145] f: Heat flow; partitioning a surface or volume up into elements, one can describe the flow vectors of heat through the elements and hence deduce a probabilistic flow network. The measured attribute could be direct (e.g. black body emission signature) or indirect (e.g. electrical resistance).

[0146] g: Nutrient and nutrient waste flow; certain nutrients get consumed more rapidly by rapidly growing parts (e.g. cancers) than by other parts. The flow of nutrients will tend to be abnormal towards such abnormal growths and similarly the flow of waste will be abnormally large away from them.

[0147] h: Improved alarm condition detection and forecasting in data communications.

[0148] i: Measurement of the performance of economic and system operational models, leading to discovery of ways to change, influence, direct or improve them. For example, stock market distributions appear to be self similar as do other economic related distributions.

[0149] The above list indicates a large number of potential applications, although additional other applications can be determined by a person understanding the invention. The following general list indicates fields in which the invention can be used, although additional fields can be determined by person understanding the invention:

[0150] biological diagnosis, model discovery and validation;

[0151] volcanic eruption and earthquake prediction;

[0152] refinery operations startup modeling for replication;

[0153] operational efficiency improvements by spotting bottlenecks and possibilities for shortcuts (in organizations and systems).

[0154] A person understanding the above-described invention may now conceive of alternative designs, using the principles described herein. All such designs which fall within the scope of the claims appended hereto are considered to be part of the present invention.