SYSTEM AND METHOD FOR NETWORK ROOT CAUSE ANALYSIS

[0001] The instant application is a Continuation of, and claims priority to, U.S. patent application Ser. No. 15/796,687 entitled SYSTEM AND METHOD FOR NETWORK ROOT CAUSE ANALYSIS filed Oct. 27, 2017, the contents of which are herein incorporated by reference in its entirety.

[0002] The present technology pertains to a system and method of a multi-step analysis for determining a root cause of a performance problem and distinguishing between a root cause of a particular performance problem and factors correlated to the particular problem.

[0003] Existing analytics platforms such as The Cisco Tetration Analytics platform developed by Cisco Technology, Inc. of San Jose, Calif., capture and analyze real-time network traffic and application/process performance data from each endpoint and network device (including sensors, network components such as servers, etc.) in a network to assess network performance, security, policy compliance, etc. By correlating flow data and application/process data, additional insights can be obtained on application dependencies, performance, etc. When an issue occurs, existing analytics platforms may capture all the network traffic and application/process performance data from all over the network that occurs at a similar time. However, it is difficult to determine from all such collected network traffic and application/process data, the root cause of the network issue.

[0010] Various examples of the disclosure are discussed in detail below. While specific implementations are discussed, it should be understood that this is done for illustration purposes only. A person skilled in the relevant art will recognize that other components and configurations may be used without parting from the spirit and scope of the disclosure.

[0011] References to one or an example embodiment in the present disclosure can be, but not necessarily are, references to the same example embodiment; and, such references mean at least one of the example embodiments.

[0012] Reference to “one example embodiment” or “an example embodiment” means that a particular feature, structure, or characteristic described in connection with the example embodiment is included in at least one example of the disclosure. The appearances of the phrase “in one example embodiment” in various places in the specification are not necessarily all referring to the same example embodiment, nor are separate or alternative example embodiments mutually exclusive of other example embodiments. Moreover, various features are described which may be exhibited by some example embodiments and not by others. Similarly, various features are described which may be features for some example embodiments but not other example embodiments.

[0013] The terms used in this specification generally have their ordinary meanings in the art, within the context of the disclosure, and in the specific context where each term is used. Alternative language and synonyms may be used for any one or more of the terms discussed herein, and no special significance should be placed upon whether or not a term is elaborated or discussed herein. Synonyms for certain terms are provided. A recital of one or more synonyms does not exclude the use of other synonyms. The use of examples anywhere in this specification including examples of any terms discussed herein is illustrative only, and is not intended to further limit the scope and meaning of the disclosure or of any exemplified term. Likewise, the disclosure is not limited to various examples given in this specification.

[0014] Without intent to limit the scope of the disclosure, examples of instruments, apparatus, methods and their related results according to examples of the present disclosure are given below. Note that titles or subtitles may be used in the examples for convenience of a reader, which in no way should limit the scope of the disclosure. Unless otherwise defined, technical and scientific terms used herein have the meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. In the case of conflict, the present document, including definitions will control.

[0015] Although the terms first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element could be termed a second element, and similarly, a second element could be termed a first element, without departing from the scope of this disclosure. As used herein, the term “and/or,” includes any and all combinations of one or more of the associated listed items.

[0016] When an element is referred to as being “connected,” or “coupled,” to another element, it can be directly connected or coupled to the other element or intervening elements may be present. By contrast, when an element is referred to as being “directly connected,” or “directly coupled,” to another element, there are no intervening elements present. Other words used to describe the relationship between elements should be interpreted in a like fashion (e.g., “between,” versus “directly between,” “adjacent,” versus “directly adjacent.” etc.).

[0017] The terminology used herein is for the purpose of describing particular examples only and is not intended to be limiting. As used herein, the singular forms “a”, “an”, and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises”, “comprising,”, “includes” and/or “including”, when used herein, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

[0018] It should also be noted that in some alternative implementations, the functions/acts noted may occur out of the order noted in the figures. For example, two figures shown in succession may in fact be executed substantially concurrently or may sometimes be executed in the reverse order, depending upon the functionality/acts involved.

[0019] Specific details are provided in the following description to provide a thorough understanding of examples. However, it will be understood by one of ordinary skill in the art that examples may be practiced without these specific details. For example, systems may be shown in block diagrams so as not to obscure the examples in unnecessary detail. In other instances, well-known processes, structures and techniques may be shown without unnecessary detail in order to avoid obscuring examples.

[0020] In the following description, illustrative examples will be described with reference to acts and symbolic representations of operations (e.g., in the form of flow charts, flow diagrams, data flow diagrams, structure diagrams, block diagrams, etc.) that may be implemented as program services or functional processes include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types and may be implemented using hardware at network elements. Non-limiting examples of such hardware may include one or more Central Processing Units (CPUs), digital signal processors (DSPs), application-specific-integrated-circuits, field programmable gate arrays (FPGAs), computers or the like.

[0021] Additional features and advantages of the disclosure will be set forth in the description which follows, and in part will be obvious from the description, or can be learned by practice of the herein disclosed principles. The features and advantages of the disclosure can be realized and obtained by means of the instruments and combinations particularly pointed out in the appended claims. These and other features of the disclosure will become more fully apparent from the following description and appended claims, or can be learned by the practice of the principles set forth herein.

[0022] In one aspect of the present disclosure, a method includes receiving an inquiry to determine a root cause of a network problem and performing a multi-level process to determine the root cause, wherein a first level of the multi-level process includes, collecting data from one or more network components, generating a set of system metrics, each system metric of the set representing a portion of the data, at least one system metric of the set being a target metric corresponding to the network problem, ranking the set of system metrics based on a level of correlation of each system metric to the network problem to yield a ranked set of system metrics, and providing a visual representation of the network problem and each of the ranked set of system metrics. A second level of the multi-level process includes receiving an input identifying one or more of the ranked set of system metrics to be excluded from analysis and performing a conditional analysis for determining the root cause of the network problem using only ones of the set of system metrics that are not identified for exclusion.

[0023] In one aspect of the present disclosure, a system includes a memory having computer-readable instructions stored therein, and one or more processors. The one or more processors are configured to execute the computer-readable instructions to perform functions of receiving an inquiry to determine a root cause of a network problem and performing a multi-level process to determine the root cause, wherein a first level of the multi-level process includes collecting data from one or more network components, generating a set of system metrics, each system metric of the set representing a portion of the data, at least one system metric of the set being a target metric corresponding to the network problem, ranking the set of system metrics based on a level of correlation of each system metric to the network problem to yield a ranked set of system metrics and providing a visual representation of the network problem and each of the ranked set of system metrics. A second level of the multi-level process includes receiving an input identifying one or more of the ranked set of system metrics to be excluded from analysis and performing a conditional analysis for determining the root cause of the network problem using only ones of the set of system metrics that are not identified for exclusion.

[0024] In one aspect of the present disclosure, a non-transitory computer-readable medium has computer-readable instructions, which when executed by one or more processors, cause the one or more processors to perform the functions of functions of receiving an inquiry to determine a root cause of a network problem, perform a multi-level process in order to present a ranked list of a set of system metrics that are correlated to the network problem, the ranked list being presented in accordance with a level of corresponding relevance to the network problem, receive an input identifying one or more system metrics of the ranked list of system metrics to be excluded from further analysis of the root cause of the network problem, perform a conditional analysis of the network problem using only ones of the ranked list of the set of system metrics not identified for exclusion in the input and present a result of the conditional analysis.

[0025] The disclosed technology addresses the need in the art for distinguishing factor(s) (system metric(s)) corresponding to a root cause of a particular network performance problem from all other factors (system metrics) that are simply correlated to the particular network performance problem. Hereinafter, a network performance problem may also be referred to as a network problem, a performance problem or simply a problem.

[0026] For example, a typical webserver application running on a cloud provided by a particular cloud service problem, can serve user requests. A corresponding network operator can monitor various network and server metrics including, but not limited to, a network bandwidth, TCP retransmissions, CPU usage of various servers used for the webserver application, memory usage by one or more processes of the network, etc. Typically such metrics can be grouped in feature families (e.g., an overall or a broader system metric) and each family can have hundreds of underlying metrics.

[0027] In this example, an assumption is made that the business goal of the application is monitored by measuring average user request latency. When the latency (an example of a network problem) goes up, the network operator would want to know what causes the increase in latency, and possibly how it can be mitigated/addressed.

[0028] Sometimes a metric or metrics is/are just correlated to the underlying problem but fail(s) to provide any insight about the root cause of the network problem, which in the above example is an increase in user request latency. Systems and methods are needed to distinguish between correlated factors (metrics) and root cause factors (metrics) of the network problem. For example, in the webserver application example above, as the user traffic increases, system memory usage also increases. This increase in memory usage is correlated to the user request latency, while the real cause is user traffic. One or more examples of the present disclosure, as described hereinafter, enable a relatively simple and more effective determination of system metrics to correlated to a network problem and taking feedback from network operators that simplify the process of determining causation of a network problem as opposed to simply the correlations between the network problem and other network performance metrics.

[0029] The disclosure begins with a description of example network environments and architectures which can be implemented for distributed streaming systems, as illustrated in FIGS. 1A-D.

[0030] The disclosure begins with a description of example network environments and architectures, as illustrated in FIGS. 1A-D.

[0031] FIG. 1A illustrates an example system, according to one aspect of the present disclosure. System (network) 100 of FIG. 1A includes a controller 102 and a distributed streaming system 120. Controller 102 can be an application, a software container, a virtual machine, a service chain, a virtual function(s), etc. Controller 102 can run on one or more devices or servers having components such as one or more processors (e.g., processor 104), one or more memories (e.g., memory 106), a transceiver 108, a display device 110 and an input device 112. Processor 104 can be configured to execute computer-readable instructions stored on memory 106 for performing the functionalities which will be described below with reference to FIGS. 4-5. Throughout the disclosure, controller 102 can be referred to as system management component 102, management device 102, device 102 and/or system controller 102.

[0032] Transceiver 108 can be any known or to be developed receiver and transmitter through which controller 102 can send and receive information to and from external components such as components of distributed streaming system 120.

[0033] Network operators and controllers (operational management component) can use display 110 to view data corresponding to status and/or management of operation of distributed streaming system 120, as will be described below. Display 110 can be any type of know or to be developed display such as a liquid crystal display (LCD), a light emitting diode display (LED), etc.

[0034] Input device 112 can be any known or to be developed input device including, but not limited to, a keyboard, a touch-based input device, etc. In one example, display 110 and input device 112 can be the same when display 110 is a touch enabled device capable of receiving inputs.

[0035] Network managers and operators can provide appropriate commands for monitoring and management of distributed streaming system 120, via input device 112.

[0036] Controller 102 can communicate with various components of distributed streaming system 120 via any known or to be developed wireless communications and/or wired communications mean. For example, controller 102 can access and obtain information (and/or send information) to each component of distributed system 120 via a network such as a local area wireless network (LAN), a virtual local area network (vLAN) and/or any other type of, known or to be developed, network through which controller 102 can communicate with each component of distributed streaming system 120.

[0037] In one aspect, controller 102 can be any known or to be developed electronic device including, but not limited to, a laptop, a desktop computer, a mobile device, a handheld device, etc.

[0038] Distributed streaming system 120 can be any known, or to be developed, distributed streaming system where various components thereof such as components 122-1, 122-2, 122-3 and 122-4 communicate with one another to provide a streaming service to users in a distributed fashion. Hereinafter, components 122-1, 122-2, 122-3 and 122-4 may simply be referred to as components 122 or nodes 122. While throughout the present disclosure, distributed streaming system is provided as an example, the present disclosure is not limited thereto and can encompass and be applicable to any distributed systems that can be abstracted into a Directed Acyclic Graph (DAG) where each vertex can denote an information/message, and information/messages are passed through edges in certain directions. Other examples of distributed systems include a distributed sensor network where signals are propagated from sensor to sensor, a multi-component data processing system where each component receives and processes chunks of data and pass it to the next component(s).

[0039] Each one of components 122 can be any know or to be developed electronic device capable of communicating remotely with other devices such as other components 122. For example, each component 122 can be a mobile device, a laptop, a desktop computer, a switch, a data center comprising one or more servers, etc. For example, while some of components 122 can be end user devices or hosts, other ones of components 122 can be servers that facilitate the streaming services provided by distributed streaming system 120.

[0040] Furthermore, distributed streaming system 120 can have a server 114 acting as a collector of information (data) for other components (end user devices) in the system. Examples of data include device metrics such as device ID, an associated timestamp, device IP address, device throughput, device latency, memory and processing speed characteristics, etc.

[0041] In one example, system 100 further includes one or more feedback servers 116, where various types of data (to be used by controller 102) on components 122 can be collected and saved. In another example, system 100 does not include any feedback servers and instead can directly receive (through push or pull operations) the intended data (which will be described below) from each component 122.

[0042] Furthermore. FIG. 1A illustrates one or more sensors 124 associated with each network component 122 and feedback server 116. As will be described below, sensors 124 can be used by controller 102 for collecting various types of data such as component status, network performance, network traffic and application data for respective one of components 122 or feedback server 116.

[0043] While certain components are illustrated as part of system 100, system 100 is not limited thereto and may include any other type of component (e.g., additional servers, access points, sensors, etc.) for providing services to clients and end users.

[0044] Distributed streaming system 120 can be a cloud based system, where each component thereof is located in a different geographical location but can communicate with one another to form distributed streaming system 120 (e.g., over the Internet).

[0045] Examples of streaming services provided via distributed streaming system 120 can include, but is not limited to, live video and/or audio content such as a speech, a concert, a TV program, music, etc.

[0046] Operations of distributed streaming system 120 for delivering a streaming service to end users can be based on any know or to be developed method for doing so, by for example, continuously processing a stream of text, graphs, videos, audios, time series data, etc in real time or near real time or periodically. The system 100 of FIG. 1A utilizes client/server based architectures. In other examples, system 100 can be implemented as a cloud or fog computing architecture.

[0047] FIG. 1B illustrates a diagram of an example cloud computing architecture (network) 130. The architecture can include a cloud 132. The cloud 132 can include one or more private clouds, public clouds, and/or hybrid clouds. Moreover, the cloud 132 can include cloud elements 134-144. The cloud elements 134-144 can include, for example, servers 134, virtual machines (VMs) 136, one or more software platforms 138, applications or services 140, software containers 142, and infrastructure nodes 144. The infrastructure nodes 144 can include various types of nodes, such as compute nodes, storage nodes, network nodes, management systems, etc. In one example, one or more servers 134 can implement the functionalities of controller 102, which will be described below. Alternatively, controller 102 can be a separate component that communicates with components of the cloud computing architecture 130 that function as a distributed streaming system similar to the distributed streamlining system 120.

[0048] The cloud 132 can provide various cloud computing services via the cloud elements 134-144, such as software as a service (SaaS) (e.g., collaboration services, email services, enterprise resource planning services, content services, communication services, etc.), infrastructure as a service (IaaS) (e.g., security services, networking services, systems management services, etc.), platform as a service (PaaS) (e.g., web services, streaming services, application development services, etc.), function as a service (FaaS), and other types of services such as desktop as a service (DaaS), information technology management as a service (ITaaS), managed software as a service (MSaaS), mobile backend as a service (MBaaS), etc.

[0049] The client endpoints 146 can connect with the cloud 132 to obtain one or more specific services from the cloud 132. The client endpoints 146 can communicate with elements 134-144 via one or more public networks (e.g., Internet), private networks, and/or hybrid networks (e.g., virtual private network). The client endpoints 146 can include any device with networking capabilities, such as a laptop computer, a tablet computer, a server, a desktop computer, a smartphone, a network device (e.g., an access point, a router, a switch, etc.), a smart television, a smart car, a sensor, a GPS device, a game system, a smart wearable object (e.g., smartwatch, etc.), a consumer object (e.g., Internet refrigerator, smart lighting system, etc.), a city or transportation system (e.g., traffic control, toll collection system, etc.), an internet of things (IoT) device, a camera, a network printer, a transportation system (e.g., airplane, train, motorcycle, boat, etc.), or any smart or connected object (e.g., smart home, smart building, smart retail, smart glasses, etc.), and so forth.

[0050] FIG. 1C illustrates a diagram of an example fog computing architecture (network) 150. The fog computing architecture 150 can include the cloud layer 154, which includes the cloud 132 and any other cloud system or environment, and the fog layer 156, which includes fog nodes 162. The client endpoints 146 can communicate with the cloud layer 154 and/or the fog layer 156. The architecture 150 can include one or more communication links 152 between the cloud layer 154, the fog layer 156, and the client endpoints 146. Communications can flow up to the cloud layer 154 and/or down to the client endpoints 146.

[0051] In one example, one or more servers 134 can implement the functionalities of controller 102, which will be described below. Alternatively, controller 102 can be a separate component that communicates with components of the fog computing architecture 150 that function as a distributed streaming system similar to the distributed streamlining system 120

[0052] The fog layer 156 or “the fog” provides the computation, storage and networking capabilities of traditional cloud networks, but closer to the endpoints. The fog can thus extend the cloud 132 to be closer to the client endpoints 146. The fog nodes 162 can be the physical implementation of fog networks. Moreover, the fog nodes 162 can provide local or regional services and/or connectivity to the client endpoints 146. As a result, traffic and/or data can be offloaded from the cloud 132 to the fog layer 156 (e.g., via fog nodes 162). The fog layer 156 can thus provide faster services and/or connectivity to the client endpoints 146, with lower latency, as well as other advantages such as security benefits from keeping the data inside the local or regional network(s).

[0053] The fog nodes 162 can include any networked computing devices, such as servers, switches, routers, controllers, cameras, access points, kiosks, gateways, etc. Moreover, the fog nodes 162 can be deployed anywhere with a network connection, such as a factory floor, a power pole, alongside a railway track, in a vehicle, on an oil rig, in an airport, on an aircraft, in a shopping center, in a hospital, in a park, in a parking garage, in a library, etc.

[0054] In some configurations, one or more fog nodes 162 can be deployed within fog instances 158, 160. The fog instances 158, 158 can be local or regional clouds or networks. For example, the fog instances 156, 158 can be a regional cloud or data center, a local area network, a network of fog nodes 162, etc. In some configurations, one or more fog nodes 162 can be deployed within a network, or as standalone or individual nodes, for example. Moreover, one or more of the fog nodes 162 can be interconnected with each other via links 164 in various topologies, including star, ring, mesh or hierarchical arrangements, for example.

[0055] In some cases, one or more fog nodes 162 can be mobile fog nodes. The mobile fog nodes can move to different geographic locations, logical locations or networks, and/or fog instances while maintaining connectivity with the cloud layer 154 and/or the endpoints 146. For example, a particular fog node can be placed in a vehicle, such as an aircraft or train, which can travel from one geographic location and/or logical location to a different geographic location and/or logical location. In this example, the particular fog node may connect to a particular physical and/or logical connection point with the cloud 154 while located at the starting location and switch to a different physical and/or logical connection point with the cloud 154 while located at the destination location. The particular fog node can thus move within particular clouds and/or fog instances and, therefore, serve endpoints from different locations at different times.

[0056] FIG. 1D illustrates a schematic block diagram of an example network architecture (network) 180. In some cases, the architecture 180 can include a data center, which can support and/or host the cloud 132. Moreover, the architecture 180 includes a network fabric 182 with spines 184A, 184B, . . . , 184N (collectively “184”) connected to leafs 186A, 186B, 186C, . . . , 186N (collectively “186”) in the network fabric 182. Spines 184 and leafs 186 can be Layer 2 and/or Layer 3 devices, such as switches or routers. For the sake of clarity, they will be referenced herein as spine switches 184 and leaf switches 186.

[0057] Spine switches 184 connect to leaf switches 186 in the fabric 182. Leaf switches 186 can include access ports (or non-fabric ports) and fabric ports. Fabric ports can provide uplinks to the spine switches 182, while access ports can provide connectivity for devices, hosts, endpoints, VMs, or external networks to the fabric 182.

[0058] Leaf switches 186 can reside at the boundary between the fabric 182 and the tenant or customer space. The leaf switches 186 can route and/or bridge the tenant packets and apply network policies. In some cases, a leaf switch can perform one or more additional functions, such as implementing a mapping cache, sending packets to the proxy function when there is a miss in the cache, encapsulate packets, enforce ingress or egress policies, etc.

[0059] Moreover, the leaf switches 186 can contain virtual switching and/or tunneling functionalities, such as a virtual tunnel endpoint (VTEP) function. Thus, leaf switches 186 can connect the fabric 182 to an overlay (e.g., VXLAN network).

[0060] Network connectivity in the fabric 182 can flow through the leaf switches 186. The leaf switches 186 can provide servers, resources, endpoints, external networks, containers, or VMs access to the fabric 182, and can connect the leaf switches 186 to each other. The leaf switches 186 can connect applications and/or endpoint groups (“EPGs”) to other resources inside or outside of the fabric 182 as well as any external networks.

[0061] Endpoints 192A-D (collectively “192”) can connect to the fabric 182 via leaf switches 186. For example, endpoints 192A and 192B can connect directly to leaf switch 186A, which can connect endpoints 192A and 192B to the fabric 182 and/or any other of the leaf switches 186. Similarly, controller 102 (which can be the same as controller 102 described above with reference to FIG. 1A) can connect directly to leaf switch 186C, which can connect controller 102 to the fabric 182 and/or any other of the leaf switches 186. On the other hand, endpoints 192C and 192D can connect to leaf switch 186A and 186B via network 188. Moreover, the wide area network (WAN) 190 can connect to the leaf switches 186N.

[0062] Endpoints 192 can include any communication device or resource, such as a computer, a server, a cluster, a switch, a container, a VM, a virtual application, etc. In some cases, the endpoints 192 can include a server or switch configured with a virtual tunnel endpoint functionality which connects an overlay network with the fabric 182. For example, in some cases, the endpoints 192 can represent hosts (e.g., servers) with virtual tunnel endpoint capabilities, and running virtual environments (e.g., hypervisor, virtual machine(s), containers, etc.). An overlay network associated with the endpoints 192 can host physical devices, such as servers; applications; EPGs; virtual segments; virtual workloads; etc. Likewise, endpoints 192 can also host virtual workloads and applications, which can connect with the fabric 182 or any other device or network, including an external network.

[0063] The disclosure now turns to FIGS. 2 and 3, which illustrate example network devices and computing devices, such as switches, routers, load balancers, client devices, and so forth.

[0064] FIG. 2 illustrates an example network device suitable for performing switching, routing, load balancing, and other networking operations, according to an aspect of the present disclosure. In one example, network device 200 can be controller 102 and/or any one of components 122 of FIG. 1A. Network device 200 includes a central processing unit (CPU) 204, interfaces 202, and a bus 210 (e.g., a PCI bus). When acting under the control of appropriate software or firmware, CPU 204 is responsible for executing packet management, error detection, and/or routing functions. CPU 204 preferably accomplishes all these functions under the control of software including an operating system and any appropriate applications software. CPU 204 may include one or more processors 208, such as a processor from the INTEL X86 family of microprocessors. In some cases, processor 208 can be specially designed hardware for controlling the operations of network device 200. In some cases, a memory 206 (e.g., non-volatile RAM, ROM, etc.) also forms part of CPU 204. However, there are many different ways in which memory could be coupled to the system.

[0065] Interfaces 202 are typically provided as modular interface cards (sometimes referred to as “line cards”). Generally, they control the sending and receiving of data packets over the network and sometimes support other peripherals used with network device 200. Among the interfaces that may be provided are Ethernet interfaces, frame relay interfaces, cable interfaces, DSL interfaces, token ring interfaces, and the like. In addition, various very high-speed interfaces may be provided such as fast token ring interfaces, wireless interfaces. Ethernet interfaces, Gigabit Ethernet interfaces, ATM interfaces, HSSI interfaces, POS interfaces, FDDI interfaces. WIFI interfaces, 3G/4G/5G cellular interfaces. CAN BUS, LoRA, and the like. Generally, these interfaces may include ports appropriate for communication with the appropriate media. In some cases, they may also include an independent processor and, in some instances, volatile RAM. The independent processors may control such communications intensive tasks as packet switching, media control, signal processing, crypto processing, and management. By providing separate processors for the communications intensive tasks, these interfaces allow the master microprocessor 204 to efficiently perform routing computations, network diagnostics, security functions, etc.

[0066] Although the system shown in FIG. 2 is one specific network device of the present invention, it is by no means the only network device architecture on which the present invention can be implemented. For example, an architecture having a single processor that handles communications as well as routing computations, etc., is often used. Further, other types of interfaces and media could also be used with network device 200.

[0067] Regardless of the network device's configuration, it may employ one or more memories or memory modules (including memory 206) configured to store program instructions for the general-purpose network operations and mechanisms for roaming, route optimization and routing functions described herein. The program instructions may control the operation of an operating system and/or one or more applications, for example. The memory or memories may also be configured to store tables such as mobility binding, registration, and association tables, etc. Memory 206 could also hold various software containers and virtualized execution environments and data.

[0068] Network device 200 can also include an application-specific integrated circuit (ASIC), which can be configured to perform routing and/or switching operations. The ASIC can communicate with other components in network device 200 via bus 210, to exchange data and signals and coordinate various types of operations by network device 200, such as routing, switching, and/or data storage operations, for example.

[0069] FIG. 3 illustrates a computing system architecture, according to an aspect of the present disclosure. As shown in FIG. 3, components of system 300 are in electrical communication with each other using a connection 305, such as a bus. Exemplary system 300 includes a processing unit (CPU or processor) 310 and a system connection 305 that couples various system components including system memory 315, such as read only memory (ROM) 320 and random access memory (RAM) 325, to processor 710. System 300 can include a cache of high-speed memory connected directly with, in close proximity to, or integrated as part of the processor 310. System 300 can copy data from memory 315 and/or storage device 330 to cache 312 for quick access by processor 310. In this way, the cache can provide a performance boost that avoids processor 310 delays while waiting for data. These and other modules can control or be configured to control the processor 310 to perform various actions. Other system memory 315 may be available for use as well.

[0070] Memory 315 can include multiple different types of memory with different performance characteristics. Processor 310 can include any general purpose processor and a hardware or software service, such as Service 1 332, Service 2 334, and Service 3 336 stored in storage device 330, configured to control processor 310 as well as a special-purpose processor where software instructions are incorporated into the actual processor design. Processor 310 may be a completely self-contained computing system, containing multiple cores or processors, a bus, memory controller, cache, etc. A multi-core processor may be symmetric or asymmetric.

[0071] To enable user interaction with the computing device 300, an input device 345 can represent any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, keyboard, mouse, motion input, speech and so forth. An output device 335 can also be one or more of a number of output mechanisms known to those of skill in the art. In some instances, multimodal systems can enable a user to provide multiple types of input to communicate with computing device 300. The communications interface 340 can generally govern and manage the user input and system output. There is no restriction on operating on any particular hardware arrangement and therefore the basic features here may easily be substituted for improved hardware or firmware arrangements as they are developed.

[0072] Storage device 330 is a non-volatile memory and can be a hard disk or other types of computer readable media which can store data that are accessible by a computer, such as magnetic cassettes, flash memory cards, solid state memory devices, digital versatile disks, cartridges, random access memories (RAMs) 325, read only memory (ROM) 320, and hybrids thereof.

[0073] The storage device 330 can include services 332, 334, 336 for controlling the processor 310. Other hardware or software modules are contemplated. The storage device 330 can be connected to the system connection 305. In one aspect, a hardware module that performs a particular function can include the software component stored in a computer-readable medium in connection with the necessary hardware components, such as the processor 310, connection 305, output device 335, and so forth, to carry out the function.

[0074] Hereinafter, a process for determining a root cause of a network problem is described with reference to FIGS. 4 and 5.

[0075] FIG. 4 illustrates a method of determining a root cause of a network problem, according to one aspect of the present disclosure. FIG. 4 will be described from the perspective of controller (system management component) 102 of FIG. 1A. However, it will be appreciated that one or more processors of controller 102 such as processor 104 execute computer-readable instructions stored on one or more memories such as memory 106 to transform the one or more processors into special purpose processor(s) for carrying out the functionalities and the process of FIG. 4.

[0076] At S400, controller 102 receives an inquiry. In one example, controller 102 receives the inquiry from a network operator. The inquiry may identify one or more network problems the root cause of which is to be determined by controller 102. For example, an inquiry received at S400 can be a request to determine a root cause of user request latency mentioned above.

[0077] At S405, controller 102 collects various types of data related to network 100, cloud architecture 130, 150, etc. As indicated above, controller 102 can collect the data using, for example, sensors 124 shown in FIG. 1. As also indicated above, the collected data can be any type of network data including, but not limited to, network traffic data (e.g., bytes sent, packets sent, TCP ACKs, etc.), network component performance data (e.g., CPU load or disk usage by one or more servers operating in a production mode such as servers 134, VMs 136, etc. of cloud 132, CPU load or disk usage by one or more servers in a development/testing mode such as servers 134, VMs 136, etc of cloud 132), application performance data or application dependencies of applications/processes such as that of applications 140 running on cloud 132, etc.

[0078] At S410, controller 102 generates one or more system metrics into each of which a portion of the data collected at S405 are organized. For example, bytes sent are organized into one metric, TCP ACKs are organized into another metric, CPU usage is organized into another metric, etc. In one example, each metric can have a tag (identifier) associated therewith. In another example, each metric is a time-series metric. Each element can also have a tag in the form of (key, value), where key indicates what the attribute is (e.g., TCP ACKs, CPU usage, etc.) and the value indicates the attribute's value at a particular time.

[0079] The one or more system metrics generated at S410 can be referred to as individual system metrics

[0080] In a real-world example, a network such as network 100 or 130 can have hundreds or thousands of different components and applications/processes running thereon. Therefore, a number of metrics into which collected data are organized can be in the thousands. Therefore, analyzing and understanding from all of these metrics, what a particular network problem may be is a resource intensive and time consuming process, if not impossible.

[0081] In one example, one or more of the individual system metrics created at S410 into which collected data are organized is referred to as a target metric(s). A target metric can represent data related to the target network problem, the root cause of which is to be identified. In examples described above, a target metric can be a time series of data collected on user request latency by controller 102.

[0082] At S415, controller 102 summarizes the generated individual system metrics, based on one or more common characteristics therebetween, into a set of metrics (i.e., generates a set of system metrics). This may be referred to as summarizing two or more metrics in an umbrella metric. As mentioned above, in a real-world example, a number of metrics into which collected data are organized can be in the thousands, the analysis of which is difficult and resource intensive, if not impossible. Therefore, by summarizing the generated metrics based on common characteristics a few to a hundred of different metrics can be combined into a single umbrella metric by controller 102 at S415.

[0083] For example, a common characteristic between metrics on bytes sent, packets sent and, TCP ACKs is network usage. Therefore, all three metrics on bytes sent, packets sent and TCP ACKs are summarized into a single network usage metric at S415.

[0084] In one example and in order to summarize two or more of the individual system metrics into an umbrella metric, controller 102 can use dimensionality reduction techniques including, but not limited to, the principle component analysis. Furthermore, controller 102 can use other techniques such as factor analysis in order to detect and collect latent/hidden features and deep learning auto encoders.

[0085] Use of techniques such as factor analysis to detect latent features and anomalies is useful because solely relying on summary features can lead to losing important data. For example, if packet counts and byte counts are summarized into a single network usage metrics, we risk losing unusual network data such as, for example, a more than typical number of small packets transmitted during certain time-period (which by itself can cause problems or can contribute to the network problem that controller 102 is to find the root cause of).

[0086] Accordingly, implementing factor analysis can result in tracking and saving such anomalies.

[0087] In one or more examples, two or more of the individual system metrics that are summarized into a single umbrella metric according to one or more corresponding common characteristics can have weights added thereto. In other words, the single umbrella can include a weighted combination of the metrics summarized into. In one example, the weight can be determined by controller 102, depending on the relevancy and/or importance of the corresponding metric. For example, number of packets sent may be less important to the user request latency that number of bytes sent. Accordingly, a higher weight can be associated with the number of bytes metric than the number of packets metric.

[0088] At S420, controller 102 tags each of the set of system metrics (summarized metrics at S415). Summarizing at S415 can sometimes results in losing labels (identifiers) of underlying metrics that are summarized into one umbrella metric, thus making it more difficult controller 102 and/or network operations of network 100/130 to understand what a particular umbrella metric conveys. For example, an umbrella metric that is a summarized version of metrics representing packets sent, bytes sent and TCP ACKs can lose their identifiers and thus be unnamed.

[0089] Accordingly, at S420, controller 102 can automatically generate a tag for the umbrella metric based on a common description associated with identifiers of the metrics representing bytes sent, packets sent and TCP ACKs (e.g., generate a “Network Usage” tag based on “Bytes”, “Packets” and “TCP ACKs” identifiers of metrics representing packets sent, bytes sent and TCP ACKs, respectively).

[0090] In another example, metrics generated at S410 can correspond to application/process data of an application running on several servers in network 100 or network 130. Each process on one of the servers can have a corresponding process ID. Accordingly and after summarizing individual server data for the application running thereon, the summarized metric can be tagged with the process ID at S420.

[0091] In one example, instead of generating the tag automatically, controller 102 prompts a network operator to provide an appropriate tag for a particular umbrella metric. In one example, controller 102 can request the network operator for the tag it, for example, controller 102 is unable to automatically generate a tag or if setting are provided prohibiting controller 102 from automatically generating a tag for an umbrella metric.

[0092] At S425, controller 102 stores the tagged and summarized metrics in an associated memory (e.g., one or more memory or disks in network 100 or network 130).

[0093] At S430, controller 102 deletes individual system metrics based on which summarized umbrella metrics (the set of system metrics) are generated at S415. In one example, controller 102 only deletes a subset of metrics generated at S415. For example, controller 102 can receive input/feedback as to which metrics to store and which metrics to delete. In another example, controller 102 deletes old/individual metrics after a certain amount of time has passed from generation thereof, where the certain amount of time is a configurable parameter determined based on experiments and/or empirical studies.

[0094] At S435, controller 102 builds a machine learning model based on the tagged and summarized set of system metrics. Any known or to be developed machine learning algorithm can be used for training the model. In one example, the tagged and summarized metrics (tagged umbrella metrics) are fed into the machine learning algorithm for building a model.

[0095] At S440, controller 102 uses the machine learning model to determine a correlation between each summarized and tagged metric of the set of system metrics at S420 and the target metric.

[0096] At S445 and based on the determined relevancy and correlation, controller 102 ranks the summarized system metrics (the set of system metrics). In one example, the ranking results in arranging the summarized metrics in an order of relevancy thereof to the target metric that represents the network problem. The machine learning model, once trained using collected data, can determine correlation between various summarized and tagged metrics and the target metric by using methods including, but not limited to, cross validation and regression techniques.

[0097] In one example the processes of S400 to S445 can be referred to as a first step (first level) of a multi-step (multi-level) process for determining a root cause of a network problem. S450 to S485, as will be described below, can be referred to as a second step (second level) of the multi-step (multi-level) process for determining the root cause of the network problem.

[0098] At S450, controller 102 presents to the network operator a visual representation of the target metric and the ranked list of correlated summarized and tagged system metrics. In one example, the visual representation of the target metric and each correlated system metric is a time-series representation (e.g., a time-dependent graph) of the corresponding one of the target metric and each correlated metric.

[0099] At S455, controller 102 receives a feedback from the network operator indicating that one or more of the metrics correlated to the target metric is/are to be excluded from a second step of the multi-steps analysis for determining the root cause of the network problem. The one or more correlated system metrics identified for exclusion can be, for example, metrics that explain variations in a time series representation of the target metric except for one or more time instances during which the target metric exhibits unconventional/different/unique characteristics and behavior relevant to the all other times. This will be further described with respect to an example in FIG. 5.

[0100] At S460, controller 102 performs a second step of the multi-step analysis using only ones of the correlated metrics not identified for exclusion by the network operator at S455. This second step of the multi-step analysis can be referred to as a conditional analysis. For example, at S455, controller 102 can receive input for excluding CPU usage metric from the analysis for determining root cause of user request latency. Accordingly, at S460 and in performing the second step of the multi-step analysis, controller 102 excludes CPU usage metric from the analysis.

[0101] The conditional analysis performed at S460, similar to the first step at S440, is based on the machine learning model with the only exception that one or more metrics identified at S455 are now excluded. With remaining correlated metrics used as inputs into the model, controller 102 now determines an updated ranked list of correlated metrics that can explain the root cause of the network problem.

[0102] At S465, controller 102 presents the updated ranked list (i.e., results of the condition analysis of S460) to the network problem. Similar to S450, the result of the conditional analysis is presented as a time series of the target metric and a ranked list of each remaining correlated metric.

[0103] At S470, controller 102 determines if further exclusion inputs (similar to the input received at S455) are received from the network operator.

[0104] If at S470, controller 102 determines that no further input is received from the network operator, then at S475, controller 102 receives an input from the network operator designating one of the presented ranked list of correlated metrics at S465 as the root cause of the network problem. This input at S475 can either be provided by a network operator or can be a confirmation of a root cause determined by controller 102 and present to a network operator for confirmation.

[0105] Thereafter, at S480, controller 102 determines if a particular solution for addressing the network problem exists in a database. This can be for example, based on previous solutions presented (implemented) by the network operator for addressing similar network problems in the past.

[0106] If a particular solution exists at S480, then at S485, controller 102 presents the solution to the network operator. Otherwise, the process ends. In one example, appropriate actions can be taken to address/eliminate the network problem. For example, if the network problem is user request latency, in one example, additional processing capabilities may be utilized (brought online) to enable an efficient and timely service of user requests according to service level agreements (SLAs), etc.

[0107] Referring back to S470, if at S470 controller 102 determines that further inputs for excluding further correlated metrics are received, the process reverts back to S460 and controller 102 repeats S460 to S470.

[0108] In another example, there may be a further input after S485 at which the network operator provides instructions to controller 102 to address the root cause of the network problem. In one example, controller 102 addressed the network problem according to the received instructions. For example, the instructions for addressing latency in user requests can be to add additional resources (e.g., bring more processors online, utilize additional containers, etc.) to the network to enable an efficient and timely service of user requests according to service level agreements (SLAs), etc.

[0109] In one example, processes at S480 and S485 are optional and may be skipped or not performed at all. Accordingly, the process ends S475.

[0110] FIG. 5 illustrates an example of the process of FIG. 4, according to an aspect of the present disclosure.

[0111] FIG. 5 illustrates an example in which an inquiry (at S400) is received for determining a root cause of user request latency. Upon performing the process of S400 to S445 (i.e., the first step/level of the multi-step/multi-level process for determining a root cause of the user request latency, at S450, controller 102 displays a visual representation of four system metrics on a display. Graph 502 is an example time-series representation of user request latency metric. As shown in graph 502, there is a spike 504 in this time-series representation, which indicated an unusual behavior/unexpected increase of latency in user requests/queries.

[0112] Furthermore, FIG. 5 illustrates examples of three other system metrics that are correlated to the user request latency. One graph is graph 506, which is an example of time-series representation of CPU usage metric. Another graph is graph 508, which is an example of time-series representation of disk usage metric. Another graph is graph 510, which is an example of time-series representation of network usage metric. In one example, graph 510 is a representation of summarized metric of network usage determined at S430 based on metrics of bytes sent, packets sent, TCP ACKs, etc., as described above.

[0113] While in example of FIG. 5, a particular network problem is illustrated with only three correlated metrics, inventive concepts are not limited thereto. For example, there can be more or less correlated metrics associated with a network problem.

[0114] In one example, the presentation of graphs 506, 508 and 510 are in order of relevancy (correlation) thereof to the user request latency. As can be seen from FIG. 5, graph 506 of CPU usage closely follows the behavior of user request latency of graph 502 except at or near the spike 504. At the same time, graph 508 does not follow the behavior of user request latency of graph 504 as closely as graph 506 does but more closely follows graph 503 at or near the spike 504. Furthermore, graph 510 does not follow/match the behavior of graph 502, except at or near spoke 504. Accordingly, controller 102, by using the machine learning model developed at S435, ranks graph 506 higher than graph 508 and ranks graph 508 higher than graph 510. Accordingly, at S450, controller 102 present graphs 506, 508 and 510 in the ranked order of 1, 2 and 3 as shown in FIG. 5.

[0115] By observing graphs 502, 506, 508 and 510, network operator can determine that, while correlated. CPU usage of graph 506 is the least possible root cause of the user request latency because the behavior CPU usage of graph 506 at or near spike 504 is not correlated (does not follow) to the behavior of user request latency of graph 502.

[0116] Accordingly, at S455, controller 102 receives a feedback from the network operator to exclude graph 502 and corresponding CPU usage metric from further analysis (step/level two of the multi-step/multi-level process) for determining the root cause of user request latency.

[0117] Thereafter, controller 102 performs a conditional analysis of the root cause of user request latency based on disk usage and network usage system metrics (graphs 508 and 510) only and presents the result thereof to the network operator at S465. Thereafter, S470 to S485 are implemented until a root cause of the user request latency (e.g., either disk usage or network usage in the example of FIG. 5) is identified and proposals for addressing the network problem (if available) are presented to the network operator at S485 or in the alternative the network operator takes active steps to address the network problem.

[0118] Examples described above with reference to the accompanying figures provide an improvement to an analytics platform designed to monitor and identify root causes of various problems that can occur in a network of computer systems. This improved analytical platform improves the state of networked service platforms that provide various types of services including, but not limited to, Infrastructure as a Service (IaaS), Software as a Service (SaaS) and Platform as a Service (PaaS) to their clients. By improving the efficiency of detecting and addressing various types of network component problems and application/service problems in the network using the example systems disclosed herein (as opposed to simply identifying correlated factors and metrics), the overall efficiency of the network and performance of its components can dramatically increase, since now problems are identified and addressed more quickly than is currently done by available analytics platforms.

[0119] For clarity of explanation, in some instances the present technology may be presented as including individual functional blocks including functional blocks comprising devices, device components, steps or routines in a method embodied in software, or combinations of hardware and software.

[0120] In some embodiments the computer-readable storage devices, mediums, and memories can include a cable or wireless signal containing a bit stream and the like. However, when mentioned, non-transitory computer-readable storage media expressly exclude media such as energy, carrier signals, electromagnetic waves, and signals per se.

[0121] Methods according to the above-described examples can be implemented using computer-executable instructions that are stored or otherwise available from computer readable media. Such instructions can comprise, for example, instructions and data which cause or otherwise configure a general purpose computer, special purpose computer, or special purpose processing device to perform a certain function or group of functions. Portions of computer resources used can be accessible over a network. The computer executable instructions may be, for example, binaries, intermediate format instructions such as assembly language, firmware, or source code. Examples of computer-readable media that may be used to store instructions, information used, and/or information created during methods according to described examples include magnetic or optical disks, flash memory, USB devices provided with non-volatile memory, networked storage devices, and so on.

[0122] Devices implementing methods according to these disclosures can comprise hardware, firmware and/or software, and can take any of a variety of form factors. Typical examples of such form factors include laptops, smart phones, small form factor personal computers, personal digital assistants, rackmount devices, standalone devices, and so on. Functionality described herein also can be embodied in peripherals or add-in cards. Such functionality can also be implemented on a circuit board among different chips or different processes executing in a single device, by way of further example.

[0123] The instructions, media for conveying such instructions, computing resources for executing them, and other structures for supporting such computing resources are means for providing the functions described in these disclosures.

[0124] Although a variety of examples and other information was used to explain aspects within the scope of the appended claims, no limitation of the claims should be implied based on particular features or arrangements in such examples, as one of ordinary skill would be able to use these examples to derive a wide variety of implementations. Further and although some subject matter may have been described in language specific to examples of structural features and/or method steps, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to these described features or acts. For example, such functionality can be distributed differently or performed in components other than those identified herein. Rather, the described features and steps are disclosed as examples of components of systems and methods within the scope of the appended claims.

[0125] Claim language reciting “at least one of” refers to at least one of a set and indicates that one member of the set or multiple members of the set satisfy the claim. For example, claim language reciting “at least one of A and B” means A, B, or A and B.