Enterprise management system

[0001] This application claims benefit of U.S. Provisional Application Ser. No. 60/206,014, filed May 19, 2000, entitled “Enterprise Management System” and U.S. Provisional applciation Ser. No. 60/185,443, filed Feb. 28, 2000, entitled “Enterprise Management System.”

[0002] The present invention relates generally to the field of network management, and more particularly to an elegant system and method for monitoring and controlling large, growing and complex networks.

[0003] With the use of networks and networking burgeoning, numerous applications requiring communications and access to remote data are being performed more efficiently and in an enhanced manner. Following this trend, many enterprises including banks, insurance companies, airlines and numerous other businesses have become ever more reliant on the timeliness and accuracy of data and applications run on a network.

[0004] Despite the productivity benefits arising from this widespread automation of tasks, as applications become more advanced, computer systems, and especially networked computer environments, are becoming ever more complex. The lack of standardization among network protocols, server platforms and individual application software typically remains a stumbling block to enterprise-wide integration of applications and data. When varied applications and services are integrated, component failures and down time often result. Moreover, in complex, integrated network environments, problems are often difficult to determine and the resumption of criticial services may take time, resulting in losses to the enterprise.

[0005] In an effort to manage complex network environments, network management systems have been developed by various software/hardware vendors. These conventional management systems are generally characterized as having a topology of a single central managing entity, which controls all the management systems. Centralized management is often implemented with one or more powerful computers that allow access to all components of the managed site, monitor all site nodes, and accept or raise alarms or notifications from such physical nodes. However, a centralized management system that is run from on one or two servers may often experience significant problems. Such systems lack scalability and create performance bottlenecks, thus making the centralized management system unsuitable for managing very large, rapidly expanding sites. Moreover, because a single point of potential failure exists (i.e., the management server), such systems often lack the availability and robustness warranted given the importance of the applications and data typically stored on an enterprise network. Moreover, such conventional network management system are limited in that the focus is often on managing and controlling physical elements (e.g., nodes connected to the management server), rather than the more abstract concepts of interest to users and site administrators (e.g., the health of services, applications). In addition, in the event of error or component failure, the lack of intelligent differentiation amongst software applications and services often makes determining the problems a more difficult task.

[0006] In view of the above, there is a need for an improved management system that overcomes the limitations of the prior art. In particular, there is a need for a scalable management system that is capable of managing a large number of servers over a wide geographic area. There is also a need for a management system that is robust, and that provides intelligent, meaningful feedback to the site administrator in the event of failure. The present invention provides a solution to these problems.

[0007] The present invention is directed to a network management system and method performed at a network site that includes a manager engine computer capable of monitoring, storing and acting upon, network state information. The network state information is organized as a series of relationships among managed network elements. The managed network elements may include physical nodes connected to the manger engine, applications, subroutines, services, required data or any other element located or peformed on the network, whether physical or logical in nature. The managed network elements assume one of a set of predefined states, thus indicating an error or the potential for failure arising from a managed element.

[0008] The manager engine is connected to a client, which is capable of providing a user with the network state information stored at the manager engine. The manager engine may also be connected to other manager engines, each assigned to monitor respective managed elements, usually including connected physical nodes. The connected manager engines may communicate with one another so that, for example, in the event of a failure, one of the manager engines remaining on line commences monitoring of managed elements assigned to the failed manager engine.

[0009] In addition to basic managed element state monitoring functionality, the manager engines may provide users with a variety of automated features ensuring the health of the network and optimal failure correction in the event of a problem. For example, the manager engine may perform root cause analysis utilizing an algorithm tracing through manged element relationships and indicating the source of a failure, whether logical or physical in nature. The engine managers may include audit service functionality, automatically logging network events leading to failure and further facilitating the early detection of any potential errors. Again, the site administrator may have access to such audit information via the client.

[0010] Preferably, the client interface displays relationships among managed elements using meaningful connectors and tree-like structures. Such interfaces further enhance error detection and remediation by a site administrator.

[0011] Thus, one adavantage of the present invention is that, by distributing management components, it eliminates or reduces performance bottlenecks, single point of failure issues, and allows independent network scaling.

[0012] Another advantage of the present invention is that, by facilitating communication among distributed manager components and self assignment of managing tasks by such components in the event of a detected failure, the mangement system achieves enhanced availability and robustness.

[0013] Another advantage of the present invention is that, because the basic monitored unit (managed element) may be a logical or software component as well as a physical component, the system can provide more detailed, intelligent information relating to failure events or potential failure events.

[0014] Another advantage of the present invention is that it provides an automated audit storage component, thus providing further useful information relating to events potentially leading to problems.

[0015] Other advantages of the present invention will become apparent from the more detailed description below.

[0064] The present invention is directed to a single network management system, which allows administrators to monitor and control physical and logical elements (i.e., managed elements) comprising the network. The enterprise manager system may preferably use any of the following technologies: a) DDS (DaVinci Designer Space) version 7.0; b) WMI; c) Enterprise Event Logs; d) Core infrastructure; e) WMI Providers; f) SQL Server; g) IIS; h) Healthmon; i)Win32—Processes; j) Win32—Memory; k) Win32 Network; 1) Win32—Disk storage; m) Win32—Hardware; n) Win32—Event Log; o) Win32—Security; p) COM+; q) Active Directory; r) DHCP Server; s) WINS Server; t) DNS Server; u) Job Object; v) SNMP; w) Exchange; x) Terminal Server; and y) Cluster. In addition, the enterprise manager system may preferably incorporate use the following protocols: a) WMI for all management functions (DCOM); b) SNMP for non-MS platforms; c) MMC for client access with Web interfaces using DDS; d) MSDE (SQL Server) for operational and performance data storage; e) MSMQ; f) DCOM; g) Perfmon; h) HTTP (IIS & IE5.x); and i)XML. However, while the invention is described with respect to specific network, server and database architectures, platforms and protocols, it should be understood at the outset that the scope of the present invention is by no means limited to those contexts.

[0065] I. Overall Network Architecture

[0066] FIG. 1 depicts an overview of an organization of a network in which an embodiment of the present invention may be implemented. In particular, the network system illustrated comprises logical and physical components. The top most component is the Managed Site (10), a logical entity, which is composed of a number of Sub Sites (20) in a one to many relationship. A Sub Site (20) is a logical component, which is composed of a number of Engines (30). An Engine (30) is a physical component. It is the channel through which monitoring occurs. An Engine (30) can be in one Sub Site (20)—a one to many relationship. An Engine (30) is the focus point for the monitoring of Managed Elements (50) which belong to a Node (40). An Engine (30) therefore can be thought of as a collection of Nodes (40). A Node (40) is a physical component but from the perspective of certain software components of the Engine (30) to be described later, it is a logical entity that describes the physical one. A Node (40) relates to its Engine (30) in a many to one relationship. A Node (40) is a collection of Managed Elements (50) (while being an Managed Element (50) itself) with the restriction, in the presently illustrated embodiment, that a Node (40) can only be monitored in one Sub Site (20). Thus, in the embodiment illustrated, all Managed Elements (50) relating to a Node (40) must be monitored by the same Sub Site (20). A Node (40) relates to its Managed Elements (50) in a one to many relationship.

[0067] II. Overall Subsite Architecture

[0068] Turning now to FIG. 2, it shows a more detailed illustration of a managed sub site (20) depicted in FIG. 1. In particular, the subsite (20) consists of the engine (30) connected to server nodes (40). One or more clients (110) are connected to the management engine (30) and access management engine (30) information relating to managed elements (50) including nodes (40). The Network client (110) here illustrated is preferably an MMC snap-in application running on a computer. In FIG. 2, two network clients (110) using the services and data of both the Network Management Engine (30) and a Capacity Planning server (120).

[0069] A management site or subsite (20) is managed by one or many Network Management Engines (30). If there is more then one engine (30) available, the client (110) still only connects to a single engine and therefore will be dealing with one complete set of data and view. It is up to the engines (30) to keep each other up to date and make sure that all the clients have a complete picture of the entire managed site. FIG. 3 illustrates a more detailed illustration of a managed subsite (20) depicted in FIG. 1 in which the site is managed by more than one engine (30). In accordance with one embodiment of the present invention, FIG. 4 illustrates two managed subsites (20) depicted in FIG. 1 which are both accessed separately by a single client (110) and capacity planning server (120).

[0070] The manager engines (30) may use Active Directory technology to employ auto discovery for servers and services installed in the enterprise. As an example, an SQL Server node (40) is installed and it registers itself in the Active Directory. The manager engines (30) system will watch for this type of activity and automatically create a new managed element (50) and notify support personnel.

[0071] a. Subsite Architecture Detail—Managed Elements

[0072] A Management Element (“ME”) (50) is a single conceptual management point. ME (50) can represent a wide range of objects, such as: a) a computer or physical node; b) an application; c) a resource grouping; d) an event, error, root cause analysis reference; or e) an external interface with the management site or other point.

[0073] In the presently described embodiment, an ME (50) has the following basic information.

[0074]

	Information	Description and Use
	Display Name	Display name of the ME and is used by
		clients and reports.
	GUID	Unique ID used for referencing the specific
		ME.
	ME Type	Type of ME (Resource, Base ME, etc . . . )
	Parent ME Type	Type of parent that the ME is dependent on
		like a SQL database must have a parent that
		is a SQL Server. There can be more than one
		parent ME type.
	Interface Object	The ME's interface object that handles and
		will abstract the management of the element.
	Properties	Named + Value properties
	Policies	Instructions on how to execute commands,
		how to react to specific messages, maximum
		state propagation, etc . . .

[0075] A Base ME is a well-known (usually pre-described) description of management object that is used to build a management group. These types of elements have management information imported into the system that will allow it to understand how to gain access to the element's information and commands. If the ME is managed by WMI then the imported information will describe what WMI object to use, what events to setup, and what commands to execute.

[0076] The following is a list of some base ME's in accordance with various embodiments of the present invention: a) SQL Server; b) SQL Database; c) DHCP Service; d) DNS Service; e) Active Directory Service; f) Domain Server; g) COM+Server; h) File Server; I) Printer Server; j) RAS Server; k) File Share Point.

[0077] A Derived ME adds additional information to a base ME. It will still reference a specific instance of a Base ME but will provide additional properties and information that is directly related to how the Base ME is being used by the application. FIG. 5 illustrates how a Derived ME derives or inherits information from a Base ME. In that Figure, there are two base MEs, SQL Server and SQL Database. When an application is installed it will create a derived ME that will reference the Base ME. It will also add additional information, properties, and policies that will support the application using the specific resource. In the example illustrated in FIG. 5, there are two applications that are using the same database. One has a critical need and the other just uses its information for reference. If the SQL Server went off line for some reason, in the presently described embodiment, the management system would propagate a “critical” state for application X and a “warning” state for application Y. This type of information will allow support personnel to prioritise their work based on business needs.

[0078] An Application ME describes a pre-packaged application that can be executed and will use resources in the managed site. An application ME will reference other MEs that will eventually reference Base MEs like a SQL database. These relationships will describe to the system the topology of what the application and what resources it will need when it is installed.

[0079] The following are possible examples of application ME's: a) Congo for ITG; b) SAP; c) CITS (Customer Information Tracking System); and d) Beta Program.

[0080] A Resource ME is an element that controls a set of resources that provide the same service. It will contain rules that allow the management system to figure if the redundant resources are above or below a critical level. The following is a list of some types of possible resource ME's: a) Group of SQL Servers; b) Group of DHCP Servers; c) Group of Mail Pumps services; d) Group of Print Servers; and e) Group of File Servers.

[0081] A Reference ME is usually a short-lived element that allows the user to drill into management system that describes some type of information that needs to be managed. One use for this type of ME is the result of a root cause analysis. The operator can drill down using this reference to see what the problem is and what resources are being affected. The following is a list of some types of Resource ME's: a) Root Cause Isolation; b) Problem short cut; c) Tracing activity; and d) Job execution.

[0082] An External Interface ME is a type of placeholder used in an application diagram that will show where the external interface(s) are located. This will allow the management system to understand at what point does the application impact the user. It is also used to calculate where the implicit state propagation points are. The following are some of the types of External Interface ME's: a) Client application; b) Web application; c) Data message stream; and d) Satellite data fee.

[0083] Any ME, service, outside application, etc. can send a message to the management system. Likewise, any service can subscribe to specific messages to extended the management system or add application capabilities. Business Messages are events received by the management system that inform it of business activities. As an example, a process (clock) can send a business message that states the market is open. This message can enable a specific ME, thus informing the management system to start paying attention to a trading system. Likewise a business message that states the market is closed would disable the specific ME.

Here is an example of a Business Message flow:

• • [0084] Market open → Enable and/or start the trading application and monitoring
  • [0085] Market close → Disable and/or stop the trading application and monitoring

[0086] A managed group is a collection of MEs that represents a complete application or service, or a logical subset. In a managed group the MEs must have an operational relationship with each other. This information is used by the management system to figure out how to propagate states to dependent groups or ME.

[0087] A ME in the management system will usually have a relationship to one or more MEs. The management engine uses relationship information to perform several activities including root cause analysis.

[0088] Basic state propagation rules are based on relationship between MEs. These rules govern how state is propagate from one ME or managed group (collection of ME) to another. The basic premise is that if you correctly model the real relationship of services and resources, the management system can use this information to understand if a service or application is running, or if there is a problem, what is the most likely cause.

[0089] FIG. 6 illustrates an exemplary set of relationships, in the form of state propagation rules, among the ME's (50) depicted in FIG. 1. In FIG. 6, the services and resources illustrated belong to the application “Congo for ITG”. At the top level is a single ME that represents the application and will display its aggregated status. This ME will also allow the support personnel to issue commands that effect the application.

Implicit State Propagation Point

[0090] Implicit state propagation point is a ME that will push its state status for the managed group. In the example illustrated in FIG. 6 there are actual four implicit propagation points, the “Agent Client”, “Web Client”, “Archive Service”, and the “Survey Agent Service”. Implicit propagation points are identified by ME's that no other ME's in the group have a dependency on.

One-way Relationship

[0091] One-way relationships indicate that the ME is dependent on the related ME in some fashion. This could mean that the ME uses the other ME's services or that it receives data from the ME. In other words, the ME that is dependent on another ME cannot operate without the other ME running. Likewise, the other ME can operate without the other ME running.

Two-way Relationship

[0092] Two-way relationships indicate that the MEs are dependent on each other and neither ME can operate effectively without the other.

Explicit State Propagation Point

[0093] A resource ME can be used to provide an explicit propagation point that can have policies that control what state to publish. In the above example the “Mail Pump Resource” ME has three rules or policies. The policy requires that all mail pumps must be operating in order to publish a “running” state. If none of the mail pumps were operating, it would publish a “critical” state. Any other combination would publish a “warning” state.

[0094] The resulting state would be pushed up to the “Mail Pump” ME in the middle layer, which in turn would push its state to the application ME at the top level.

[0095] It is possible to perform root cause analysis using the ME relationship information. This would allow the engine to figure out what is the primary cause of the problem based on the messages that the management system is receiving or generating.

[0096] In the above FIG. 6 the “Congo for ITG” application uses two SQL databases but it does not really show which SQL server these database reside on. The management system would have another model, which would group the SQL, database together based on SQL Server. Using both sets of information it is possible to figure out the dependencies.

[0097] If a specific SQL Server fails and it provides the database services that several applications are using, each application that it supports and the SQL Server itself would generate “critical” messages. Using the application's information and the SQL Server resource group, the management engine would see that the common point is the SQL Server and would create a Reference ME. The reference ME would point to the SQL Server which is the problem and would also provide links to the resources that are being affected.

[0098] Also since the management system understand the ME's relationships, the operator could drill down and research all the applications that the SQL Server was affecting.

[0099] To effectively manage an application and services, the system must understand what its structure is, where the components or services are located, and what each is dependent on. There are at least two ways to collect this required information and both way are preferably supported.

[0100] With respect to the first way, profiles are like templates that provide all the structure information but do not specify the resource identifiers. The profile would state that the application requires two databases, and maybe the required database name, but it would not specify the server name. When the application is installed the management system would require the support personnel to fill in the blanks before it could effectively manage the application.

[0101] Profiles specify several pieces of information beside structure. The following is a list of some of the information that can be specified: a) Application structure (types of resources and servers, dependencies, etc; b) ME specific information (base ME, messages to look out for and how they effect the state rating (error, events, application msg, etc)); c) Reactive Policies (react to msg and execute jobs or scripts); d) Configuration Policies; e) Monitoring Policies (how to detect healthy running state); f) Proactive Policies (jobs or scripts that watch the health of the system); and g) Business critical rating for the application—how important is the application.

[0102] The support personnel can also use the management tools to interactively setup the application structure and policies.

[0103] The management system should preferably support backup and resource capabilities. In addition it should also preferably support import and export of specific application or resources.

[0104] With respect to the second way, the management system can also dump state of some or all of its MEs to provide a base set of information. Message histories can then be used to analyze the system as a whole as the management site supports the application and users.

[0105] Preferably, clients will provide several management views and tools that will help the support personnel to organize their activities.

Job Status and Alert Display

[0106] A job status and alert display post current problems like a down server along with who is working on it, what is its problem, and what is the ETA.

Alert View

[0107] The console can come with some pre-configured management views like a view that will only display current problems. Root cause analysis will create a temporary ME that point to the problem and which will be displayed in the alert view.

[0108] b. Subsite Architecture Detail—Client

[0109] i. Client Overview

Client Installation

[0110] In the presently described embodiment, the client can be installed on all versions of Win2K, but not on NT 4.0 or Win9x. The MSI will install the following client components (it is assume that OLEDB for SQL is already be present with the OS): (1) MMC snap-in; (2) Network's Active X components; and (3) DDS components.

Client Components

ME ICON Control

[0111] This control provides the user interface for each type of ME. State icons are displayed in the lower left of the ME icon.

MMC Snap-In

[0112] In the presently described embodiment, the snap-in integrates the Network system into the MMC framework. Again in the presently described embodiment, it provides the UI and admin features for the Network system. The MMC Snap-In application preferably features: a) support DDS control; b) provide a wizard for adding an Engine and Interrogate the Active Directory for potential Management engines; c) provide a wizard for removing an Engine; d) provide a wizard for adding a ME object via a template or known topology requirement and interrogate the Active Directory for potential resources such as SQL Server; e) provide a wizard for creating a group; f) provide a wizard that allows the user to add ME to a group; g) provide support for the 5 primary views in right pane—Large ICON, Small ICON, List, Detail, and DDS (dependency); h) provide features that allow the user to describe use dependencies; i) provide support for application template support; j) provide a wizard for importing and exporting ME information; k) provide a wizard for managing properties for each ME; and 1) provide a wizard to setup monitoring policies for a ME.

[0113] Preferably the following views are supported: a) Managed Groups; b) Virtual views based on ME relationship (required and uses); c) Dynamic Groups based on a SQL query; d) problem display—ME that are critical or warning, root cause analysis result; e) trouble ticket interface (incident); f) ME Properties with query wizard; g) ME History with query wizard; h) System tracing (debug control) view; i) System message view; and j) SLA Reporting.

Client Interface

[0114] In the present described embodiment, there are basically two client interfaces to the Network Management system. There is the WMI object interface and the graphical UI presented in MMC and IE.

MMC

[0115] The MMC presents a graphical depiction of the MEs and their state. The top level of the MMC scope will list the managed views. The right pane will display the ME icons, which is really an interactive control that provide the interface to the management elements. These ME icons will immediately show the user what the aggregated status is and provide an interface to the properties, aggregated views, and commands.

[0116] FIG. 44 illustrates an exemplary interface which may be used in a client component in accordance with one aspect of the present invention. For example, FIG. 44 displays in the right pane is showing two different types of MEs, resource and application. All of the ME's represents managed groups.

[0117]

	ME	Type
	Clusters	Resource
	Congo for ITG	Application
	Congo for PSS	Application
	DHCP	Resource
	Exchange	Application
	SQL Server	Resource
	WINS Service	Resource

Drill Down

[0118] In the above display the user can immediately see the overall operational status of all the managed elements (ME). If one of the icons shows a problem the operator can drill down to show what managed elements are contributing to the problem.

[0119] In the above example there is two applications listed, “Congo for ITG” and “Congo for PSS”. While these are the same application there are two instances installed, with each using their own IT resources. Double clicking or executing the drill down command will cause the console to display the children of the virtual ME.

[0120] FIG. 45 illustrates a user interface which may be used in a client component in accordance with one embodiment of the present invention. As illustrated in FIG. 45, Network System will show the elements that make up the “Congo for ITG” application. Along with the icons, the console may also display the relationships that each ME has with other ME(s). In come cases two MEs can have a relationship with each other that really indicates information flow. Note that in the example the “Mail Pump” icon has a one to one relationship with the “Congo DB”. This is telling the management system that the application cannot run unless these two resources are up and running.

[0121] This application will also support multiple instances of specific services like the “Mail Pump”. Specifically this application has three types of service that can have multiple instances. Drilling down again shows the dependency structure and allows the operator to interact each managed element.

[0122] FIG. 46 illustrates an exemplary user interface which may be used in a client component in accordance with one embodiment of the present invention. As illustrated in FIG. 46, client station is showing a resource ME that controls how the mail pump services as a group are managed.

ME Icon Layout

[0123] FIG. 47 illustrates the information conveyed by the ME icon in the present embodiment. State icons are displayed in the lower left of the ME icon. Possible state icons are illustrated in FIG. 48. Action icons let the user quickly inquire, change a configuration, or invoke a command or WMI method. Possible Action icons are illustrated in FIG. 49.

Resource ME

[0124] The top icon in the “Mail Pump” display above is another example of a virtual ME that is configured to understand several operational parameters. Some of these rules control how the management system propagates health state (running, warning, and critical). Some of the potential rules are outlined as follows: a) how many healthy resources are required to show this resource as running; b) how many healthy resources are required to support basic functionality (this is used to figure out if the resource is critical or a warning state should be propagated).

ME Properties

[0125] When an operator execute the property icon, the ME's properties are displayed. The MMC console can do several things based on the ME configuration stored in the management database, e.g.: a) display a dialog box that displays and may allow the user to modify the ME properties; b) dialog boxes can be tabbed as well as implemented as wizards; or c) display a Web page that basically acts like a dialog box.

[0126] There are several ways to allow the operator to see the properties, change configurations, and issues commands. The MMC console can display ME's properties using standard dialog boxes and other techniques. Some of these possibilities are listed as follows: a) launch standard MS MMC console passing the target information; b) launch a windows executable passing the target information on the command line; c) launch a MMC console passing the target information on the command line; d) display an OLE property page dialog box; e) display a generic WMI property page control; f) open a command window (cmd or telnet) to the target server; or g) open a web page for the target.

Web Page Properties

[0127] There are several advantages to using Web technology as the primary interface for managed element's properties including: a) server based support will result in a smaller client footprint; b) broadens the number of people that can extend the Network Management System (this is primarily due to the fact that the hardest issue to adding a new managed element is creating the management interface); and c) easier to update and maintain because of the web update technologies.

Aggregated Displays

[0128] An aggregated display will allow the support personnel to see information from several sources, displayed in a single control or window.

[0129] The following are standard aggregation displays. Each type of display is control by a profile that will allow for filtering and specifying targets.

[0130]

Type	Description
CPU Utilization	The CPU utilization would display all the CPU
Event	Display the events for all supporting services and
Object Type	Display performance or event counters for any

ME Execute

[0131] A floating menu is displayed when the operator clicks on the icon. Any type of execution command including scripts can be hooked up to this icon. Even though the user can change configuration using the properties icon, the commands available through execute icon can be also executed on all the children of the ME.

[0132] In the above example for the “Congo for ITG” application, it would be possible to create scripts for each ME that would start and stop each service. The operator can then chain these commands together using the dependency information. When the operator execute the “start” command on the top ME icon, all the children icon will also get their “start” command executed in the order specified by the dependences and other configuration information.

Management Consoles

[0133] The management console usually provides the primary interface with user for the management system. Consoles communicate with management servers via standard protocol (WMI). Management consoles may be responsible for: a) logging in and authenticating users (network support personnel, administrators, etc. . . . ); b) providing access to all management servers and functions; and c) provide response back to the user, display current status of management elements, etc.

Network Client

[0134] The Network client support is composed of three parts, a MMC snap-in application, API (scriptable) interface, and applets. There is no imposed limit to the number of Network clients that the management system can support.

MMC Snap-in Application

[0135] MMC snap-in is a root object that provides a graphical UI interface for managing the enterprise. This snap-in will use the API interface and along with the ME applets will provide the primary UI interface for Network.

API Interface

[0136] All Network services should preferably present a management and data interface. All management interfaces should be provided by implementing a WMI provider. Approaching the requirement this way will also provide customers with the documentation of the management objects (schema, WMI object browser, etc. . . . ), infrastructure support, and scriptable interface. Where every possible, all data interfaces should preferably be provided by a WMI provider (High Performance Interface). Where this is not practical the interfaces should be Automation COM objects.

Graphical Interface

[0137] Graphical interface can present a large amount of information and in some cases are easier to use the non-graphical interaction. Graphical interface can be, for example, a Web page, custom control, or custom application.

[0138] ii. Client—Managed Element Control Component

[0139] The primary responsibility of the Managed Element Component in the presently described embodiment of the present invention is to display graphically the state of the Managed Element, to provide user interfaces for activating/invoking actions and to modify the properties of the control.

[0140] For the purposes of illustrating an embodiment of the present invention, the icons for the command are assumed to be fixed. The command will be referenced by an id which will have a predefined icon name which can not be changed by the user. In this embodiemnt, this is a single COM component that will be used in DDS

[0141] The Managed Element Control Component is primarily responsible for graphically representing the state of the element and to provide user interfaces for setting and viewing the properties. It also should provide a user interface for activating or invoking action commands for the control. It should provide an interface to the aggregated view which will be used to display the custom status icons. FIG. 50 illustrates an exemplary ME icon.

[0142] The following items represent in the present described embodiment an ME: a) ME Title/Description (each ME has a title which will be displayed on the top of the icon); b) ME ICON (each ME type will have its own icon so that it can be easily identified). Users can select any image to represent the element, there are a few ways this can be achieved, the control can maintain a list of images and the icon can be selected at any time and it can be remembered, the other icons for State, Properties and custom icon can also be stored; and c) ME State (each ME can be in one of five possible states). The state of an ME is displayed as an icon at the bottom left corner of the control. The table illustrated in FIG. 48, lists possible values and corresponding icons.

ME Properties

[0143] This is a button and so when the user left mouse clicks on it, it will send an event to the container to show a user interface to manipulate the properties.

ME Action

[0144] This is represented by an icon and when the user right clicks on it the control will display a list of commands in a drop-down menu. The user can select an item in the menu and it will generate an event to the container/client. The menu can have separator items in between commands.

Interfaces

[0145] The ME Element will provide two interfaces in the presently described embodiment. The first interface is used to set/read the properties of the component. The state of the control can be get/set. Other properties are read only properties which will be used by the container and they do not need an user interface. The interface may have methods to Add, Remove commands in the commands drop down list. It should be possible to set, command text, id and icon for the menu item. This interface may also have methods to get the count of commands and get command by an index.

[0146] The second interface in the presently described embodiment is a connection point interface which will send events to the container so that it can be processed by the container. This interface may send an event when one of the following happens: a) any Keyboard Event; b) the user right mouse clicks on the Action Icon; c) the user selects a command from the action menu; d) the user double clicks on the control; e) the user right mouse clicks on the top portion of the control (to generate context sensitive menus by the owner.)

Component Interface

[0147] The owner of the process uses this interface to set/get the properties, to add, remove commands in the commands drop down list, to get the count of available commands and get a command by an index.

[0148] The following are possible classes and methods associated with the Managed Elemement Control Component.

Class INetworkMEControl

• [0149] {
  • [0150] public:
  • [0151] //Methods
• [0152] //Properties
  • [0153] BSTR m_—strMEIconName;
  • [0154] UINT m_—nCustomlcon1;
  • [0155] UINT m_—nCustomIcon2;
  • [0156] UINT m_—nCustomlcon3;
  • [0157] BSTR m_—strMETitle;
  • [0158] Short m_—nMEState;
• [0159] HRESULT AddCommand(BSTR strCommandText, short nCommandId, BSTR strIconName);
• [0160] HRESULT RemoveCommand(BSTR strCommandText);
• [0161] HRESULT GetCommandsCount(short &nCount);
• [0162] HRESULT GetCommandByIndex(short nIndex, BSTR &strCommandText);
• [0163] Private:
• [0164] HRESULT OnDrawAdvanced (ATL_—DRAWINFO &di);
• [0165] }

[0166] Note that to support property change notifications derive this control using the IPropertyNotifySink interface.

Property: m_—strMEIconName Type BSTR

[0167] This property can be used to set the icon name of the control.

Property: m_—nMECustomIcon1 Type UINT

[0168] This property can be used to set the icon used to set the first custom icon.

Property: m_—nMECustomIcon2 Type UINT

[0169] This property can be used to set the icon used to set the second custom icon.

Property: m_—nMECustomIcon3 Type UINT

[0170] This property can be used to set the icon used to set the third custom icon.

Property: m_—strMETitle Type BSTR

[0171] This property can be used to set the title of the ME.

Property: m_—nMEState Type short

[0172] This property can be used to set the state of the ME.

Method: HRESULT AddCommand

[0173] This method can be used to add a command to the drop down list of commands in the control.

RESULT AddCommand (BSTR strCommandText, short nCommandId, BSTR strIconName);

[0174]

	Parameter	Description
	Return	HRESULT
	StrCommandTe	Command Text
	NcommandId	Position of the command in the menu
	StrIconName	Name of the icon to denote the menu command

Method: HRESULT RemoveCommand

[0175] This method can be used to remove a command from the drop down list of commands in the control.

HRESULT RemoveCommand(short nIindex);

[0176]

	Parameter	Description
	Return	HRESULT
	Nindex	Command Index

Method: HRESULT GetCommandsCount

[0177] This method will can be used to get the count of active commands added to the menu.

HRESULT GetCommandsCount(short &nCount);

[0178]

	Parameter	Description
	Return	HRESULT
	Ncount	This will return the count of commands available

Method: HRESULT GetCommandByIndex

[0179] This method can be used to retrieve a command from the list of active commands in the control using an index.

HRESULT GetCommandByIndex (short nIndex, BSTR &strCommandText);

[0180]

	Parameter	Description
	Return	HRESULT
	Nindex	Index of the command in the commands list
	StrCommandText	The command text of the retrieved command

Connection Point Support Interface

[0181] This interface can be used to fire events to the owner of the component when one of the following happens: a) a Key is pressed; b) a RightMouse Click on the Action Icon; c) a Click on the menu item in the commands list; d) double Click on the Control; d) RightMouse Click on the top portion of the control. The following is sample code which may be associated with the connection point interface.

Class INetworkMEEvent

• [0182] {
  • [0183] public:
  • [0184] void Fire_—OnKeyPressed(long wParam, long lParam);
  • [0185] void Fire_—OnRightClick( );
  • [0186] void Fire_—OnCommandInvoked(short nCommandId);
  • [0187] void Fire_—DoubleClick( );
• [0188] }
  Method: Fire_—OnKeyPressed

[0189]

	Parameter	Description
	Return	Void
	Wparam	Wparam of the message
	Lparam	Lparam of the message

Method: Fire_—OnRightClick

[0190] This event can be generated only when the user right clicks mouse on the top portion of the control. This can be used by the container to show context sensitive menu

[0191]

	Parameter	Description
	Return	Void

Method: Fire_—OnCommandInvoked

[0192] This is an event method which can be used to fire an event to the owner of the component. This will fire OnCommandInvoked method on the owner when the user selects a command from the command list

[0193]

	Parameter	Description
	Return	Void
	NcommandId	Command id to show the command invoked

[0194] c. Subsite Architecture Detail—Capacity Planning Server

[0195] In the presently described embodiment of the invention, a Capacity Planning service may may consume the performance data that has been collected (real time or stored) and may provide analysis and planning capabilities. This service may also generate reports based on schedules. Capacity Palnning Servers (220) are illustrated in FIGS. 2–4.

[0196] d. Subsite Architecture Detail—Manager Engine

[0197] i. Manager Engine Overview

[0198] Management engines are the workhorses of the management system. A management engine may communicate with its managed nodes (Windows NT Servers, Cisco routers, and other network equipment) through proprietary or standard protocols like WMI or SNMP. In the presently described embodiment, management engines are responsible for: a) accepting, filtering, correlating events and information from managed nodes in its jurisdiction or managed site; b) gathering, storing, and analyzing performance information; c) updating and maintaining configuration parameters on managed elements; d) supporting automated and reactive tasks; e) support remote and local agents for managing nodes; f) provide a complete, relationship view of the managed site nodes, services, and resources; g) querying on demand the resources of the managed resource for their current configuration, state, and performance data; h) provide a means of configuring and fine-tuning the resources of the managed element; i) monitoring the resources and receiving unsolicited notifications and events from management elements; j) Support for a virtual management element with policies; k) Management Policy enforcement; and l) Complex job execution.

Network Management Engine

[0199] The management engine is a service or a set of services that provide the core capability to job management (policies), collect performance information, monitoring, react to problems, manage enterprise resources, and provide a platform for enhancing and extending management capabilities.

[0200] The Network Management Engine is really a set of services that provides the following capabilities: a) Client interface (WMI Provider)—Provides the COM object interfaces for all management functions; b) Messaging infrastructure (MSMQ) for communicating between Management Engines; c) Management coordinator (COM+Service); d) Performance collector; e) Management database (MSDE-SQL Server); and f) Performance database (MSDE and/or custom format). FIG. 7 depicts the services provided by the manager engines (30) depicted in FIG. 1.

WMI Provider

[0201] The WMI Provider is the primary API interface to the Network Management system. All clients may use its WMI schema to interact and extract information from the Managed Site.

Service Comm (Communication)

[0202] This module may handle the communication of messages between Network Management Engine services. This will may be COM+asynchronous object, or MSMQ queue, or some other type of guarantee message delivery system.

MSDE

[0203] Microsoft Data Engine—SQL Server 8.0 may be used for the database engine.

Perf Collector

[0204] The Performance Collector is a module that is responsible for collecting performance information from multiple targets. Performance information is stored in a performance database that is accessible to the Capacity Planning Service and other type of clients.

Management Data

[0205] The Management Data is a database the stores all of the ME objects, their properties, policies, relationships, etc. Some ME objects can be owned by other Network Management Engines but each Engine will still have a copy.

Performance Data

[0206] The Performance Data is a database that stores the result of performance collection activities. The data will be broken up into two types, active and historical.

Managed Coordinator

[0207] The Management Coordinator module may implements the bulk of the management features.

[0208] FIG. 8 shows the components of the management coordinator service illustrated in FIG. 7 in greater detail. The following table also lists those components in the presently described embodiment.

[0209]

Function	Description
Data	The Data Coordinator is responsible for presenting
Coordinator	and maintaining the management site's data. It
	may have an understanding of which Management
	Server own which Management Element's data.
	Using this information it will query or update the
	appropriate management database.
Root Cause	The Root Cause Evaluator will try to figure out
Evaluator	where the real problem resides. It will use the
	dependency information for the Managed
	Elements and cross-reference this to the problem
	events that are being received. If it is successful it
	will create additional objects (ME) and/or events
	that will point to the specific problems thus
	allowing support personnel drill into the details.
Message	The Message Coordinator will act like a traffic
Coordinator	manager for messages that it will receive and route
	to the appropriate services or clients. It basically
	acts as a distributor for publishers and subscribers
	of messages.
	For example if an Network client want to see the
	events from a specific SQL Server, the event
	coordinator will make sure that the WMI provider
	for that service will publishes the events.
	Another example of an event is when a client
	changes a property of a ME, the Data Coordinator
	will sent update events to all Network
	Management Engines so they can update their
	stores as well.
	This module will also coordinate message received
	from agents working on managed nodes.
Policy	The Policy Enforcement will understand how to
Enforcement	react to specific events like a SQL Server goes
	offline. A policy can also create a ME or events
	that can invoke other policies. In the long run we
	should be looking at completely using WMI policy
	system or adding fuzzy-logic capabilities.
Ownership	Each ME is will be owned by a specific Network
Coordinator	Management Engine associated with the
	management site. Also if there is more than one
	engine in the management site, each ME will also
	be assigned to another management engine to act
	as the backup owner.
	By assigning each ME to a primary and backup
	management engine, the Network system can
	provide high availability in case of an engine
	failure or ease the transition of removing an engine
	from the managed site.
	As Network Management Engines are added or
	subtracted from a managed site, the Ownership
	Coordinator will be reassigned these ownerships to
	the remaining management engines. This process
	allows the workload to be distributed among all
	the active management engines.
ME State	The ME State Manager will store and managed the
Manager	current status of all MEs owned by the
	management engine. This module will interpret
	the policies that tell it how to set the appropriate
	state for a specific ME. MEs that have parents will
	also propagate its slate to them. When a state is
	changed it will send events that can be displayed
	on the Network Client or kick off other policy.
Command	The Command Execute Scheduler will handle the
Execute	execution of a command remotely or via an
Scheduler	Network Agent. Each command can be designed
	to execute via an Network Agent installed on the
	target platform or it can execute the
	command/script locally on the Network Managed
	Engine's server. These commands are usually
	invoked by other messages, events, policies, or
	properties/command execute for the ME.
Management	The Management Service Coordinator will
Services	monitor the active state between each Network
Coordinator	Management Engine. When a new Network
	Management Engine is enlisted into a Management
	Site, this service will manage the load balancing
	work that is necessary to share the load equally
	with other services. It will also monitor the state
	of the managed services. If a service stops for
	whatever reason, either manually or automatically
	the service will be removed from the Managed
	Site. The ME that is currently owns will be
	reassigned to remaining Network Managed
	Engine.
Schema	The Schema Manger will accept or produce text
Manager	and/or XML schemas that describe the managed
	structures. This information includes describing
	what makes up a ME, what are the necessary
	conditions for each state, the resources that the ME
	depends on, policies, etc . . .
Discovery	The Discovery Manager is responsible watching
Manager	and adding new computers based on discovery
	policies like “add new SQL Server to this managed
	group”.
ME Legacy	If an element that needs to be managed is not WMI
Interface	enabled then it must have a COM object that
	provides its management interface. There is one
	component interface for every type of managed
	component that is not WMI enabled.

[0210] The primary interface for managing elements is WMI. This allows the management infrastructure to use schemas (WMI and Network) to describe the component's properties and how to do specific activities.

[0211] An element can be a server, a service on a server, or application specific. Elements can be grouped or related to other elements to form a specific picture of a managed element.

[0212] For components that are not WMI enabled, i.e. systems that don't provide WMI providers, components can be managed via ME Legacy Interface objects. These classes are COM objects and implemented a predefined interface that can be used by the engine to manage the element.

[0213] The Network Management Engine is designed to manage MEs in a site either remotely or with a resident agent. If the workload becomes too great the customer can add another Network Management Engine server to the Managed Site. Likewise, in the presently described embodiment, when an Network Management Engine fails, the remaining engines will mark it off line and reassign its responsibilities.

[0214] In the presently described embodiment, when the new Network Management Engine registers itself with the Managed Site's data residing in Active Directory, it will retrieve a list of other management engines that currently support the management site. With this information it will contact the other engines and begins several processes. The Ownership Manager will start assigning active ME to its self until the workload is balanced.

[0215] The Data Coordinator will also start to build the new management database with data from the other management engines. When this process is done the new management database will have a complete copy of the entire management configuration and supporting data. The only exception is the performance data will not be replicated. The Data Coordinator will accept instructions from the Ownership Manager that will allow it to prioritize its work so that active MEs can be assigned to the new engine as soon as possible. When an Network Management Engine is removed from the Management Site the Ownership Managers of the other engines will reassign the active ME to rebalance the workload.

[0216] High availability of the management system is achieved by providing the ability for the management engines to fail over management responsibilities to the remaining management engines. When an Network Management Engine fails, the remaining engines, based on ME's backup assignment, will now process the MEs that are currently assigned to the failed engine. The Ownership Managers will then begin their work to reassign MEs to primary and backup management engines with the goal of rebalancing the workload.

[0217] The clients of the Network Management Engine use WMI to configure and retrieve management information. The Network Management Engine will have a WMI provider that will supply the core interface between the engine and its clients.

[0218] The Network Management system must support a SDK that allows 3^rd parties and customer to extend its capabilities.

Managed Element Interface

[0219] In the presently described embodiment, all components managed by the Network Management Engine are managed through ME Legacy or the standard WMI Component Interface. The following lists possible requirements for adding a new management element to the Network system: a) create a WMI provider and MOF or create a COM object that implements the ME Legacy Interface; b) create a Management Element profile that describes the command, property page implementer; and c) Execute Com objects or scripts.

Installing a Manager Engine in the Network

[0220] This section will cover all the primary installation scenarios for the Network. It will show the process flows and outline the messages and how the maanger engine components interact. For engine installtion, the MSI will setup and install the following (it is assume that “OLEDB for SQL Server” is installed by the SQL's MSI or already be present with the OS): (1) MSDE or SQL Server standard (Network will use its own instance of SQL of shipping schedules work out); (2) Network service executable; and (3) Network service's COM components (WMI consumer, Message Switch, ME Assignment, etc.) Next, during insatllation the MSI attaches the Network's database and starts the Network's service.

[0221] This section will detail the start up process for the Network service. (See the detail design for the individual components for a review of the startup process) First, the service control manager starts the Network' service based on settings or a net start command. The Network service shell then creates and call “Init” on the following components in specific order listed: (1) Message Switch component; (2) A Database state update component; (3) Database replication manager component; (4) Engine to Engine component; (5) State Tracking component; (6) WMI Consumer component; (7) Job Processing component; (8) Auto Discovery component; and (9) WMI Provider Proxy component.

Engine Components

[0222] During engine installation, all components with the exception of the Message Switch will register with the Message Switch on which message they need to subscribe to.

Database State Update Component

[0223] This component subscribes to all messages that notify ME state changes. It creates a database connection to the Network database.

Engine to Engine Component

[0224] This component reads in the current engine list from the database and connect to MSMQ queues. It will also create an input queue so that it can receive messages. In addition it will retrieve the current Management Site Configuration Version and compare it with what its last record number is. If the number does not match the component will place the engine in an off-line mode. This will then require the support personnel to start the engine as if it was new (see WMI Consumer component discussed below).

State Tracking Component

[0225] This component reads the ME parent and child relationships from the database including the current ME state. This component will need to retrieve the state information again for MEs that are not owned by this engine to cover the whole that is created by the SQL replication update cycle. (Note: The database may need to be modified to record the date when the ME was last updated). All MEs that are owned by this engine have their state set to unknown. This component sends a single message to all engines that all of this engine's MEs are in an unknown state. (When the WMI Consumer is started up it will send out new messages regarding the new states of the ME, thus updating the system).

WMI Consumer Component

[0226] This component reads the database regarding the monitoring policies for all the ME's assigned to this engine. When a new engine is added to the Management Site several steps, set forth below, must be completed in order (The state-tracking component will be used to track the various states associated with adding an Network engine for the UI): (1) Install the Network service on a computer in the sub site (the installation process should ask for the Management Site and sub site's name, which is posted to the Active Directory); (2)using the Network snap-in wizard to enroll an engine, the user informs the system to add the engine (it should also be possible to do this at installation time); (3) the wizard executes a command that sends out a message to start the Network service (MSG=Start Engine); (4) DB Replication Manager receives this message and starts the process of initializing the new engine's database; (5)when the DB Replication Manager has completed the initialization of the new engine's database it sends out a message that the engine has been started (MSG=Engine is running); (6) the Engine-to-Engine component receives this message and setup a queue for communication; (7) the Engine Monitoring component then receives this message and adds the engine to its list of system to watch for heartbeats; and (8) the ME Assignment component also receives this message and starts load-balancing process.

[0227] The overall interacton of the above described components during engine installation is shown in FIG. 13, which depicts manager engine installation steps in acordance with an embodiment of the present invention. Such steps are described in the table below.

[0228]

Step	Description
1a	The WMI Consumer will read in all the determination and
	monitoring policies for all MEs that are owned by this engine. ME
	assignments are made based on ME affinity to its node.
1b	The State Tracking engine will read in all the ME and their
	relationships for the entire managed site. For MEs that it does not
	own it will also read in the current ME state. All of the Mes that
	this engine owns will have their ME state set to unknown.
2	The WMI Consumer will setup all WMI event syncs to receive
	state changes based on the ME's monitoring policies.
3	The WMI Consumer will use the ME's state determination policy
	to figure out the initial state.
4	The ME determined state will is sent to the message switch.
5	The message switch sends the initial ME state message to the State
	Tracking component.
6	The State-Tracking component will update its internal store with
	this information. Other ME states could be changed based on state
	propagation policies.
7	New state changes messages will be sent to the Message Switch.
8a	The client will be notified of any ME state changes for MEs that
	is has registered for.
8b	The Database Update component will receive the same state
	change message.
9	The Database Update component will update the database with
	the new state. This information is replicated out to all the
	other Management engines.

Manager Engine Operation in the Event an ME State Change is Received

[0229] A primary function of manager engines during operation is to notify the client (i.e., user) of stat changes. As noted above, when the WMI Consumer receives its assignments, it will implement the WMI monitor policies for each WMI and then use the determination policy to calculate the current ME state. Once this has been accomplished the WMI consumer waits for events to tell it that a ME's state has changed. FIGS. 14 and 40 depict the processing of an ME event in accordance with one embodiment of the present invention. Such proccesing is described in the table below.

[0230]

Step	Description
1	A WMI event is received by the WMI Consumer indicating a
	change in the ME's state.
2	The WMI Consumer generates a state change message for the ME.
2a	The client is notified of this state change via the WMI provider
	if it has registered for this event.
2b	The ME state change message is broadcasted to all
	Management engines
2c	The Engine's repository is updated regarding the ME event change.
3	The State Tracking component receive the ME state change
	message and processes dependent MEs based on state
	propagation policy.
4	A ME state change message is generated for each dependent ME
	that had its state changed because of the original event.
4a	The Engine's repository is updated regarding the dependent
	MEs state changes.
4b	The client is notified of the new state changes via the WMI
	provider if it has registered for this event.

It is required that all Management engines have the same ME state information and repository. The SQL Merge replication system will keep the repositories update to date but the state engines must receive and process the primary ME state change messages.

[0231] As depicted in FIG. 14, the Management engine that owns the physical node will detect the original ME state change, at which point the WMI Consumer component will change the WMI event into an Network Message stating the current ME state is running, warning, or critical. This message is broadcasted to all Management engines in the Managed Site.

[0232] Apart from notification of state changes appicable to ME's which a given manager engine is assigned, the monitor engine may also receive and act upon state changes applicable to other ME's as well. FIGS. 41 and 42 depicts a method of processing a foreign state change (i.e., a state change notification from another engine) in accordance with one embodiment of the present invention.

[0233]

Step	Description
1	Another engine detects the ME state change and send a ME
	state change message to all the Management engines.
2	State change message is received from another engine and
	is forward to the message switch.
2a	All the Network Clients are notified of the ME state change.
2b	The state change message is received by the State
	Tracking component.
2c	The Network Repository is updated regarding the ME state change
	received from the other engine.
3	The State Tracking component will process ME state change
	and change the dependent ME's state based on propagation
	policies.
4	Additional ME state change messages are created based on the
	dependent ME states.
4a	All the Network Clients are notified of the ME state change
	for the dependent Mes.
4b	The Network Repository is updated regarding the ME state change
	for the dependent MEs.

[0234] Thus far, what has been described with respect to typical manager engine response to ME state changes involves simple notification of the client/user. This allows the user/administrator to remedy any problems eveidenced by the satet change. However, the maneer engine is also preferably capable of exeucuting its own remdial schemas. In particiular, the management engine can execute a custom COM object or a script defined in the relevant ME's profile.

[0235] The following are examples of how powerful the job schemas can be. The two key attributes are the ability to execute command in parallel and the ability to wait for external messages to complete the job.

Storage Capacity Management

[0236] A job can be written to manage a storage folder if a capacity message is received when a physical drive is nearing its capacity. The job would clear out temporary files and backup others. When the files have been backed it will then delete then these files.

[0237] FIG. 9 depicts a storage capacity management job schema associated with an a managed element in accordance with an embodiment of the present invention.

Printer Problem

[0238] A printer is critical to a line of business application. If the printer or print server fails the management system can be programmed to make corrective actions automatically. In the below example the job will make the backup printer the default, move the current print jobs to the backup printer, wait for the primary printer to come back on line, and then make it the default again. In this case, all the operational staff needs to deal with is fixing the printer problem. FIG. 10 depicts a printer problem schema associated with a managed element in accordance with an embodiment of the present invention.

Service Problem

[0239] A stopped service has been detected by the management system. The support staff have programmed a job that will attempt restart this service. If it is successful, the system generates a “warning” message. If not, the job will generate a “critical” message. FIG. 11 depicts a service problem schema associated with a managed element in accordance with one embodiment of the present invention.

Diagnostic Help

[0240] Another key advantage of message processing is that the support staff can build a simple job that tells the management system to take some automatic action to help debug an application.

[0241] FIG. 12 depicts a diagnostic help schema associated with a manged element in accordance with one embodiment of the present invention. In the scenario depicted in FIG. 12, a message is received that indicates that a transaction has failed. As it happens this type of failure is intermittent and hard to reproduce. The support personnel set up a job that will turn on detail performance monitoring and system tracing at the first sign of a transaction problem.

Manager Engine Components Description

Engine-to-Engine Communication

[0242] The Engine-to-Engine Communication sub system of a managed engine in the presently described embodiment, has responsibility for establishing, routing, and maintaining communication links between all Network servers. It provides an interface for external COM objects to send and receive messages that are broadcasted to the engine community. This communication infrastructure should be implemented using MSMQ node-to-node configuration. This manager should insure that all engines receive the same message. This might require using DTC and transactional processing capability of MSMQ. The interface should support asynchronous notification of messages received. It should keep a list of all active engines and provide services for synchronizing this list. The Engine-to-Engine component provides primary support for ME state changes, fail over notification, configuration changes, ME assignments, etc.

[0243] Thus, in the present described embodiment, this component is responsible for handling communication between engines via MSMQ queues. This component's responsibilities are as follows: signup with the message switch to receive all messages that is required to be broadcasted to all engines in the Managed Site; manage the MSMQ queue connections to each engine in the Managed Site; support the ability to send a message to a specific engine; send message that indicate communication failures with an engine. This should be treated as an engine failure and may result in the eviction of an engine.

ME Assignment Manager

[0244] In the presently described embodiment, this manager is responsible for handling the ME assignments for primary and backup. The assignment manager should be elected for a sub site. It should make ME primary assignments first and send message to all engines in the sub site. It should make the backup ME assignments based on n−1 and send this message to all engines in the sub site. It should preferably only assign up to 3 servers for backups. In the presently described embodiment, a physical node can only be associated with a single sub site.

[0245] Thus, the ME Assignment manager is required to implement the load-balancing feature for the Network systemin the presently described embodiment. This component's responsibilities are as follows: provide an election process for promoting a server as the primary assignment manager; the primary assignment manager must monitor the load on each engine by sub site and make correction accordingly; assign MEs to engines for primary and backup roles based on sub site; respond to message of new engines.

[0246] Respond to message of engine failures; send message to engines when assignments are executed; and update database with assignments and version information.

Engine Database Replication Manager

[0247] In the presently described embodiment, this manager is responsible for handling and insuring that all changes to the Network's database are replicated to all the other engine's databases. It should preferably feature: self-configuration for publisher and subscribers with a backup publisher; monitoring of replication state and services; automatic reaction capabilities to fix any problems, which would include having a backup for the publisher; and support for adding or removing engines.

[0248] The manager should support database maintenance activities (backup, DBCC, etc.) for the engine(s) and publication databases and any other information required to support replication. The database support should include using MSDE with an instance called “Network” and should be treated as an internal resource and not available to other application.

Engine Monitoring Manager

[0249] In the presently described embodiment, each engine should provide a heartbeat message that is sent to all other engines based on management site configured value. This heartbeat should be sent via IP data gram and control by a policy stored in the management database configuration. The engine monitoring manager provides an interface that allows external COM objects to receive notifications that indicate an engine that has failed based on policy.

[0250] Thus, the Engine Monitoring component is responsible for monitoring the health of the Management engines in the sub site in the presently described embodiment. This component must also be aware of engine health policies that effect when an engine is marked off line and when it will be evicted from the sub site. This component's responsibilities are as follows: keep a complete list of Management engines based on sub site; listen for heartbeats from engines in a sub site; send a message when an engine has violated the off line policy; send a message when an engine has violated the eviction policy; this will cause the engine to be removed from its monitoring list; and respond to engine add and remove messages1; and manage the Management Site Configuration Version number.

Events and Tracing

[0251] The above managers must use a single trace log system and post significant event information to the NT event log. The trace system must support the capability to increase or decease the amount of detail while the system is up and running.

Interface Support Requirement

[0252] The above managers should provide information to the “State Machine” regarding state and Engine-to-Engine communication messages.

COM Object Interface Requirement

[0253] Each COM object in the Management engine must support the system object interface that allows the controlling system to control or provide information necessary to run the system as a whole.

[0254] The system interface should support the following: initialization of the COM object; shut down of the COM object; pause and Continue of the COM object's services; and COM Object's configuration is stored in the registry HKEY_—LOCAL_—MACHINE. Management Site configuration should be retrieved from the management's database.

[0255] There are several COM components that are required for the Management engine. Each component supports a INetworkControl interface in the presently described embodiment.

Network Service Executable

[0256] This is an executable for the Management engine and will provide the primary interface between it and the OS. This component's responsibilities in the presently described embodiment are as follows: handle the initialization of all the server's components when the service is started; manage the pause process; handle the shutdown process for the service.

DB Replication Control

[0257] The DB replication control is responsible for managing the SQL replication process in the presently described embodiment. This component's responsibilities are as follows: initialize new Engine's database (snap shot, adding it to the subscriber, etc); removing an engine from the publisher database; monitoring the health of the SQL replication (replication failures are treated as engine failures); support an election process for deciding which server will provide the publisher database and which server will provide the backup publisher database; monitor the health of the publisher database; keep the backup publisher database in sync with the primary; and provide the facilities to promote the backup publisher database to the primary role.

WMI Provider Proxy

[0258] In the presently described embodiment, this provides the proxy interface for the WMI Provider for the Management engine. This component may be required where the WMI provider will be in the Win Management address space. The component's responsibilities are as follows: provide an interface for sending messages; provide an interface for receiving debug messages; provide an interface for receiving system message via the message switch.

WMI Consumer

[0259] This is the primary monitoring component in the presently described embodiment. It reads the monitoring policies from the Network database and interacts with WMI on the target computers for the supported ME. This component's responsibilities are as follows: read ME policies from the Network database and setup the WMI events; respond to Engine failures message and instantiate WMI events where it is assigned as a backup; respond to Engine recover messages and remove its backup monitoring of ME; convert WMI events into state messages; and respond to ME Assignment messages and create/destroy WMI events.

Message Switch

[0260] The message switch is responsible to route message from one component to others in the presently described embodiment. The client must register for the messages they wish to receive and they will notify them when the message is received via a callback COM class. It will also provide queue services so that the clients do not need to respond immediately. This component's responsibilities are as follows: support for clients to register to receive messages; support for asynchronous message notification via callback; support for client queued messages via reference to primary queue; support for sending messages; support for retrieving and peeking of messages for a client.

State Tracking

[0261] The state-tracking component is responsible for tracking each active ME's state in the presently described embodiment. It will also track the state of an object but will not propagate its state to parents. This component's responsibilities are as follows: map out the dependent relationship between ME objects, parents (dependent of) and children (dependents); support ME state propagation model and policies; signup to receive ME state change messages; send ME state change messages when ME states are changed because of propagation; understand ME ownership so it can determine when to send state change messages; support non-ME objects that are not a ME but support some type of state without state propagation; support special ME that represent engines so that they can be monitored.

ME Database State Update

[0262] This component will receive messages and update the Network database accordingly in the presently described embodiment. The following messages will be subscribed to: ME state change; Engine failure; results of elections of components promoted to sub site managers.

Job Processing

[0263] This component will start jobs on the Management engine or on the target computer based on the job specifications in the presently described embodiment. These jobs are in response to messages that have been mapped out in the Network database.

[0264] Election Manager

[0265] The election manager will provide the communication coordination and election process for promoting a specific component to a manager role in the presently described embodiment. All assignments are based on sub site relationships.

[0266]

	Roles	Description
	ME Assignment	Controls the ME assignments for a sub
	Manager	site. This process implements the load
		balancing and fail over feature for
		Network.
	DB Replication	This is the role that identifies the server
	Primary	that will provide the primary SQL
		replication publisher's database.
	DB Replication	This is the role that identifies the server
	Backup	that will provide the SQL replication's
		backup database.

This component's responsibilities are as follows: a) internal track which server is the providing a manager role for the sub site; b) respond to engine failure messages by holding an election for the roles that the failed engine was providing; and c) for each role there can be suitable candidates that will be detailed in the Network database.

[0267] ii. Manager Engine—Database Design

[0268] The database design should fulfill the following requirements in the presently described embodiment: only the Network Management Engine or other specific services will access this database (no outside client will have access and the database will be treated as an internal resource); the database must store the complete set of information required by the management engine (the engine should not need to reference any other storage of information. See the following table regarding the information required.); support engine-to-engine synchronization and coordination of the database's information; and supports technical features required to provide scalability and high availability.

[0269] Required Information in the presently described embodiment is as follows:

[0270]

Object	Description
ME	Managed Element - This is the element object
	that Network can identify. There are several types
	of MEs (Server, SQL Server, database, file share,
	etc . . . )
ME Relationships	MEs can have three types of relationships.Group
	to ME
	ME to Group
	ME to ME (uses or requires)
Configuration	System configuration information like what sub
	site is the default.
ME Type	Describes the types of ME like Adv-Server, SQL
	Server, Group, Database, etc . . .
Possible Parents	Each type of ME may require a parent like SQL
	Server is required to run on a server of some type.
	This information allows the system to understand
	what are the possible parent types if any is
	required.
Sub Site	Provides a list of sub sites that are available.
Management Engines	Provides a list of engines that are enlisted in the
	managed site.
Management Engines	Provides a cross reference of which engines are
assigned to Sub Sites	assigned to support which sub sites. An engine
	can support more than one sub site.
ME assigned to Sub	Allows the engine to understand which sub site
Sites	the ME is assigned. A ME can only be assigned
	to one sub site.
Assignments of ME	The assignment of ME to engines provides the
to Engines	information necessary to implement high
	availability. These assignments are broken down
	into two types.
	Primary
	Backup with priority
	There is only one engine assigned to an ME as the
	primary. There can be multiple engines assigned
	to an ME based on priority.
Command List	Each ME can have a set of associated commands
	that can be invoked by operators or scripts.
Job	Stores the instructions on how to invoke a set of
	instructions be it either a script or executable.
	There are also standard environment
	variables that can be specified.
Monitor	Provides a list of performance objects that can be
	monitored based on a schedule.
Messages	Provides a list of messages by class, routing
	information, and message history.
Trouble Ticket	A simple list of what to be one, who it is
	assigned to, and if done, when. This can be used
	to post enterprise wide notification of work in
	progress.
Security	Security is integrated into the Network to permit
	and control users in creating groups, ME, jobs,
	schedules, and defining roles. User can be
	grouped into security groups, which can be
	applied to object in the same way as a user.
Properties	Many objects in the database can have many
	properties. These properties can be used to help
	manage a ME, sub site, engine, etc . . .
Schedules	Provides schedule information for when
	commands are available, when ME should be
	active, when to record performance information,
	etc . . .
Policy	Several objects in the database can have policies
	associated with them. Examples of these policies
	are monitor, state propagation, etc . . .

[0271] FIG. 15 depicts a manager engine (30) depicted in FIG. 1 with a database component. FIGS. 16 and 51 are exemplary overviews of database schemas. Below the figures are explained.

ME Role

[0272] The basic object in the Network system is the ME. Likewise the ME table is one of the primary tables and is used to represent all MEs including groups. Each ME is strongly typed and must have a reference to a ME Type row in the MeType table.

ME Relationship

[0273] There are two basic relationships that a ME can have with other MEs. A group ME can have zero, one, or many children ME. These ME can be any type including another group ME. A physical, resource, or virtual ME can have one of two relationship types, uses and depend upon. A use relationship indicates that the parent ME uses the child ME. A dependent relationship indicates that the parent and the child ME depend on each other. A problem with one ME will affect the other.

ME Types

[0274] Each ME is typed and based on this information the management system applies specific rules. It is also used to inform the user how the resources work with each other.

[0275]

	Description	Internal Code
	Windows Cluster	WINCLUSTER
	Windows Workstation	WINWORKSTATION
	Windows Server	WINSERVER
	Windows Advance Server	WINADVSERVER
	Windows Data Center	WINDATACENTER
	SQL Server	SQLSERVER
	Group	GROUP

Possible Parents

[0276] Each ME type can require a parent. The following table will list the possible parents for each ME Type. If there is a listing for a ME type then one of the parents is required. The values in the columns are the internal code for ME Types.

[0277]

	Type	Possible Parent
	WINADVSERVER	WINCLUSTER
	WINDATACENTER	WINCLUSTER
	SQLSERVER	WINWORKSTATION
	SQLSERVER	WINSERVER
	SQLSERVER	WINADVSERVER
	SQLSERVER	WINDATACENTER

Configurations

[0278] There is a table in the Network database that is used to store specific operational parameters such as the default sub site.

[0279]

Name	Value	Description
Enabled	Y	Indicates that the Network database is
		enabled and can be used by an Engine.
Default SubSite	Primary	Specify the default sub site to use when
		adding an active ME and not sub site is
		specified.

Default Groups

[0280] The database load scripts add the following default groups that are associated with ME type for auto discovery: SQL Server Group; Windows Workstation Group; Windows Server Group; Windows Advance Server Group; Windows Data Center Group; and Windows Cluster Group.

Messages

[0281] The database load scripts will create message for the Network name space.

Message Class

[0282]

	Description	Internal Code
	Microsoft	MS

Message

[0283]

	Class	Msg Number	Msg Type	Description
	MS	1	0	ME Running
	MS	2	0	ME Warning
	MS	3	0	ME Critical
	MS	4	0	ME Paused
	MS	5	0	ME No information

Security

[0284] The database is loaded with security roles and attributes. These are standard values that are used by the engine and snap-in to understand what and who can do what.

User Roles

[0285] User roles can define what the user or group can do and are based on the following. They can also provide additional information that is used for support purposes.

[0286]

	Description	Internal Code
	User Security	USER
	Group Security	GROUP
	User Role - Owner	UR-OWNER
	User Role - Primary Contact	UR-PRI-CONTACT
	User Role - Contact	UR-CONTACT

Security Attributes

[0287] Security attributes describe what action can be done on an object in the database.

[0288]

	Description	Internal Code
	Read Access	READ
	Write Access	Write
	Full Control	FULL
	Destroy	DESTROY
	Execute	EXECUTE

Policies

[0289] There are two default types of policies that are setup by the build scripts for the Network database. These are Monitor and State. Monitor specifies what is the WMI command to execute to monitor the object and state indicates what is the highest ME state the propagated.

Database Tables

[0290] Standard Columns

[0291] There are several columns used in the database schema that have a standard meaning.

[0292]

Column Type	Description
—ID	Any column name ending with “—ID” indicates that
	this is the unique ID that will represent a single row
	in the table.
	This column is usually an integer with the property of
	“identity”. This property tells the SQL Server to
	create a new ID for every new row.
—GID	Any column ending with “—GID” indicates that this is
	the unique ID column that will represent a row in the
	table.
	This column is a “uniqueidentifier” SQL type that
	stores GUID (Global Unique Identifier). This type if
	ID is guaranteed to be unique in any database. These
	types of ID are used by the Network system to
	support using multiple databases and the multi-
	mastering requirement.
—Flag	Any column ending with “—Flag” indicates that the
	column will only accept NULL, “Y”, or “N”.
	Usually there is a default rule associated with the
	column that forces its value to be “Y” or “N”.
Int—Code	This name stands for “Internal Code”. This code is
	used by the Network system to identify specific rows
	in the table. This allows the description column to be
	store the text is required by the customer.
Des	User supplied description for the object or row.

Data Types

[0293] The table and store procedure scripts all reference custom data types and these are defined as follows. Although the table definitions that follow do not specify their data types based on these custom data types, the database build scripts do.

[0294]

	Data Type	SQL Data Type	NULL Rule
	DES—Type	Nvarchar(60)	NOT NULL
	Flag	Nvarchar(1)	NULL
	GUID—ID	Uniqueidentifier	NOT NULL
	ID	Int	NULL
	IntCode—Type	Nvarchar(30)	NOT NULL
	IntNum	Int	NULL
	IP—Type	Nvarchar(20)	NULL
	Name—Type	Nvarchar(60)	NOT NULL
	OBJECT—ID	Uniqueidentifier	NOT NULL
	TextData	Nvarchar(3000)	NULL
	URL—Type	Nvarchar(200)	NULL

Tables

[0295] The following tables are defined in the logical model of the Network database. The tables required to implement the many to many relationship are not specified here but do exist in the ER Win model and the build scripts.

Table: ME

[0296] The ME table stores all information regarding a ME. ME can represent anything physical or virtual like a group or the result of a root cause analysis.

[0297]

Column Name	Data Type	Null Option	PK	FK
CurrentState	Int	NULL	No	No
Des	Nvarchar(60)	NOT NULL	No	No
Disable—Flag	Nvarchar(1)	NULL	No	No
IconName	Nvarchar(60)	NULL	No	No
InstanceName	Nvarchar(60)	NULL	No	No
IPAddress	Nvarchar(20)	NULL	No	No
ME—GID	Uniqueidentifier	NOT NULL	Yes	No
MeType—ID	Int	NOT NULL	No	Yes
NetName	Nvarchar(60)	NULL	No	No
SubSite—GID	Uniqueidentifier	NULL	No	Yes
URL	Nvarchar(200)	NULL	No	No

Current State

[0298] Current state is a numeric value that describes what the system thinks about the ME. This column is only valid for active ME.

[0299]

Value	Description
0	Paused
1	No information
2	Critical
3	Warning
4	Running

Disable Flag

[0300] The disable flag indicates that this ME is not currently in operation.

Icon Name

[0301] The name of the icon to be displayed in the ME is stored here.

Instance Name

[0302] There are services like SQL Server that can have multiple instances. Each instance is named and must be know to the Network Management Engine.

IP Address

[0303] If the ME represents a computer that has an IP address it value is stored here. This is only used for statically allocated address and is not useful for computers that use the DHCP service.

MeType_—ID

[0304] A foreign key that references a row in the MeType table. This creates a strong type for each ME.

Net Name

[0305] Used to store the network name for MEs that represent a computer on the network. It can also be used in a cluster ME to store the network name of the cluster.

SubSite_—GID

[0306] References the sub site that the ME is assigned to. This will force the engines to only assign the ME to engines that are supporting the sub site.

URL

[0307] This is the friendly unique name for a ME. It is usually created by the database's stored procedures based on what type of ME is being created, e.g. a SQL Server hosted on a computer named “DB1” would be “DB1.SqlServer”.

Table: MeRelationship

[0308] The ME Relationship stores the parent and child relationships for all ME. It also provides the storage for relationship type, watch window, and control data for displaying the ME in the DDS surface.

[0309]

Column Name	Data Type	Null Option	PK	FK
ChildME—GID	Uniqueidentifier	NOT NULL	No	Yes
ControlData	Nvarchar(3000)	NULL	No	No
ParentME—GID	Uniqueidentifier	NOT NULL	No	Yes
TwoWay—Flag	Nvarchar(1)	NULL	No	No
Uses—Flag	Nvarchar(1)	NULL	No	No
Watch—Flag	Nvarchar(1)	NULL	No	No
Watch—ICON	Int	NULL	No	No

Parent ME GID

[0310] This column will store the parent's ME GID.

Child ME GID

[0311] This column will store the child's ME GID. A parent can have multiple children.

Control Data

[0312] This column is used by the Network MMC snap-in to store display specific information for the ME. An example is that the parent ME is a group and each relationship to the children store the current position in this field.

Two Way Flag

[0313] This flag indicates that the ME to ME relationship is really two way. A problem with one ME will affect the other.

Uses Flag

[0314] This flag indicates that the parent uses the child instead of depends on relationship.

Watch Icon

[0315] If the child ME is to be display in the watch panel, this field will store the icon's name. This is usually a smaller version of the ME icon.

Watch Flag

[0316] This flag indicates that the child is to be displayed in the parent's ME control.

Table: METype

[0317] The ME Type table list all the possible types of MEs and provide the system requirement of strongly typing all ME in the system. This table is preloaded when the database is created can be added to as needed.

[0318]

Column Name	Data Type	Null Option	PK	FK
AutoDiscovery—Flag	Nvarchar(1)	NULL	No	No
Des	Nvarchar(60)	NOT NULL	No	No
GroupMeName	Nvarchar(60)	NULL	No	No
Int—Code	Nvarchar(30)	NOT NULL	No	No
MeType—ID	Int	IDENTITY	Yes	No

Auto Discovery Flag

[0319] This flag indicates to the engine to setup with the Active Directory to be notified when a new server of this type is added.

Group ME Name

[0320] This field stores the name of the ME group that will be used when a new ME type is discovered. If no group is listed the “unassigned” group will be used.

Table: PossibleParent

[0321] The possible parent table provide the information that states what ME can be a parent of another ME. It links a ME type to other ME types. If there is no relationship to another ME type then the ME can be independent of other ME like a group. If there is a listing in the possible parent table for a ME type then the ME associated with the type must have a parent of the given ME type.

[0322]

	Column Name	Data Type	Null Option	PK	FK
	Child—ID	Int	NOT NULL	Yes	Yes
	Parent—ID	Int	NOT NULL	Yes	Yes

Table: SubSite

[0323] The sub site table list all the sub sites that the managed site supports. Sub sites are associated with MEs and engines. This allows the installed engines to perform load balancing and fall over assignments.

[0324]

Column Name	Data Type	Null Option	PK	FK
Name	Nvarchar(60)	NOT NULL	No	No
SubSite—GID	Uniqueidentifier	NOT NULL	Yes	No

Name

[0325] This column stores the name of the sub site. This can be unique in the management site.

Table: Engine

[0326] The engine table stores all the Network Management Engines that support the Management site. These can further be grouped by sub site.

[0327]

Column Name	Data Type	Null Option	PK	FK
Engine—GID	Uniqueidentifier	NOT NULL	Yes	No
Name	Nvarchar(60)	NOT NULL	No	No

Name

[0328] This column stores the name of the Network Management Engine. This value must be unique in the management site.

Table: Assignment

[0329] The assignment table stores the relationship between the ME and the Engine. This information is used to provide the engine the information about which ME is assigned to which engine and what type of an assignment it is. Each active ME is has a primary assignment to an engine. It also has engines assigned to it as backups. Up to 3 backup engines can be assigned to an ME and the priority value is used to indicate the role. A priority value of 0 will represent the primary assignment and higher numbers will represent the backup. Backup assignments are priorities so that the system can handle up to 3 engine failures.

[0330]

Column	Data Type	NULL Rule	PK	FK
Engine—GID	Uniqueidentifier	NOT NULL	Yes	Yes
ME—GID	Uniqueidentifier	NOT NULL	Yes	Yes
Priority	Int	NOT NULL	Yes	No

Priority

[0331] This column specifies the assignment priority.

[0332]

Value	Description
0	Primary
1	First backup assignment
n	The nth backup assignment

Table: CmdList

[0333] The command list table stores the command(s) for each ME. It also includes display ordering, icon name, and separator indicator.

[0334]

Column Name	Data Type	Null Option	PK	FK
CmdList—GID	Uniqueidentifier	NOT NULL	Yes	No
Des	Nvarchar(60)	NOT NULL	No	No
IconName	Nvarchar(60)	NULL	No	No
Item	Int	NOT NULL	No	No
Job—GID	Uniqueidentifier	NULL	No	Yes
ME—ID	Uniqueidentifier	NOT NULL	No	Yes
Property—Flag	Nvarchar(1)	NULL	No	No
Separate—Flag	Nvarchar(1)	NULL	No	No

Icon Name

[0335] This column stores the name of the icon to be used when the commands are listed for the user by the snap-in.

Item

[0336] This field is used to control the order of commands to be listed by the snap-in.

Separate Flag

[0337] This flag indicates that a separation line is to be displayed when the snap-in list the commands.

Property Flag

Table: Job

[0338] The job table stores the actual job commands in script or command line form. Jobs are associated with “CmdList” or messages. A message can invoke a job, which can also send additional messages.

[0339]

Column Name	Data Type	Null Option	PK	FK
Cmd	Nvarchar(3000)	NULL	No	No
Des	Nvarchar(60)	NOT NULL	No	No
Job—GID	Uniqueidentifier	NOT NULL	Yes	No

Cmd

[0340] This column is used to store the text representing of the command to execute for the job. This can be a script or a command line program that is to be invoked by the snap-in.

Table: MessageClass

[0341] The message class table store the message types listed in the message master table. This table is actually used to provide multiple name space so that message numbers and descriptions of different systems can co-exist in the Network database.

[0342]

	Column Name	Data Type	Null Option	PK	FK
	Des	Nvarchar(60)	NOT NULL	No	No
	Int—Code	Nvarchar(30)	NOT NULL	No	No
	MsgClass—ID	Int	IDENTITY	Yes	No

Table: Message

[0343] The message table stores the message reference information of all the messages that the system can send or receive. This table is used as a master reference to map the message routing and provide support for the message history table.

[0344]

Column Name	Data Type	Null Option	PK	FK
Des	Nvarchar(60)	NOT NULL	No	No
Msg—ID	Int	IDENTITY	Yes	No
MsgClass—ID	Int	NOT NULL	No	Yes
MsgNumber	Int	NOT NULL	No	No
MsgType	Int	NOT NULL	No	No

Message Number

[0345] The message number column stores the message or event number based on the message class.

Message Type

[0346] The message type column describes the type of message, warning, error, or informational.

Table: MsgRoute

[0347] The message route table stores the different paths a message can take with the system. It maps what happens when a message is received from a ME and what message or actions (jobs) should be invoked.

[0348]

Column Name	Data Type	Null Option	PK	FK
Item	Int	NOT NULL	Yes	No
Job—GID	Uniqueidentifier	NOT NULL	No	Yes
ME—ID	Uniqueidentifier	NOT NULL	Yes	Yes
Msg—ID	Int	NOT NULL	Yes	Yes

Item

[0349] The item column value is used to order the message routing invocations.

Table: MsgHistory

[0350] The message history table stores all the messages that have been processed by the system for each ME. The data in this table must be retired based on a schedule to prevent the database from getting too big. This history is only valuable for a limited time and should not be kept for over a month.

[0351]

Column Name	Data Type	Null Option	PK	FK
CustDataType—ID	Int	NOT NULL	No	Yes
CustomData	Ntext	NULL	No	No
Des	Nvarchar(60)	NOT NULL	No	No
DetailDes	Nvarchar(3000)	NULL	No	No
FromME—ID	Uniqueidentifier	NULL	No	Yes
Msg—ID	Int	NOT NULL	No	Yes
MsgHistory—GID	Uniqueidentifier	NOT NULL	Yes	No
OrgDate	Datetime	NULL	No	No
ProcessedDate	Datetime	NULL	No	No
ToME—ID	Uniqueidentifier	NULL	No	Yes

Custom Data

[0352] The custom data column stores custom data from the message if there is any. The custom data type will reference the viewer that can be used to decode the message.

Detail Description

[0353] The detail description will provide a textual description that can be reported on or displayed in the snap-in.

Organ Date

[0354] The organ date will record the date when the message was first received by the Network system.

Processed Date

[0355] The processed date column stores the date when the Network processes the message. This is usually done when the message is processed by the message routing system.

To ME ID

[0356] This column will reference the ME that the message is directed to.

From ME ID

[0357] This column will reference the ME that generated the message. This value could be NULL if the message was externally generated.

Table: CustDataType

[0358] The Custom Data Type table describes the viewer to be used when displaying the data stored in the message history table.

[0359]

Column Name	Data Type	Null Option	PK	FK
CustDataType—ID	Int	IDENTITY	Yes	No
Int—Code	Nvarchar(30)	NOT NULL	No	No
Viewer	Nvarchar(3000)	NULL	No	No

Viewer

[0360] This column provide the snap-in and reporting systems on what type of viewer can be used to decode the custom data field in the message table.

Table: Monitor

[0361] The monitor table lists all the performance monitor objects available to the performance collector. It also provides a reference to the system to allow the monitor schedules to be created and maintained.

[0362]

Column Name	Data Type	Null Option	PK	FK
Monitor—ID	Int	IDENTITY	Yes	No
PerfObjectName	Nvarchar(60)	NOT NULL	No	No

Performance Object Name

[0363] This column details the performance object that can be monitored by the performance collector.

Table: MonitorSched

[0364] The monitor schedule table is used to create a unique ID and a relationship between the ME and a monitor object. The unique ID is required to provide a reference to a schedule that is associated with the object table.

[0365]

Column Name	Data Type	Null Option	PK	FK
ME—ID	Uniqueidentifier	NOT NULL	No	Yes
MonitorSched—GID	Uniqueidentifier	NOT NULL	Yes	No

Table: Object

[0366] The Network database is broken into two sections. One section is designed as a classic relationship model between information. The second part is designed to provide an object model that allows policies, properties, and security to be associated with any uniquely identified piece of information in the database.

[0367] Triggers on tables that require this object support added and remove rows for each row in their respective tables.

[0368]

Column Name	Data Type	Null Option	PK	FK
Object—GID	Uniqueidentifier	NOT NULL	Yes	No

Table: UserGroup

[0369] The user group table stores the current groups and is associated with n users.

[0370]

	Column Name	Data Type	Null Option	PK	FK
	Group—ID	Int	IDENTITY	Yes	No
	Name	Nvarchar(60)	NOT NULL	No	No

Name

[0371] This column details the name of the user group. The value must be unique for the management site.

Table: Users

[0372] The user table stores the users that are allow to interact with the Network system. User can be associated with objects with security attributes or associated with a user group.

[0373]

	Column Name	Data Type	Null Option	PK	FK
	Alias	Nvarchar(60)	NOT NULL	No	No
	User—ID	Int	IDENTITY	Yes	No

Alias

[0374] This column stores the alias for the user. This value must be unique for the management site.

Table: Config

[0375] The configuration table stores database and system configuration based on name+value.

[0376]

Column	Data Type	NULL Rule	PK	FK
Disable—Flag	Nvarchar(1)	NULL	No	No
Name	Nvarchar(60)	NOT NULL	Yes	No
Value	Nvarchar(3000)	NULL	No	No

Name

[0377] The value in this column will provide the name of the configuration.

Value

[0378] This column stores the value for the configuration defined by the name.

Table: Policy

[0379] The policy table stores policy information for objects in the database. These policies are strongly typed.

[0380]

Column Name	Data Type	Null Option	PK	FK
Data	Nvarchar(3000)	NOT NULL	No	No
Des	Nvarchar(60)	NOT NULL	No	No
Policy—GID	Uniqueidentifier	NOT NULL	Yes	No
PolicyType—ID	Int	NOT NULL	No	Yes

Data

[0381] The data in this column is used by the Management engine to setup the WMI command for implementing a policy or internal process.

Table: PolicyType

[0382] The policy type table provides the type information for all policies associated with objects like ME in the database.

[0383]

Column Name	Data Type	Null Option	PK	FK
Des	Nvarchar(60)	NOT NULL	No	No
Int—Code	Nvarchar(30)	NOT NULL	No	No
PolicyType—ID	Int	IDENTITY	Yes	No

Table: Property

[0384] The property table list the name+value properties for a given object. Like policies there are several ME that can have properties.

[0385]

Name	Nvarchar(60)	NOT NULL	Yes	No
Object—GID	Uniqueidentifier	NOT NULL	Yes	Yes
Value	Nvarchar(3000)	NULL	No	No

Name

[0386] This column stores the name of the property.

Value

[0387] This column stores the value for the named property.

Table: Relationship

[0388] The relationship table details the relationships between user, user groups and object with security attributes. Security attributes are usually read, write, destroy, and execute.

[0389]

Column Name	Data Type	Null Option	PK	FK
Group—ID	Int	NOT NULL	Yes	Yes
Object—GID	Uniqueidentifier	NOT NULL	Yes	Yes
RelType—ID	Int	NOT NULL	Yes	Yes
Resolved—Flag	Nvarchar(1)	NOT NULL	Yes	No
SecAttr—ID	Int	NOT NULL	Yes	Yes
User—ID	Int	NOT NULL	Yes	Yes

Resolved Flag

[0390] This flag is used to indicate that the security profile is a generated one to improve the performance of the system.

Table: RelType

[0391] The relationship type table describes the type of relationships object can have with users or user groups. These types are treated like roles where the user or group can perform specific function(s).

[0392]

	Column Name	Data Type	Null Option	PK	FK
	Des	Nvarchar(60)	NOT NULL	No	No
	Int—Code	Nvarchar(30)	NOT NULL	No	No
	RelType—ID	Int	IDENTITY	Yes	No

Table: Schedule

[0393] The schedule table details the schedules for a given object. This schedule provides weekly setting with start and stop times.

[0394]

Column Name	Data Type	Null Option	PK	FK
Fri—Flag	Nvarchar(1)	NULL	No	No
Mon—Flag	Nvarchar(1)	NULL	No	No
Object—GID	Uniqueidentifier	NOT NULL	Yes	Yes
Sat—Flag	Nvarchar(1)	NULL	No	No
StartDate	Nvarchar(1)	NOT NULL	No	No
StartTime	Datetime	NOT NULL	No	No
StopDate	Nvarchar(1)	NOT NULL	No	No
StopTime	Datetime	NOT NULL	No	No
Sun—Flag	Nvarchar(1)	NULL	No	No
Thr—Flag	Nvarchar(1)	NULL	No	No
Tue—Flag	Nvarchar(1)	NULL	No	No
Wed—Flag	Nvarchar(1)	NULL	No	No

Start & Stop Date

[0395] These dates define when the schedule is active. The values are inclusive.

Start & Stop Time

[0396] These time fields define when the schedule is active and the values are inclusive.

Sun->Sat Flags

[0397] These flags indicate which day the schedule is active.

Table: SecAttr

[0398] The security attribute table details the types of security the system supports. Currently it is read, write, destroy, and execute.

[0399]

	Column Name	Data Type	Null Option	PK	FK
	Des	Nvarchar(60)	NOT NULL	No	No
	Int—Code	Nvarchar(30)	NOT NULL	No	No
	SecAttr—ID	Int	IDENTITY	Yes	No

Table: TroubleTicket

[0400] The trouble ticket table stores all the current and completed work requests the system or support personnel make. It is intended that this sub system not be used if Network is integrated with a customer trouble ticketing system. This table is also used in the current work display to show what outstanding work is being done and when it will be completed.

[0401]

Column Name	Data Type	Null Option	PK	FK
AssignedToUser—I	Int	NOT NULL	No	Yes
CreatedDate	Datetime	NOT NULL	No	No
Des	Nvarchar(60)	NOT NULL	No	No
EtaDate	Datetime	NULL	No	No
ME—ID	Uniqueidentifier	NULL	No	Yes
Priority	Int	NULL	No	No
ResolvedDate	Datetime	NULL	No	No
TroubleTicketNum	Uniqueidentifier	NOT NULL	Yes	No

Created Date

[0402] This column store the date when the trouble ticket was created.

ETA Date

[0403] This column may store the estimated date when the trouble will be fixed.

Priority

[0404] The priority filed store a value that indicates how important the work ticket is. The higher the number is, the higher the priority.

Resolved Date

[0405] This column stores the date when the ticket is resolved and closed.

Stored Procedures

Parameter Table

[0406] This section details the stored procedures that support the Network database. Included is information about the parameters and what the stored procedure is used for. The following describes the information provided for each stored procedures.

Parameter

[0407] The parameter name is listed and when the SP is executed an “@” symbol should be prefixed. e.g. AddGroup @DES=‘Group Name’, @URL=‘URL name’ The parameters to the “AddGroup” SP are DES and URL.

Data Type

[0408] See the data type definition table above.

Default

[0409] This column will list the default value if any that will be used if the parameter is not specified.

Required (Req)

[0410] This column will detail if the parameter is required and must be specified when the stored procedure is called.

Linked

[0411] The linked column will display the name of the parameter, which can be specified instead if the information is not available in the specific format. This is normally used when the ID of a row can be supplied by another type of key. For example a specific ME can be referenced by its ID or URL. The SP can accept either of these values but not both. If both parameters are specified the ID is only used.

Stored Procedures

[0412] The following procedures are currently used as a primary interface to the Network database.

Add Group

[0413] The stored procedure “AddGroup” is used to add a group to the database.

[0414]

Parameter	Data Type	Default	Req	Linked
DES	DES—Type,		Yes
URL	URL—Type	NULL
PARENT—GID	GUID—ID	NULL		PARENT—URL
PARENT—URL	URL—Type	NULL		PARENT—GID

Description (DES)

[0415] The description parameter is used to name the group and will be displayed in the snap-in.

URL

[0416] This is a unique name for the group and if this parameter is not specified the value specified in the “DES” parameter will be used.

Parent (GID or URL)

[0417] These parameters allow the option of linking the new group to a parent ME.

Add ME

[0418] The stored proce1dure “AddMe” is used to add a new ME to the database.

[0419]

Parameter	Data Type	Default	Req	Linked
METYPE	IntCode—Type,		Yes
DES	DES—Type,		Yes
ICONNAME	Name—Type	NULL
NETNAME	Name—Type	NULL
INSTANCENAME	Name—Type	NULL
URL	URL—Type	NULL

ME Type (METYPE)

[0420] This parameter requires the ME type for the new ME. See the default values listed in the next section.

Description (DES)

[0421] This parameter is used to provide a description of the ME and will be displayed in the snap-in.

ICON Name (ICONNAME)

[0422] This parameter is used to supply the name of the ICON to be displayed by the snap-in ME control. If no value is specified then a default ICON will be displayed.

Network Name (NETNAME)

[0423] Use this parameter to specify the network name of the server if the ME represents a physical node that can be resolved via a DSN or WINS. ME types for servers and workstation will require this information.

Instance Name (INSTANCENAME)

[0424] Several services like SQL Server can have multiple instances running on a server. This value is used to specifically address a specific instance. If this value is NULL then the default instance is assumed.

URL

[0425] This parameter supplies a unique name for the ME. If this value is NULL then the ME cannot be looked up via a URL code and must be referenced by its ID.

Add ME Child

[0426] The stored procedure “AddMeChild” creates a relationship between a parent ME and a child. It also allows specific properties to be created for the relationship.

[0427]

Parameter	Data Type	Default	Req	Linked
PARENT—URL	URL—Type	NULL	PARENT—GID
PARENT—GID	GUID—ID	NULL	PARENT—URL
CHILD—URL	URL—Type	NULL	CHILD—GID
CHILD—GID	GUID—ID	NULL	CHILD—URL
USES—FLAG	FLAG	“N”
TWOWAY—FLAG	FLAG	“N”

Parent(PARENT_—URL or PARENT_—GID)

[0428] Use either of these parameters to specify the parent ME of the relationship. One or the other is required in the present embodiment.

Child (CHILD_—URL or CHILD_—GID)

[0429] Use either of these parameters to specify the child ME of the relationship. One or the other is required.

Uses Flag (USES_—FLAG)

[0430] This parameter specifies that parent is dependent on the child based on resources.

Two Way Flag

[0431] This parameter is used to indicate that the relationship is two way, the parent and child depend on each other. This is to mean that if one of the ME has a problem it will affect the other.

Add Server

[0432] The stored procedure “AddServer” is used to add a server ME to the database.

[0433]

	Parameter	Data Type	Default	Req	Linked
	NETNAME	NAME—Type,		Yes
	DES	DES—Type	NULL
	SVRTYPE	IntCode—Type,		Yes
	SUBSITE	NAME—TYPE	NULL
	GROUP	URL—Type	NULL

Network Name (NETNAME)

[0434] This parameter may be used to specify the network name of the server.

Description (DES)

[0435] This parameter may provide the description for the server. If a description is not specified the network name is used.

Server Type (SVRTYPE)

[0436] Specify the internal code for the ME Type for the server. See the next section for a list of internal codes.

Sub Site (SUBSITE)

[0437] If the server is to be associated with a sub site that is not the default, its name can be specified.

Group

[0438] If the server is to be associated with a ME group then its URL can be specified.

Add SQL Server

[0439] The stored procedure “AddSqlServer” is used to add a reference to a SQL server to the database. If the server that is host the SQL server is not present in the database then it too will be added.

[0440]

	Parameter	Data Type	Default	Req	Linked
	NETNAME	NAME—Type,		Yes
	DES	DES—Type,		Yes
	SVRTYPE	IntCode—Type	NULL
	SUBSITE	NAME—TYPE	NULL
	INSTANCE	NAME—TYPE	NULL
	GROUP	URL—Type	NULL

Network Name (NETNAME)

[0441] Use this parameter to specify the network name of the server.

Description (DES)

[0442] This parameter will provide the description for the server. If a description is not specified the network name is used.

Server Type (SVRTYPE)

[0443] Specify the internal code for the ME Type for the server. See the next section for a list of internal codes.

Sub Site (SUBSITE)

[0444] If the server is to be associated with a sub site that is not the default, its name can be specified.

Instance

[0445] There can be multiple instance of SQL Server running on the same computer. The instance name can be specified if the reference is not the default.

Group

[0446] If the server is to be associated with a ME group then its URL can be specified.

Add Sub Site

[0447] The stored procedure “AddSubSite” is used to add a sub site to the database. Each sub site name must be unique in the management site which is basically the database.

[0448]

Parameter	Data Type	Default	Req	Linked
NAME	NAME Type		Yes

Name

[0449] This parameter is used to specify the name of the new sub site. It must be unique in the management site.

Assign ME to a Group

[0450] The stored procedure “AssignMeToGroup” will assign a ME to another ME, which is a group type.

[0451]

Parameter	Data Type	Default	Req	Linked
ME URL	URL Type	NULL		ME GID
ME GID	GUID ID	NULL		ME URL
GROUP URL	URL Type	NULL		GROUP GID
GROUP GID	GUID ID	NULL		GROUP URL

[0452] The paramters are the following:

ME (ME_—URL or ME_—GID)

[0453] Specifies the ME that is to be assigned to the group.

Group (GROUP_—URL or GROUP_—GID)

[0454] Specifies the Group ME that represents the group that the ME is to be added as a child.

Assign Server to Sub Site

[0455] The stored procedure “AssignServerToSubSite” will assign a ME that is of a server type to a sub site. This SP is normal used internal by other SP and should not be execute directly.

[0456]

	Parameter	Data Type	Default	Req	Linked
	SERVER GID	GUID ID,		Yes
	SUBSITE	NAME Type	NULL

Server ID (SERVER_—GID)

[0457] This parameter is used to specify the server ME that is to be assigned to the sub site.

Sub Site (SUBSITE)

[0458] This parameter is used to specify the name of the sub site that the server ME is to be assigned to. If this parameter is not specified then the ME is assigned to the default sub site.

Is Group

[0459] The stored procedure “IsGroup” is used to test to see if the referenced ME is of a group type. This is normal used by other stored procedures and should not be called directly.

[0460]

	Parameter	Data Type	Default	Req	Linked
	GROUP GID	GUID ID,	Yes
	RAISE	int	NULL

Group ID (GROUP_—GID)

[0461] Specify the group ME ID that is to be tested.

Raise

[0462] This flag is used to indicate to the stored procedure to raise an exception “RAISERRROR” if the ME is not of the group type.

Return

[0463] This store procedure uses the “RETURN” function to return the test state. The value 0 indicates that the ME is not a group while the value of 1 indicates it is a group.

List ME Children

[0464] The stored procedure “ListMeChildren” will list the MEs that are children to the specified ME.

[0465]

	Parameter	Data Type	Default	Req	Linked
	ME URL	URL Type	NULL		ME GID
	ME GID	GUID ID	NULL		ME URL

ME(ME_—URL or ME_—GID)

[0466] These parameters are used to specify the parent ME's reference. One or the other parameter must be specified.

Output

[0467] The following table describes the output of this stored procedure.

[0468]

Column	Description
Type Int Code	ME Type's internal code
Des	ME's description
IconName	ME's icon name
NetName	ME's network name
InstanceName	ME's instance name
URL	ME's URL
CurrrentState	ME's current state value
Disable Flag	Indicates if the ME is active
Watch Flag	Indicates that the child should appear in the parent's
Watch ICON	This ICON should be used in the parent's ME
ControlData	The ME's control data for the relationship

List ME Per Group

[0469] The stored procedure “ListMePerGroup” will enumerate the ME associated with a group ME.

[0470]

Parameter	Data Type	Default	Req	Linked
GROUP URL	URL Type	NULL		GROUP GID
GROUP GID	GUID ID	NULL		GROUP URL

Group (GROUP_—URL or GROUP_—GID)

[0471] These parameters are used to identify the group to be reported on. One of these parameters must be specified.

Output

[0472] The following table describes the output of this stored procedure.

[0473]

	Column	Description
	ME GID	ME'sID
	Type Int Code	ME's type internal code
	Des	ME's description
	IconName	ME's ICON name
	NetName	ME's network name
	InstanceName	ME's instance name
	URL	ME's URL
	CurrentState	ME's current state value
	Disable Flag	Indicates if the ME is currently active
	ControlData	The ME's control data for the relationship

List Top Groups

[0474] The stored procedure “ListTopGroups” will enumerate the group MEs that have no parent and therefore are the top groups. There are no parameters for this stored procedure.

Output

[0475] The following table describes the output of this stored procedure.

[0476]

	Column	Description
	ME GID	ME's ID
	Des	ME's description
	URL	ME's URL
	IconName	ME's ICON name
	CurrentState	ME's current state value
	Disable Flag	Indicates if the ME is currently active

List Unassigned MEs

[0477] The stored procedure “ListUnassignedME” will return a list of ME that are not associated with any groups. There are no parameters for this stored procedure.

Output

[0478] The following table describes the output of this stored procedure.

[0479]

	Column	Description
	ME GID	ME's ID
	Des	ME's description
	URL	ME's URL
	IconName	ME's ICON name
	InstanceName	ME's instance name
	Type Int Code	ME's type internal code

Locate ME ID

[0480] The stored procedure “LocateMeID” is used by internal stored procedures to return the ME ID if the ME's URL is specified. To make the consumer stored procedures simple this stored procedure will accept both ID and URL and will return the current information.

[0481]

	Parameter	Data Type	Default	Req	Linked
	ME URL	URL Type	NULL		ME GID
	ME GID	GUID ID		Yes	ME URL

ME (ME_—URL or ME_—GID)

[0482] These parameters are used to specify the parent ME's reference. One or the other parameter must be specified. The ME_—GID must have the “OUTPUT” modifier so that the ID can be returned to the calling stored procedure.

Set Configuration

[0483] The stored procedure “SetConfig” will add or change current configuration for the Network database.

[0484]

	Parameter	Data Type	Default	Req	Linked
	Name	NAME Type		Yes
	Value	TextData		Yes

Name

[0485] This parameter is used to specify the name of the configuration and must be unique. If the name already exists in the config table the value will be updated.

Value

[0486] This parameter specifies the value to be stored for the named configuration.

Set ME Control Data

[0487] The stored procedure “SetMeControlData” is used to update existing control data for a ME to ME relationship. This SP is designed to be used by the snap-in to update the its control data for the ME control.

[0488]

Parameter	Date Type	Default	Req	Linked
GROUP URL	URL Type	NULL		GROUP URL
GROUP GID	GUID ID	NULL		GROUP URL
ME URL	URL Type	NULL		ME GID
ME GID	GUID ID	NULL		ME URL
CONTROLDATA	TextData		Yes

Group (GROUP_—URL or GROUP_—GID)

[0489] This parameter Specify the Group ME that represents the group.

ME (ME_—URL or ME_—GID)

[0490] This paramter specifies the ME that the control data is to be updated.

Control Data (CONTROLDATA)

[0491] This parameter specifies the control data for the ME to ME relationship.

[0492] iii. Manager Engine—Database Replication Manager

[0493] FIG. 28 is an overview of the operation of the database replication manager in the present embodiment. The database replication manager must keep all data within the Network databases synchronised across all engines within a site. A site may span only a single domain forest (unless suitable trusts can be established outside of the tree). This will be achieved using a single publisher/subscriber, multiple subscriber model as shown above. The data publisher should resolve all data conflicts. Replication is a background task that occurs automatically. It can also be directly requested by other parts of the Management engine.

[0494] In the presently described embodiment, the replication manager has the following requirements and dependencies: a list of all Management engines within a site is required to maintain replication subscriptions and publications; replication can only transmit a maximum of 6000 bytes per row; for replication to succeed, a domain account will have to be used as the SQL Server Agent service account; and conflict resolution will be based on the “first wins” principal.

[0495] In the presently described embodiment, backup publishers are specified in a similar way to ME assignment backups. Each backup publisher will create a snapshot of the database. The first backup publisher will update this snapshot at regular but infrequent intervals (once per day). All subscribers will use pull subscriptions.

[0496] In the presently described embodiment, the replication manager will create the following dependencies on itself: a public interface to start, stop and pause replication (INetworkControl); a message class and set of messages that are sent via the message switch (the messages will include notifications that replication has started, stopped and (potentially) a percentage complete).

[0497] FIG. 29 illustrates the interaction between the data replication manager and the message switch component in the presently described embodiment. As shown in FIG. 29, all information required by or sent from the Database Replication component will be transmitted via the Message Switch component.

[0498] The component is initialised via the INetworkControl interface. Control of replication is handled by the SQL Server Merge ActiveX control. Initialisation of the publication and subscriptions are handled by stored procedures. Below are classes and methods which may be associated with such procedures.

class INetworkControl

• [0499] {
  • [0500] HRESULT Init(void);
  • [0501] HRESULT Term(void);
  • [0502] HRESULT Pause(BOOL fTrue);
• [0503] };
  Method: Init

[0504] The “Init” method is used to initialise the Database Replication component. It may include the following steps: 1) retrieve the database connection string from the registry; 2) detect SQL Server is started and SQL Server agent is using a domain account. Start the SQL services if necessary; 3) create local callback class; 4) create an instance of MsgSwitchCommand; 5) register callback class; 6) register for message switch messages merge replication ActiveX control; 7) create IDispatch class implementing merge replication events and register; 8) create thread proc Listener; 9) create thread proc Replicate; and 10) create a timer.

Received Messages

[0505]

Msg	Description
8	Engine has stopped
9	Engine has been deleted
10	Engine has been evicted
11	Request replication to start
20	Engine has failed
21	Engine has recovered

Transmitted Messages

[0506]

Msg	Description
7	Engine has started
12	Replication has started
13	Replication has stopped
15	Initialisation has started
16	Initialisation has stopped

Method: Term

[0507] The “Term” method is used to shut down replication in the present1 described embodiment. It may include the following steps: 1) destroy the timer; 2) signal a Stop Listening event; 3) wait for the Listener thread to stop, or terminate it; 4) signal a Stop Replication event; 5) wait for the Replicate thread to stop, otherwise terminate the thread; 6) Release the MsgSwitchCommand instance.; and 7) release the Merge Replication ActiveX control.

Method: Pause

[0508] This method may include the following steps: 1) If the flag=TRUE and currently running: destroy the timer and suspend the Listener thread; 2) If the flag=FALSE and currently paused: create the timer and resume the Listener thread.

Callback Interface

[0509] The call back interface method raises an internal event that is caught on a private thread. Below is code which may be associated with this method.

Interface IMsgRegisterCallback

• [0510] {
  • [0511] HRESULT Notify( );
• [0512] };
  Method: Status

[0513] This method logs any errors that occur during replication.

Method: Notify

[0514] This method may consist of two steps: 1) increment the number of messages waiting counter; and 2) signal the MsgWaiting event.

Private Methods

Method: Listener (Thread)

[0515] This method may consist of the following steps: 1) wait for Stop Listening or MsgWaiting events; 2) If the event raised is Stop Listening, then close the thread; 3) if the event raised is a MsgWaiting event, retrieve all outstanding messages from the message switch and process them in turn; 4) if the engine has stopped, been deleted, evicted or failed, call RemoveEngine with the engine GUID and the message type; 5) if the engine has recovered or started, call AddEngine with the engine GUID; 6) if the message details a new publisher/backup publisher list, call ChangePublisher with the list details; and 7) if the message requests replication to start raise the Start Replication event then decrement the message waiting counter by the number of messages read (if nonzero, resume reading messages) otherwise resume waiting for events.

Method: RemoveEngine

[0516] This method may consist of the following steps: 1) If the engine that has been evicted or failed is ourselves: log an error and call Pause with the flag=TRUE; 2)If the engine that has been deleted is ourselves: unsubscribe from the publisher (if not ourselves)—SP DropMCCSubscription and drop the replication—SP DropMCCReplication; 3)if the engine to be removed is the publisher/a backup publisher: the existing primary backup publisher becomes the new publisher, send the resulting publisher/backup publisher list to all engines; and request an election of a new backup publisher; 4) if the engine has been deleted (not stopped/failed/evicted) and we are currently the designated publisher: unsubscribe the engine SP DropMCCSubscriber; and method: AddEngine; 5)if the engine is ourselves: subscribe to the publisher (this may fail as the publisher may not have enabled us yet) and SP AddMCCSubscription; 6)if we are the publisher or the only known engine (first engine): allow the engine as a subscriber—SP AddMCCSubscriber; and 7) if there is no publisher, enable the publication—SP AddMCCReplication, create the initial snapshot and send the list of publisher/backup publishers to the new engine.

Method: ChangePublisher

[0517] This method may have the following steps: 1)if the publisher has changed/was not previously known and is not this engine: unsubscribe any existing subscription (SP DropMCCSubscription), drop any existing replication (if previously a publisher/backup) (SP DropMCCReplication), subscribe to the publisher (SP AddMCCSubscription), and update the publisher/backup publisher list within the database; if the publisher has changed/was not previously known and is this engine: ensure the publication is present (SP AddMCCReplication), ensure the initial snapshot is available, and update the publisher/backup publisher list within the database.

Method: Replicate (thread)

[0518] This method may include the following steps: 1) wait for the Stop Replicate and Start Replicate events; 2)if the Stop Replicate event is signalled, exit the thread; if the Start Replicate event is signalled, retrieve the location of the publisher from the database (added by the installation program or already in existence); and 4) if the database location is known or can be assumed (NetworkDB), then a remote query against any existing engine will reveal a publisher's locatio; 5)send a Replication Started or Initialisation Started (if this is the first time replication has been run since the service started) message to the message switch component; 6)start replication; 7)send a Replication Stopped or Initialisation Stopped message to the message switch; and 8)if this is the first time replication has completed successfully since the service started, send an Engine Started message to all engines via the message switch.

Messages

[0519] The table below lists the messages that will be serviced by the Database Replication component. No additional properties are required for the messages.

[0520]

Message
Number	Object ID	Description	Sent To
7	Engine GID	Engine started.	Site
11		Start replication.	Local
12		Replication has started.	Local
13		Replication has stopped.	Local
15	Engine GID	Initialisation started.	Local
16	Engine GID	Initialisation stopped.	Local
?		Announce publisher changes	Site
?		One or more errors have occurred	Local
		(Replication failed).

Replication Database Tables
Replication Articles Table

[0521] This table lists all articles that must be replicated. Any local tables (e.g. replication tables and local configuration information) are not listed within this table.

Replication Control Table

[0522] This table lists all the publisher and backup publishers.

Security Requirements

[0523] A domain account is required for the SQL Server agent to run under. If Windo1ws NT integrated security is used, this account can also log into the Network database.

[0524] A method for backing up and restoring the Network database is required as well as a method to perform general database management. Potentially an MMC snap-in is required if MSDE is to be used.

Process Flow for the Replication Manager

[0525] FIG. 30 shows a network topology illustrating a replication manager schema.

Overview of Process Flow when the Engine is Installed.

[0526] First, an engine is added to the sub-site and the location of the publisher is either specified during installation or retrieved from the active directory along with a list of all engines currently within the site.

[0527] Next, if subscriptions are not anonymous, a remote query is performed against the publisher by the new engine to enable the engine as a subscriber. Once enabled, the new engine subscribes to the publisher and starts merge replication.

[0528] Next, if the database has not been initialised before, the initial snapshot is sent to the new engine. All changes that have occurred since the snapshot was taken are also transmitted.

[0529] Next, if the database contains existing data that is older than the allowed retention period (currently 7 days), the database is reinitialised as above. Any existing data modifications will be lost. Otherwise, if the database contains existing data that is not older than the allowed retention period and the publisher has not changed, all changes are merged with the publisher.

Replication Manager Process Flow when a New Engine Appears

[0530] Briefly, the engine should already be subscribed. If there are less than three backup publishers, an election is held and the engine enabled as a backup publisher. Replication Manager Process Flow When an engine stops or is deleted.

[0531] Briefly, the publisher will remove the engine's subscription from within its database. If the engine was assigned as a backup to the publisher, a new backup engine will be elected.

Replication Manager Process Flow when an Engine Fails

[0532] The publisher will not remove the engine's subscription until the subscription has expired. If the publisher does not fail, the failed engine can be merged successfully within the allowed subscription expiry period (currently 7 days). A scheduled job can be used to clean up dated subscriptions (this may be provided by SQL Server 2000?).

Publisher Processes

Publisher Process Flow Where the publisher/distributor fails

[0533] In this situation, in the presently described embodiment, it is not possible for merge replication to take place across any of the boxes as all changes pass via the central publisher. The primary backup publisher will re-establish all subscriptions and push down the initial snapshot plus changes.

Publisher Creation Process Flow

[0534] A single publisher is created within the site. All other engines within the site must subscribe to this publisher. A backup publisher, also a subscriber to the main publisher, is also defined. All databases are loosely consistent. In a presently described embodiment and preferred embodiment, the exact implementation will be as defined by SQL Server 2000. Publisher/backup publisher selection is done via the election manager, but should also be user specifiable.

Process Flow in the Event of Publisher Conflicts (More than One Publisher)

[0535] Briefly, if more than one publisher exists, one publisher must stand down and be overwritten. The decision on which publisher wins will be made by the election component. Once elected, the publisher announces itself. Existing subscriptions are unchanged. However, all subscriptions to the other publisher and the other publisher itself drop replication and subscribe to the elected publisher. These engines are all receive reinitialised.

Process Flow for Scheduling Replication Updates

[0536] Modification of any rows within the database will be cached within the distributors and sent at regular intervals (user-definable). The granularity on this is one minute if SQL Server 7.0 is used for the scheduling—awaiting SQL Server 2000. If a separate timer is used, it may be possible to further reduce the delay between each automatic replication. Additionally, SQL Server 2000 supports remote pull agents—further information required. A second schedule defaulting to one hour (arbitrary figure, exact implementation may vary) is used to maintain an updated copy of the initial snapshot on the publisher.

Database Processes

[0537] Database schema updates Process Flow.

[0538] In a preferred embodiment, database schema processes are implemented in accordance with SQL Server 2000.

SQL Server Service Packs Process Flow

[0539] Any changes made to Merge Replication will have some effect on Network. This should not break the system, but will probably require the engine to stop for an unspecified period of time—this should not cause the engine to request the initial snapshot again assuming the engine is stopped gracefully (and not just unplugged).

Modifications to Stored Procedures Process Flow

[0540] In a preferred embodiment, modifications to stored procedures are accomplished via Microsoft SQL Server 2000. All Stored Procedure updates will either be packaged with the merge replication, in a separate snapshot or inside of a separate table. If a snapshot is used, a check of the snapshot must be made each time merge replication occurs. If a table is used, the Stored Procedure's within the table must be dropped and recreated when changes occur.

Network Failure Process Flow

[0541] Failures within the network structure can cause sub-site and/or site fragmentation. Resolution of the subsequent multiple master and multiple publisher scenario is resolved using the election manager. The ‘losing’ site is overwritten.

[0542] iv. Manager Engine—Managed Element Assignment Manager

[0543] The Managed Element Assignment Manager (“MEAM)” is responsible for the control of the monitoring configuration of Managed Elements (“ME's”). The Managed Element Assignment Manager does this by the dynamic assignment of an ME to a Monitoring Engine. In doing this it has two functions: implementing High Availability (HA) by the use of Fail Over (FO) through the technique of assigning standby engines to monitor an Managed Element (ME); and Dynamic Load Balancing (DLB) through run time reassignment of the primary and standby monitors (engines). Below are Manager Engine components required for proper functioning of the MEAM in accordance with an embodiment of the present invention.

Engine Monitoring Manager—EMM

[0544] The MEAM requires the list of operational Monitoring Engines in the system. The EMM maintains and publishes this list along with change events for new and deleted, running and shutdown Monitoring Engines.

Database Replication Manager—DBRM

[0545] The MEAM must be informed of the commencement/completion of a local replication event.

Election Manager—ELM—

[0546] The ELM is responsible for the holding of elections on the behalf of other components. The MEAM requires election on a SubSite basis and must therefore send the ELM a message on startup to inform it of the MEAM's requirements. The ELM will send the MEAM a promotion/demotion message on the event of it being elected or deposed.

[0547] In addition, for MEAM of the presently described embodiment to function properly, all of the following configuration items must be stored in the Network database table “Config” as Name/Value pairs.

[0548] Allowable % Deviation (Δ) Item

[0549]

	Name	Default Value
	TideMark	5

[0550] This is the figure within which deviation from the mean engine load is allowed without being considered as either excess load or spare capacity and therefore subject to load balancing. The deviation above the mean is called the High Tide Mark (HTM) and the deviation below the mean is called the Low Tide Mark (LTM).

Maximum % Deviation (Γ) Item

[0551]

	Name	Default Value
	WaterMark	15

[0552] This is the figure within which deviation from the mean engine load is allowed if no corresponding capacity is detected (outside of the allowable deviation). For example take the situation where the allowable deviation (Δ) is set to 5% and the maximum deviation (Γ) is set to 20%. There are 10 engines running at 46% (all within the allowable deviation) but one engine is running at 69%. Is it worth rebalancing the whole system? By the above scenario the answer is no. If a load balance operation were to execute it would detect 19% excess capacity on one engine, but no spare capacity to move it to. This figure will determine when to force a full load balance. A full load balance is one where all engines loads are taken into account. By reducing the maximum deviation to 10% the above scenario would force a full load balance. The deviation above the mean is called the High Water Mark (HWM) and deviation below the mean is called the Low Water Mark (LWM).

Transfer Block Count (T) Item

[0553]

	Name	Default Value
	MaxTransferBlockCount	0

[0554] When a load balance has been calculated and is ready for execution there will be a number of transfers of assignments between the same machines. Preferably, these are not executed one at a time. The process would be slow and consume maximal amounts of network traffic in setting each connection up and tearing it down again. There will also be the possibility of engines running on the other side of slow lines (modems etc). It therefore makes sense to optimise this process as much as possible. At the other end of the extreme there may be situations where one engine has 100's of ME's to handover. Preferably, these ME's are not sent all at once. An optimisation of each engine will configure each engine such that it transfers multiple blocks at a time up to the transfer block count T.

Start Up

[0555] On start up the MEAM is dependant on having the engine in an active state. This may be dependant on one or several factors but the EMM or the DBRM will notify those concerned of the state change (“Engine Running”). When the MEAM detects that the local engine has gone into active state it is ready to start assuming its membership of its assigned sub-site. At this stage the MEAM will become quiescent.

[0556] Promotion

[0557] It is not until it receives an election “Promotion to Master” message (from the ELM) that the MEAM becomes active. It must then initialise itself as necessary and upon success notify the ELM of its acceptance of the post. Below the the elected (ie., active) MEAM is described.

Demotion

[0558] It is only the elected Master MEAM that can be sent a “Demotion from Master” message. This would then require the MEAM to abandon all processing and become quiescent, listening only for a “Promotion to Master” message (from the ELM). In the presently described embodiment, upon successful demotion the MEAM must notify the ELM of its acceptance of its deposition from the post.

[0559] An ME is activated (monitored) by a client, when that client assigns it a Sub Site. This change is made on the ME table by assigning ME.SubSite_—GID to a known Sub Site in the SubSite table. Through WMI (and the local message switch component) the MEAM will detect this change of status as an ME Add and assign up to four monitors in the presently described embodiment: Primary, Backup1, Backup2, Backup3 depending on the number of engines in the Sub Site. The choice of which engine is assigned to which role is made on the basis of current load. This will facilitate load balancing on the addition of an ME. The MEAM will then save these assignments to the DB which will then, through the mechanism of replication, be passed to all other engines.

[0560] An ME is de-activated or deleted, by a client when that client deletes it via the MMC snap in application, or by natural causes when a transitory monitor job ends (such as a print job). In either case the ME is removed from the system by either deleting it entirely or by removing its Sub Site association. This is done by setting ME.SubSite_—GID=NULL. In either case the MEAM will detect this though a WMI notification from the Message Switch component and where necessary do any clean up (such as assignments). The MEAM will then save this change to the DB, which will then, through the mechanism of replication, be passed to all other engines.

[0561] An engine is added in the sme way it is started. When a new engine is starts up it will announce itself to the Network world that is its Managed Site. This is really the collection of Sub Sites the Network database is aware of. It will instigate a refresh of its local database via replication and when that has completed the engine will be marked as “Engine is Running”. At this point the engine will be fully capable of taking an active part in the Sub Sites it has been assigned to. The Master will take the follwong actions: load balance.

[0562] When an engine is stopped, naturally of abnormally, the EMM will eventually mark the engine as evicted. The action is the same. It disappears from the community. The community is notified of this by each engine's EMM whereby three actions will occur on the Master: Fail Over, Assignment Refresh and Load Balance.

[0563] If the engine being deleted is this engine then immediate termination of all community activity is required. The local EMM will soon stop sending heartbeats and all other engines will take the same action as for the Engine Stopped situation. The MEAM must take whatever housekeeping activities are necessary and cease responding to any events that may still come from the system. When an Engine is deleted the following actions are taken: Step responding to System Events and Do Housekeeping and Clean Up.

[0564] Replication Start: The MEAM will pause when “Replication Start” message is received.

[0565] Replication End is the “Engine is Running” message. The MEAM will start or continue if paused when this message is received.

[0566] The Load Balance Algorithm

[0567] The Master MEAM in the affected Sub Site must recalculate each engine in the Sub Site's load and evenly distribute the assignments. This must be done as efficiently as possible to minimise the impact of any changes. Thus wherever possible the first choice from a re-assignment should be the Backup 1.

[0568] The Assignment Refresh Algorithm

[0569] For all ME's for this Sub Site, the algorithm finds replacement backups to fill the missing entries.

[0570] The Fail Over Algorithm

[0571] For each ME for which the failed engine(s) were the primary engine, make Backup 1 the primary engine and upgrade all other backups (B2->B1, B3->B2).

[0572] The design of the MEAM hinges on the configuration decisions it makes. As such the algorithms and rules it uses to make these decisions are core to its design. Thus, below are decribed some load environment parameters which are defined to optimise the MEAM in a presently preferred embodiment of the present ivention.

[0573] The Mean or Average engine load (AV) of a Sub Site is given by that Sub Sites total commitment divided by the number of active engines in the Sub Site. $\mathrm{AV}=\frac{\sum _1^{x=N}{E}_x·P·\mathrm{Count}}{N}$
where

• [0574] N is the number of engines in this Sub Site.
• [0575] E is an Engine
• [0576] P is an Engines Primary ME List

[0577] The Low Tide Mark (“LTM”) is given by the Sub Site AV—Δ% where Δ is a configurable value TBD. AV is calculated from the total ME count in a Sub Site/Engine count in a Sub Site. In particular, in the presently described embodiment, LTM is defined as $\mathrm{LTM}=\left(1-\frac{\Delta }{100}\right)\frac{\sum _1^{x=N}{E}_x·P·\mathrm{Count}}{N}$
where
Δ is the allowable percentage difference from the mean before an engine can be called out of band. In the case of the low tide mark it is the percentage from the mean below which an engine is defined to have spare capacity.

• [0578] N is the number of engines in this Sub Site.
• [0579] E is an Engine
• [0580] P is an Engines Primary ME List

[0581] The High Tide Mark (“HTM”) is given by the Sub Site AV—Δ% where Δ is a configurable value TBD. AV is calculated from the total ME count in a Sub Site/Engine count in a Sub Site. In particular, in the presently described embodiment, HTM is defined as: $\mathrm{HTM}=\left(1+\frac{\Delta }{100}\right)\frac{\sum _1^{x=N}{E}_x·P·\mathrm{Count}}{N}$
where
Δ is the allowable percentage difference from the mean before an engine can be called out of band. In the case of the high tide mark it is the percentage from the mean above which an engine is defined to have excess load.

• [0582] N is the number of engines in this Sub Site.
• [0583] E is an Engine
• [0584] P is an Engines Primary ME List

[0585] The Low Water Mark(“LWM”) is given by the Sub Site AV—Γ% where Γ is a configurable value TBD. AV is calculated from the total ME count in a Sub Site/Engine count in a Sub Site. In particular, the LWM of the presently described embodiment is defined as $\mathrm{LWM}=\left(1-\frac{\Gamma }{100}\right)\frac{\sum _1^{x=N}{E}_x·P·\mathrm{Count}}{N}$
where
Γ is the allowable percentage difference from the mean before an engine can be called out of limits. In the case of the low water mark it is the percentage from the mean below which an engine is defined to have spare capacity that must be utilised.

• [0586] N is the number of engines in this Sub Site.
• [0587] E is an Engine
• [0588] P is an Engines Primary ME List

[0589] The High Water Mark (“HWM”) is given by the Sub Site AV—Γ% where Γ is a configurable value TBD. AV is calculated from the total ME count in a Sub Site/Engine count in a Sub Site. In particular, in the presently described embodiment, the HWM is defined as $\mathrm{HWM}=\left(1+\frac{\Gamma }{100}\right)\frac{\sum _1^{x=N}{E}_x·P·\mathrm{Count}}{N}$
where
Γ is the allowable percentage difference from the mean before an engine can be called out of limits. In the case of the high tide mark it is the percentage from the mean above which an engine is defined to have excess load that must be distributed to other less utilised engines.

• [0590] N is the number of engines in this Sub Site.
• [0591] E is an Engine
• [0592] P is an Engines Primary ME List
  MEAM Component Procedures
  ME Add procedure

[0593] With respect to the ME Add procedure, when a client adds a new, or activates an unmonitored ME by assigning it to a Sub Site the MEAM Master in the Sub Site it has been added to will receive notification of the event. At this stage it has no assignments other than Sub Site and the MEAM Master is then responsible for assigning P, B1, B2 and B3 (notwithstanding the fact that there might not be sufficient engines to fully support all possible assignments). It will then save the assignment to the local database. Replication will take care of synchronising the rest of the community.

[0594] The following is pseudocode for the ME Add procedure:

• [0595] //
• [0596] // This algorithm is run from the Master.
• [0597] //
• [0598] MEAdd(ME)
• [0599] BEGIN
• [0600] Build Engines list from Sub Site this ME has been added
• [0601] If Engines.Count>0
• [0602] Sort Engines on Engines.P.Count in ASC
• [0603] ME.P=Engines.Item(0)
• [0604] Engines.Remove(0)
• [0605] EndifIf Engines.Count>0
• [0606] Sort Engines on Engines.B1.Count in ASC
• [0607] ME.B 1=Engines.Item(0)
• [0608] Engines.Remove(0)
• [0609] Endif
• [0610] If Engines.Count>0
• [0611] Sort Engines on Engines.B2.Count in ASC
• [0612] ME.B2=Engines.Item(0)
• [0613] Engines.Remove(0)
• [0614] Endif
• [0615] If Engines.Count>0
• [0616] Sort Engines on Engines.B3.Count in ASC
• [0617] ME.B3=Engines.Item(0)
• [0618] Engines.Remove(0)
• [0619] Endif
• [0620] Save to database
• [0621] END
  ME Delete Procedure

[0622] With respect to the ME Delete procedure, when a client deletes or de-activates an monitored ME by removing it from a Sub Site one or more engines in the Sub Site it has been deleted from will receive notification of the event. If the engine receiving the notification is not the primary P then it must notify the assigned primary of the deletion. This might mean the target gets multiple messages to the same effect. Maybe the target also got the original event. One way or the other it gets notified of the deletion event and must delete all entries from the local database. Replication will take care of synchronising the rest of the community. Due however to the fact that all other engines are subservient and inactive with respect to the ME it means there will be no merge contention at the subscriber(s). It may however be necessary to notify the engine assigned to B1 of the deletion if B1's are to passively monitor Me's. Should the deletion cause the local engine primary count for this Sub Site to fall below the Low Tide Mark the MEAM must send a message to all of the other MEAM's in the Sub Site to execute a Load Balance.

[0623] The following is Pseudocode for the MD Delete procedure.

• [0624] //
• [0625] // This algorithm is run from the MEAM Master
• [0626] //
• [0627] MEDelete(ME)
• [0628] BEGIN
• [0629] PrimaryEngine=ME.P
• [0630] Set ME.P=NULL
• [0631] Set ME.B I=NULL
• [0632] Set ME.B2=NULL
• [0633] Set ME.B3=NULL
• [0634] Save to database
• [0635] If Engines.Item(“PrimaryEngine”).P.Count<LWM
• [0636] LoadBalanceSubSite(ThisSubSite)
• [0637] Else If Engines.Item(“PrimaryEngine”).P.Count<LTM
• [0638] LoadBalanceSubSite(ThisSubSite)
• [0639] Endif
• [0640] END
  Engine Added Procedure

[0641] With respect to the Engine Added Procedure, it is the same as the Engine Started procedure in the presently described emboidment. See the Engine Started Procedure below.

Engine Deleted Procedure

[0642] With respect to the Engine Deleted Procedure, the following is pseudocode for the procedure.

• [0643] // Called when an engine or engines deleted message is received.
• [0644] // This algorithm is run from the MEAM Master
• [0645] //
• [0646] OnEngineDeleted(DeletedEngines)
• [0647] BEGIN
• [0648] // If NULL !=DeletedEngines.Item(ThisEngine)
  • [0649] // We've been deleted
  • [0650] //
  • [0651] Stop MEAM by calling IAvalaonControl->Term( );
  • [0652] Endif
• [0653] END
  Engine Started PROCEDURE
• [0654] The following is pseudocode for the Engine Started Procedure
• [0655] // Called by startup code on local machine and by EngineAdded and EngineStarted
• [0656] // handlers. It presumes also that local handlers are also in
• [0657] // effect
• [0658] OnEngineStart(StartedEngine)
• [0659] BEGIN
  • [0660] If StartedEngine==ThisEngine
  • [0661] WaitFor(EngineActiveEvent)
• [0662] Endif
  • [0663] LoadBalanceSubSite(ThisSubSite)
• [0664] END
  Engine Stopped Procedure

[0665] The following is pseudocode for the Engine Stopped procedure.

• [0666] //
• [0667] // Only have consider other engines here
• [0668] //
• [0669] OnEngineStopped(DeadEngines)
• [0670] BEGIN
  • [0671] //
  • [0672] // Fail Over
  • [0673] //
  • [0674] FailOver(DeadEngines)
  • [0675] //
  • [0676] // Reassignment
  • [0677] //
  • [0678] RegenerateAssignments(DeadEngines)
  • [0679] //
  • [0680] // Load Balance
  • [0681] //
  • [0682] LoadBalanceSubSite(ThisSubSite)
• [0683] END
  Failover Procedure

[0684] With respect to the Failover Procedure, when one or more engines fail (i.e., exceed the eviction time limit) two things must happen. All of the failed engine(s) ME's must be failed over to the backups (i.e., these are the engines that tool over monitoring when the engine originally went off line). This moves the assignment chain up leaving one, or more, if multiple engines went down, empty slots at the end. Of course if two engines go down the situation can (and will) arise where both the primary and the backup are unavailable. In this situation the first available backup must take over as primary and move the chain below it (if any) upwards.

[0685] The followng is pseudocode for the failovre procedure:

• [0686] //
• [0687] // This algorithm is run by the MEAM Master
• [0688] //
• [0689] FailOver(DeadEngines)
• [0690] BEGIN
  • [0691] For each DeadEngine in DeadEngines
    • [0692] CheckMEsAndFailOver(DeadEngine, DeadEngines)
  • [0693] Next
• [0694] END
• [0695] CheckMEsAndFailOver(DeadEngine, DeadEngines)
• [0696] BEGIN
  • [0697] For each ME in DeadEngine.MEs
    • [0698] //
    • [0699] // We know the primary is dead
    • [0700] //
    • [0701] ME.P=NULL
    • [0702] //
    • [0703] // Check the backups
    • [0704] //
    • [0705] If NULL !=DeadEngines.Item(ME.B1)
      • [0706] ME.B1=NULL
    • [0707] Endif
    • [0708] If NULL !=DeadEngines.Item(ME.B2)
      • [0709] ME.B2=NULL
    • [0710] Endif
    • [0711] If NULL !=DeadEngines.Item(ME.B3)
      • [0712] ME.B3=NULL
    • [0713] Endif
    • [0714] Move all engines in assignment chain up to close gaps
    • [0715] Save to database
  • [0716] Next
• [0717] END
  Assignment Regeneration Procedure

[0718] The second thing that must happen when one or more engines go down is regeneration of the assignment chain. All engines ME's where the failed engine(s) are in the assignment chain must be fixed from the available pool of engines without producing duplicates and be done in a way that produces a balanced fix.

[0719] The following is pseudocode for the assignment regeneration procedure:

• [0720] //
• [0721] // This algorithm is run by the MEAM Master
• [0722] //
• [0723] RegenerateAssignments(DeadEngines)
• [0724] BEGIN
  • [0725] For each DeadEngine in DeadEngines
    • [0726] ValidateAndFixMEsFromFailedEngine(ThisSubSite)
  • [0727] Next
• [0728] END
• [0729] ValidateAndFixMEsFromFailedEngine (SubSite)
• [0730] BEGIN
• [0731] Build List of Engines by PME count in ASC order for SubSite
• [0732] //
• [0733] // Build list of Me's that have one or more missing assignments
• [0734] //
• [0735] Build List of BrokenMEs sorted by highest missing assignment priority and ME_—GID for this SubSite
• [0736] //
• [0737] // Calculate mean engine load . . . do some basic load assignment optimisation
• [0738] // this calculation is based on ALL Me's in the SubSite including those of
• [0739] // the dead engine
• [0740] //
• [0741] AV=Total ME Count for this SubSite/Running Engine Count for this SubSite
• [0742] //
• [0743] // Optimal assignment is all we attempt to do here . . . not load balance
• [0744] //
• [0745] For each Me in BrokenMEs
• [0746] Engines.Refresh
• [0747] If No ME.Primary
  • [0748] Sort Engines by Engine.P.Count in ASC order
  • [0749] ME.Primary=Engines.Item(0)
  • [0750] Increment ME.Primary.P.Count
  • [0751] Engines.Remove(0)
• [0752] Endif
• [0753] If No ME.B1 and Engines.Count>1
  • [0754] Sort Engines by Engine.B1.Count in ASC order
  • [0755] ME.Backup1=Engines.Item(0)
  • [0756] Increment ME.Backup 1.B1 Count
  • [0757] Engines.Remove(0)
• [0758] Endif
• [0759] If No ME.B2 and Running Engine Count>2
  • [0760] Sort Engines by Engine.B2.Count in ASC order
  • [0761] ME.Backup2=Engines.Item(0)
  • [0762] Increment ME.Backup2.B2.Count
  • [0763] Engines.Remove(0)
• [0764] Endif
• [0765] If No ME.B3 and Running Engine Count>3
  • [0766] Sort Engines by Engine.B3.Count in ASC order
  • [0767] ME.Backup3=Engines.Item(0)
  • [0768] Increment ME.Backup3.B3.Count
  • [0769] Engines.Remove(0)
• [0770] Endif
• [0771] Next
• [0772] UpdateDB with BrokenMEs now fixed
• [0773] //
• [0774] // The SE might already be monitoring these
• [0775] //
• [0776] END
  Load Balancing Procedure

[0777] Load balancing is an activity that can arise from a number of reasons. Those reasons can be due to engine or ME events that unbalance the engine load across the SubSite. FIG. 17 depicts load balancing variables which may be used by the Managed Element Assignment Managre component of the Manager Engine of the presently described embodiment of the invention.

[0778] A load balance operation is carried out when a system (Sub Site) has become loaded in such a fashion that one or more machines (engines) are either under or over stressed. To define what this means we need a number of variables that an algorithm can use to determine what actions (if any) to take. These have been defined above.

[0779] In a preferred embodiment, the load balancing algorithm described below is implemented. This particular algorithm makes the following assumptions and rules: First, there is a band around a Sub Sites mean load where load balancing would not be cost effective. This band is called the tidemark (low tidemark LTM and high tidemark HTM). If the engines whose load is within this band were to be included in the load balance the effect on their load would probably be minimal. The load balance algorithm excludes these from the balance. The bandwidth will be configurable. This rule does leave a small hole in that a Sub Site with 11 engines 10 of which are within the tidemark whose level is set at 5% (say) could leave the remaining engine running at 100% with no spare to distribute it to or 0% with no excess to load it from. As such there is a limit to the inequality outside of which a full load balance must be done with all engines in a Sub Site involved. This is band is called the watermark (low watermark LWM and high watermark HWM).

[0780] In the presently described load balancing algorithm, for the particular Sub Site all engines must do the following:

[0781] First, calculate the Sub Site mean engine load. That is, for each engine in this Sub Site get the total ME count it is primary for. That is its total load. The AV is given by the Sub Site total divided by the number of engines in the Sub Site. This will be the ultimate balance figure for all engines although there may be very small differences due to odd numbers but nothing significant. If the Sub Site maximum and minimum engine loads are within the HWM and LWM then discard all engines where the engine load is within the HTM and LTM.

[0782] Next, sort the engines into two lists: Engines whose load is greater than the AV and engines whose load is less than the average. All of the remaining engines will fit into one of these two categories (an engine cannot have excess and spare at the same time).

[0783] Next, sort the engines in the greater than average list, descending, on primary ME count. These will be called the FROM's. Sort the engines in the less than average list, ascending, on primary ME count. These will be called the TO's.

[0784] What we now have is the pre-rquisite for the load balance operation for a subsite—the remaining engines that have spare capacity active in the operation. If there are no engines with spare capacity—there will be no load balance (this must be an optimal load balance as opposed to a full one where there are no engines between the LTM and the LWM). This situation only arises if total engines excess load is less than the allowable tidemark for all of the other engines.

[0785] The optimisation of the algorithm is to find as many Backup 1 (B1) to Primary(P) reassignments as possible. It may be a requirement that all engines also monitor their B1 assignments to speed up fail over. The MEAM Master must now calculate the optimal transfer blocks from a FROM to a TO. It will do this by walking the two lists—crab wise. If the FROM block is larger than the remaining TO capacity it will create a transfer block the size of the TO spare capacity. If the FROM block is smaller than the remaining TO capacity it will create a transfer block equal to the FROM excess. A Transfer block is a collection of ME's that is being reassigned from the one primary to another (that optimally should have been the backup). And so on down the two lists till all of the excess has been assigned to the spare. Once this is done the MEAM Master can save the operation to the Database.

[0786] Given a total of N engines in a Sub Site using this algorithm the maximum number of engine to engine rearrangements in a load balance is given by (N−1) where each rearrangement defines a transfer of 1 or more ME's from one engine to another.

[0787] Turning now to FIG. 18 it depicts a transfer calculation algorithm used by the Managed Element Assignment Manager Component of the Manager Engine in accordance with load balancing algorithm as described above. The following is pseudocode for the load balancing algorithm.

[0788]

PSEUDOCODE
//
// Load Balance Algorithm:
//
// SSEG is the subsite engine group
//
// ECG is the Excess Capacity Group List
// e is the count of engines in the ECG
// SCG is the Spare Capacity Group List
// s is the count of engines in the SCG
//
// TL is the Transfer List where each item is a Transfer item
// TI is a Transfer item where TI is a list of ME's, Engine From (EF) and Engine To (ET)
// T is the count of transfers.
//
// NOTE: given a total of N engines in a Sub Site using this algorithm the maximum T is
given by (N − 1)
// where each T defines a transfer of 1 or more ME's from one engine to another.
//
LoadBalanceSubSite( SubSite )
BEGIN
For each Engine in SubSite.Engines
	TOTALOAD += Engine.MEs.Count
	Next
	AV = TOTALOAD/SubSite.Engines.Count
	HTM = AV * (1 + Δ)
	LTM = AV * (1 − Δ)
	HWM = AV * (1 + Γ)
	LWM = AV * (1 − Γ)
//
// Check engine load ranges
//
EMax = SubSite.Engines maximum ME count/TOTALOAD
EMin = SubSite.Engines minimum ME count/TOTALOAD
If EMax < HWM AND EMin> LWM
//
// Discard all engines whose load is between the
// LTM and the HTM
//
	For each Engine in SubSite.Engines
	If LTM <= Engine.MEs.Count <= HTM
	//
// Discard it
//	SubSite.Engines.Remove( Engine )
	Endif
	Next
	Endif
	//
	// Build ECG and ESG lists - noting an engine cannot be in both lists
	//
	For each Engine in SubSite.Engines
	If Engine.MEs.Count > AV
	Add to ECG
	Else
	Add to ESG
	Endif
	Next
	//
	// Crabwise walk the two lists calculating the transfer
	// block size
	//
	For SCG = 1 to s and ECG = 1 to e
	Create TI and ET = ESG(s) and EF = ECG(e)
	//
	// Compute block (e) on the ECG with block (s) on the SCG
	//
	SpareCount = AV − ESG(s).MEs.count
	ExcessCount = ECG(e).P.Count − AV
	TI.MEs = GetBestMEsForTransfer( SubSite, ECG(e), SCG(s),
MIN( SpareCount, ExcessCount ) )
	If TI.MEs.Count > 0
	TL.Add TI
	Endif
	If ExcessCount > SpareCount
	//
	// Excess > Spare
	// Next spare if we transferred the max across
	//
	Increment s
	Else If ExcessCount == SpareCount
	//
	// Excess == Spare
	// Next Excess and Spare if we transferred exactly
	//
	Increment e and s
	Else
	//
	// Excess < Spare
	// Next excess if transferred less than the max
	//
	Increment e
	Endif
	Loop
VERSION 1: Trickle balance
	While there are some blocks left to transfer
	Start transaction
	Take up to N blocks update from an EF
	Update DB
	On error Rollback Transaction
	Loop
VERSION 2: Block balance
	Start transaction
	Take all blocks from TL
	Update DB
	On error Rollback Transaction
END
//
// This algorithm finds as many ME's for transfer that already have the
// TO engine as Backup1 as possible moves them from the FROM to the TO
// keeps a copy for return
//
// This is an algorithm and therefore does not include checks or error handling
//
MEs GetBestMEsForTransfer( SubSite, From, To, Count )
BEGIN
	//
	// This would be implemented in a much more sophisticated manner
	//
	For each ME in From.MEs
	If ME.Backup1 = To.Name
	Move to MEs
	Endif
	If MEs.Count >= Count
	break
	Endif
	Next
	If MEs.Count < Count
	//
	// Only up to the min of capacity left and capacity available -
	//
	Move (Count − MEs.Count) items from From.MEs to MEs
	Endif
	Copy MEs to To.MEs
	FixAssignmentsOnTransferedMEs( MEs, To)
	Return MEs
END
//
// Rearrages assignments
// This is an algorithm and therefore does not include checks or error handling
//
FixAssignmentsOnTransferedMEs( MEs, To )
BEGIN
	For each ME in MEs
	If ME.B1 == To
	//
	// Wont have to tear down monitors if B1's are to
	// backup monitor
	//
	ME.B1 = ME.P
	Else If ME.B2 == To
	ME.B2 = ME.P
	Else If ME.B3 == To
	ME.B3 = ME.P
	Endif
	ME.P = To
	Next
END
MEAM Assignment Packages

[0789] Turning now to FIG. 19, it depicts Managed Element Assignment Manager Packages. The Managed Element Assignment Manager component of the presently described embodiment consists of 5 packages or modules. Each of these is implemented as a separate COM component with a set of internal interfaces (not necessarily COM) and a set of external COM interfaces. Wherever possible COM interfaces will be custom, dual interfaces only being supported where absolutely necessary.

MEAM Class Structure

[0790] Turning now to FIG. 20, it depicts the Managed Element Manager class structure. The file name is BodMem.dll in a windows based system. It is top level controller component of the Managed Element section. It performs the following: 1) initialisation; 2) termination and cleanup; 3) control of system logic—i.e. what to do when an engine disappears, an ME is added etc.; 4) detection of system events i.e. reception of Message Switch messages; and 5) decision making in respect of state analysis i.e. Engine events outweigh ME events when both exist.

[0791] The following are MEAM classes:

CMEManager

Derived from CComObjectRootEx<CComMultiThreadModel>, CComCoClass<CMEManager, &CLSID_—MEManager>, IMEManager, INetworkControl, IMsgSwitchNotify, IErrorInfo.

Private Properties

IMsgSwitchCommand* _—MessageSwitch

Instance of the Message Switch class.

The MEAM sends the following messages:

[0792]

Msg #	To	Msg	Description
	ELM	Acknowledge	Promotion to or demotion from Master

CMEMessageQManager& _—MessageQManager

[0793] Instance of the CMEMessageQManager class.

CMEMessageQ& _—ElmQ

[0794] Instance of the CMEMessageQ that is specialised for ELM messages.

CMEMessageQ& _—WmiQ

[0795] Instance of the CMEMessageQ that is specialised for WMI messages.

CMEMessageQ& _—EmmQ

[0796] Instance of the CMEMessageQ that is specialised for EMM messages.

CMEMessageQ& _—DbRepQ

[0797] Instance of the CMEMessageQ that is specialised for DbRep messages.

Public Methods

Static DWORD WINAPI CMEManager::MessageHandlerProc (LPVOID Data)

[0798]

	Parameter	Description
	Return	0 if successful
		system HRESULT on error
	Data	Data passed to CreateThread( ) function - will be the
		this pointer of the CMEManager object.

[0799] System thread process function passed to the CreateThread( ) function call in the Init( ) method. This method is the main processing point for the Managed Element Assignment Manager. It makes decisions regarding ME assignments based on the data collected by the Notify method.

NOTE: No exceptions are thrown from this method

HRESULT INetworkControl::Init (DWORD Phase)

[0800]

	Parameter	Description
	Return	S—OK if successful
		System HRESULT on error.
	Phase	The initialisation phase

[0801] This method is called to initialise the MEAM. This method must create all internal resources such as the relevant internal queues by calling the CMEMessageQManager::AddQ( ) method for each queue type required.

[0802]

	Exceptions	Description
	—com—error	With system HRESULT on error

HRESULT INetworkControl::Term (DWORD Phase)

[0803]

	Parameter	Description
	Return	S—OK if successful
	Phase	The de-initialisation phase

[0804] Thi smethod is called to terminate the MEAM.

[0805]

	Exceptions	Description
	—com—error	With system HRESULT on error

HRESULT INetworkControl::Pause (BOOL fTrue)

[0806]

Parameter	Description
Return	S—OK if successfulSystem HRESULT on error.
Ftrue	If TRUE causes the system to halt. All methods will return
	S—FALSE.If FALSE causes the system to resume if
	“Paused”.

[0807] This method is Called to pause or resume the MEAM.

[0808]

	Exceptions	Description
	—com—error	With system HRESULT on error

[0809] This method will suspend the thread function and so should be called with care. If the system is paused while other activity is allowed to occur that could invalidate the MEAM's state the MEAM will need to do a restart.

HRESULT IMsgSwitchNotify::Notify ( )

[0810]

	Parameter	Description
	Return	S—OK if successful
		System HRESULT on error.

[0811] This method is called by the Message Switch to notify the MEAM of an event it has registered for. NOTE: No exceptions are thrown from this method. The MEAM subscribes to the following messages:

[0812]

		Msg
Msg #	From	Description	Action
	ELM	Promote to	Initialise to be Master for SubSite
		Master
	ELM	Demote from	Terminate Master status and release
		Master	all resources and processing and enter
			quiescent state listening only for
			promotion.
	WMI	AddME	Calculate target engine and send it a
			MEAM.AddME message.
	WMI	DeleteME	Calculate target engine and send it a
			MEAM.DeleteME message.
	WMI	EngineAdded	Wait till engine is in the Active state
			(replication ended) if this engine.
			Else ignore
	WMI	EngineDeleted
	EMM	EngineStarted	Causes Load Balancing.
	EMM	EngineStopped	Causes Fail Over, ReAssignment and
			Load Balancing.
	DBRM	Replication
		Start
	DBRM	ReplicationEnd

[0813] This method is responsible for retrieving a message from the Message Switch and placing it in the correct internal queue. It does this by calling the CMEMessageQManager::AddQ( ).

CMEMessageQ

Protected Attributes of the class:

HANDLE _—Arrived

[0814] Event Handle signalled when a new message has arrived.

GUID _—Type

[0815] Type of queue. This is equivalent to the Message Type GUID.

Public Operations of the Class:

HANDLE GetArrivedEvent ( )

[0816]

Parameter	Description
Return	Handle of event to be signalled when a new message arrives.

CMEMessage* Add ( )