Topology for a hierarchy of control plug-ins used in a control system

[0001] The technical field relates generally to generating a topology for a hierarchy of control plug-ins used in a control system.

[0002] Grid computing services, utility-based data centers, and other types of resource-on-demand systems are becomingly increasingly popular as a highly scalable means for utilizing computer resources to meet the computing demands of users. However, managing these resource-on-demand systems is a difficult task that typically requires a significant amount of time and labor and which conventional network management software is not designed to handle.

[0003] Many management tasks are performed manually, which tends to be time consuming and costly. For example, selecting computer resources from a pool of resources to assign to a particular user's computing demands is typically performed manually. The monitoring of the pool of resources may be performed using conventional management tools. However, several hundred system administrators may be required, especially for large resource on demand systems such as large utility data centers, to make resource allocation decisions.

[0004] According to an embodiment, information from control plug-ins organized in a hierarchy is received. The control plug-ins control an allocation of resources for an application. Relationships between the control plug-ins are determined based on the received information, and a topology of the control plug-ins is generated.

[0011] For simplicity and illustrative purposes, the principles of the embodiments are described. However, one of ordinary skill in the art would readily recognize that the same principles are equally applicable to, and can be implemented in, all types of systems, and that any such variations do not depart from the true spirit and scope of the embodiments. Moreover, in the following detailed description, references are made to the accompanying figures, which illustrate specific embodiments. Electrical, mechanical, logical and structural changes may be made to the embodiments without departing from the spirit and scope of the embodiments.

[0012] FIG. 1 illustrates a system 100 according to an embodiment that provides computer resources on demand and is operable to adjust allocated computer resources as needed. The system 100 includes a resource infrastructure 110 and a resource assessment computer system 120. The resource assessment computer system 120 includes a control plug-in 121 that includes a specifiable policy for allocating computer resources in the resource infrastructure 110. The control plug-in 121 may include parameters and a control function. The control plug-in 121 and other control plug-ins, not shown, may be software that is customizable and executed by the resource assessment computer system 120 for implementing different policies. Control plug-ins are described in further detail below. Furthermore, the resource assessment computer system 120 may include one or a plurality of computer systems, such as one or more servers. Each resource assessment computer system may include and run one or more control plug-ins.

[0013] The resource infrastructure 110 includes a pool of computer resources 111, a resource manager computer system 112, and a data monitoring system 113. The resource manager computer system 112 receives instructions for allocating computer resources from the pool of computer resources 111 for a user's computing needs, such as allocating computer resources for a user's application 114. Based on the instructions the resource manager computer system 112 increases or decreases the computer resources allocated for the application 114. The application 114, which is shown in the pool of computer resources 111, is running on a subset of the pool of resources 111 allocated to the application 114 by the resource manager computer system 112. The data monitoring system 113 monitors predetermined metrics of the allocated computer resources as well as the remaining computer resources in the pool and sends monitoring data, which includes measurements of the predetermined metrics, to the resource assessment computer system 120. Examples of the predetermined metrics may include CPU utilization, process queue lengths, IO and network bandwidth, memory utilization and swap rate. If the system 100 is implemented in a data center, the metrics may be collected by data center management software. If application data monitoring is provided, application-level metrics for the application 114 can also be collected, such as transaction rates, response times, numbers of simultaneous sessions, number of users, etc.

[0014] The pool of resources 111 may include computer systems or even components of computer systems. One feature of the system 100 is that the number of resources from the pool of resources 111 allocated to an application, such as the application 114, may change during run-time depending on demand the application is facing. Those resources can be flexed which means their capacity can be varied over time. Examples of those resources may include number of CPUs, number of variable server partitions on computer systems that support CPU partitioning, number of entire servers, or varying CPU allocations for or numbers of virtual machines. Resources from the pool of resources 111 are allocated for the application 114. Allocating may include but is not limited to adding servers into an application environment, configuring the servers to run in the application environment, which may include loading the necessary software, releasing servers and un-configuring the released servers from the application environment. CPUs, virtual machines, server partitions, and other types of resources may be similarly configured or un-configured as needed. The allocated computer resources for the application 114 may be adjusted by the resource manager computer system 112.

[0015] In one embodiment, the resources are comprised of a server group including a plurality of servers of the same type. Server types may include servers for applications such as web servers, database servers, application servers, and servers used for any type of enterprise application. Examples of server types may include IA32 or lp2000. The application 114 executed by the server group may be a horizontally-scalable application. A horizontally-scalable application is an application that runs on a number of servers in the server group simultaneously. Scalable means that the number of servers may be changed over time. Several instances of the application may be run by the server group. For example, an instance of a web application for online shopping may be duplicated serving different users buying goods or services using the web application. During busy shopping seasons when demand to the e-commerce application is increasing, the resource manager computer system 112 may increase the number of servers in the server group as needed to accommodate the heavier workload. The amount of resources allocated to the application 114 may be adjusted for several reasons, including workload, improving resource utilization (e.g., one or more servers are idle), or supporting other applications hosted by other resources from the pool of resources 111. The environment of the system 100 may be any type of resources on demand environment, which may range from grid computing services to data center services where resources are allocated as needed to one or more applications.

[0016] The resource infrastructure 110 contains the pool of computer resources 111 managed by the resource manager computer system 112. The resource manager computer system 112 may be managing several different sets of allocated computer resources for different users running different applications, and the resource assessment computer system 120 may be running several different control plug-ins for each application. A protected domain is formed around the resource infrastructure 110 to prevent unauthorized access to the allocated resources and the resource manager 112, for example, from an unrelated control plug-in, a computer virus, or other unauthorized applications.

[0017] An interface 130 is used to implement a protected domain around the resource infrastructure 110. The interface 130 provides security measures for crossing protected domains, such as a domain for the resource assessment computer system 120 and a domain for the resource infrastructure 110. Examples of the data the flows between the protected domains may include requests for monitoring data, such as from the resource assessment computer system 120, and the monitoring data included in the responses to the requests, such as provided by the data monitoring system 113. Also, instructions for adjusting allocated resources may be sent from the resource assessment computer system 120 to the resource manager computer system 112 via the interface 130. Examples of the security measures provided by the interface 130 may include control plug-in authentication and establishing encrypted communication channels. Control plug-ins may be authenticated using certificates. For example, a control plug-in's certificate is validated by the interface 130. Only certificates that are registered at the interface 130 can be validated. Encrypted communication channels can be created as is known in the art. The interface 130 may include software and/or hardware in the computer assessment system 120 and the resource manager computer system 112. In one example, the Open Grid Services Infrastructure (OGSI) or the Web Service Resource Framework (WSRF) is used to implement the interface 130. OGSI and WSRF are standards that define mechanisms for creating, managing, and exchanging information among entities. The mechanisms specified in these standards may be used to provide one or more of the security measures for the interface 130.

[0018] The system 100 also includes a management system 150 including a topology generator 151. The management system 150 communicates with the control plug-in 121 and other control plug-ins (not shown). The system 100 may include a plurality of control plug-ins organized as a hierarchy, such as shown in FIG. 3. The management system 150 receives relationship information from the control plug-ins and the topology generator generates and maintains a topology of the control plug-ins. The topology may be represented visually by the management system 150, such that a system administrator may quickly identify hierarchal relationships between control plug-ins for implementing or revising control policies. The management system 150 also includes an administrator console 151 for inputting new or revised control policies to the control plug-ins. The management system 150 may also be connected to the data monitoring system 113 via the interface 130 to receive metrics for the pool of computer resources 111. These metrics may be used to determine whether to reallocate computer resources if needed.

[0019] The resource assessment computer system 120, using the policy provided in the control plug-in 121 and the monitoring data, including the metrics from the data monitoring system 113, determines whether the application 114 and the allocated resources executing the application 114 are operating within defined bounds, such as specified in the control plug-in 121. The rules for this assessment is the policy specified in the control plug-in 121. Based on the assessment, the resource assessment computer system 120 may take corrective action, such as instructing the resource manager computer system 112 to increase or decrease the computer resources allocated to the application 114.

[0020] Policies in control plug-ins may be specified through control functions and parameters in the control plug-ins. The control function in a control plug-in may use one or more parameters for assessing allocated resources. In a simple case, a control function in a control plug-in uses upper and lower threshold parameters. For example, parameters of the control plug-in 121 specify an upper and lower threshold for the current load level of a server group allocated for the application 114. When the load level surpasses the upper threshold, the control plug-in 121 makes a decision to flex up, i.e., add more servers into the server group. When the load level falls below the lower threshold, the control plug-in 121 decides to flex down, i.e., release servers from the server group. A third parameter may include a time interval within which load levels have to remain above the upper or below the lower threshold in order to trigger a resource adjustment. The load level may be determined by the metrics measured by the data monitoring system 113 and transmitted to the resource assessment computer system 120. For example, a current load level of a server group may be an aggregate number of metrics for each server, such as CPU load, memory usage, IO activity, and possibly other metrics.

[0021] An example of a control function used in a control plug-in may be described as follows:

• • [0022] F(p1, p2, p3, . . . )→newN with newN=curN (no change)
    • [0023] newN>curN (flex up)
    • [0024] newN<curN (flex down)

[0025] The control function F calculates a new target number of resources (newN) that satisfies desired performance under current conditions based on the parameters (p=p1, p2, p3 . . . ). The newN is the target number of resources, such as servers in a server group, and curN is the current number of resources allocated for an application, such as the current number of servers in a server group for the application. If the target number of servers is equal to the current number of servers, then, for example, there is no change instructed by the resource assessment computer system 120 running the above-described control plug-in. If the target number of servers is less than the current number of server, then the resource assessment computer system instructs the resource manager computer system 112 to flex up. If the target number of servers is greater than the current number of server, then the resource assessment computer system instructs the resource manager computer system 112 to flex down.

[0026] The parameters p used by the control function F may include “operational conditions” in the underlying controlled system, such as the load level in a server group. The parameters p may also describe the “desired conditions” such as boundaries or thresholds within which a control system operates, such as the control system 200 shown in FIG. 2 and described in detail below. Desired conditions also include preferences or constraints to be obeyed by the control system. An example is described below for a set of parameters p1 . . . p7 for a control function.

[0027] An example of a parameter set for operational conditions may include:

• • [0028] p1:LL (current load level in a server group);
  • [0029] p2:curN (current number of servers in a server group).
  • [0030] An example of a parameter set for desired conditions may include:
  • [0031] p3:LPSupp (upper threshold for load per server: (LL/curN), e.g. 80%);
  • [0032] p4:LPSlow (lower threshold for load per server: (LL/curN), e.g. 40%).

[0033] An example of hard constraints within the desired conditions may include:

• • [0034] p5: min.servers (minimal number of servers that must be maintained in the server group);
  • [0035] p6: max servers (maximal number of servers that must be maintained in the server group).

[0036] An example of preferences within the desired conditions may include:

• • [0037] p7: pref_LoadLevelPerServer (preferred load level of a server with: LPSlow≦pref_LoadLevelPerServer≦LPSupp).

[0038] The behavior of a control plug-in can be influenced by the parameter set p1 . . . p7. An example below includes a control function utilizing the parameter set p1 . . . p7. This control function is invoked any time a change in one of its input parameters p1 . . . p7 occurs, which may be reported to the resource assessment computer system 120 as a parameter change event from the data monitoring system 113.

• • [0039] int control Function(LL, curN) //arguments from parameter set p1 . . . p7
  • [0040] {LLPS=LL/curN; //current load level per server
  • [0041] newN=curN; // don't flex when LLPS remains within bounds
  • [0042] if (LLPS>LPSupp or LLPS<LPSupp or surplusControl)
    • [0043] {newN=RoundUp (LL/pref_LoadLevelPerServer);
    • [0044] newN=max(min(newN, max_Servers), min_servers);}
  • [0045] return newN;}

[0046] When the load per server exceeds the upper threshold or falls below the lower threshold, or the parameter set has changed, the new number of servers is calculated by:

• • [0047] newN=LL/pref LoadLevelPerServer.

[0048] The new number of servers is bound to constraints: newN=max(min(newN, max servers), min servers).

[0049] The control plug-ins are customizable. Adjusting any of the parameters describing desired conditions changes the control decisions made by the control plug-in. By providing a new or changing the existing parameter set p1 . . . p7, users, system administrators operators, or systems, such as a server surplus control system, can influence the result of the control function performed by the control plug-in. Also, control plug-ins may comprise software provided by the user, the system administrator, or another system. By using control plug-ins, the system 100 can easily receive and accommodate new policies for allocating the resources 111.

[0050] Furthermore, control plug-ins may be included in a hierarchal control plug-in architecture, where each intermediate layer in the hierarchy receives input that could change a parameter and generates an output, such as a control decision to change a parameter of a lower level control plug-in. At the bottom layer, the control plug-in's may output flex up or flex down instructions.

[0051] FIG. 2 illustrates a structure 200 for control plug-ins organized in a hierarchy in a control system. The structure 200 may be used for a control plug-in at any level of the hierarchy. An example of a control system using a hierarchal control plug-in architecture is described below and shown in FIG. 3.

[0052] The structure 200 includes an input function 220, an assessment function 221, and an adjustment function 222. The input function 220 receives information from lower-level control plug-ins. The input function 220 may also receive monitoring data from the data monitoring system 113 shown in FIG. 1, which can be forwarded in part or all to higher-level control plug-ins depending on whether the higher-level control plug-ins subscribed to receive the monitoring data. This is one form of filtering performed by a control plug-in. That is the input function 220 may determine the metrics in the monitoring data that the higher-level control plug-in subscribed to and sends those metrics to the higher-level control plug-in.

[0053] The assessment control function 221 processes the input information using a control function and determines whether to adjust parameters for a lower-level control plug-in or to instruct the resource manager 112 shown in FIG. 1 to allocate computer resources, such as increasing or decreasing the number of servers in a server group. The assessment function 221 may also send processed information to a higher-level control plug-in, such as whether as the application is performing “in range” or not, or over which periods of time the application has performed “in range” or “out of range”. The range may be related to workload or other factors.

[0054] The adjustment function 222 generates instructions, if needed, for adjusting parameters for the lower-level control plug-in or instructions for the resource manager 111 to adjust computer resources allocated for applications. The assessment control function 221 and the adjustment function 222 may also generate and output event subscription requests. These requests are received by a lower-level control plug-in which notifies the higher-level control plug-in when the event occurs. Examples of event information received may include the number of servers in a server group, a current farm state (active or inactive), the number of servers in the server free pool 212, workload information (CPU utilization, memory utilization, etc.), and others. Events may be categorized by type, such as monitoring and farm life cycle, by topic, such as determined by a computer resource infrastructure, and by origin, such as a particular server group. Events may be subscribed to using these categories.

[0055] In addition to the functions 220-222, the structure 200 includes a data structure 230. The data structure 230 stores relationship and property information for a control plug-in. The relationship information includes a parent reference 231 to a parent control plug-in, which is a control plug-in at a higher level in the hierarchy that is operable to send instructions to the control plug-in. The parent reference 231 includes an identification of the parent control plug-in. The relationship information also includes one or more children references 232, which may include control plug-ins at a lower level in the hierarchy that are operable to receive instructions from the control plug-in. The children references 232 may include an identification for each child control plug-in.

[0056] The data structure 230 also holds property information 233. The property information 233 includes attributes of the control plug-in, such as version, system administrator responsible for the control plug-in, customer associated with the control plug-in, and the like. The property information 233 may be stored as key, value pairs which are values for a plurality of parameters, such as vendor, version, customer, system administrator, and the like. The property information is used for management operations performed by the managements system 150 shown in FIG. 1 for the control system.

[0057] The structure 200 may also include a discovery port 240, a maintenance port 241, and a property port 242. The discovery port 240 receives and subscribes to the relationship information. Instructions, such as getParent( ) and getchildren( ) may be sent to other control plug-ins via the discovery port 240 to get the relationship information. Also, instructions for subscribing to events, such as subscribe (eventDest, eventType), are communicated via the discovery port 240 and are used to subscribe to event notifications issued by control plug-ins to all recipients that have subscribed for those events. These events may be associated with changes that affect the topology, such as when control plug-ins are added or removed. Examples of event types are child change events with subtypes for a child insert event or a child remove event. When a new control plug-in is inserted in the control hierarchy, its parent will trigger a child insert event informing all subscribers about this change. The primary subscriber for this event is a control plug-in related to the added control plug-in, such as a parent, grandparent, or another child. Similarly, when a control plug-in is removed from the hierarchy, its parent will trigger a child remove event to the control plug-ins that subscribed to that type of event. Root nodes of the control hierarchy always exist and thus a topology can always be generated.

[0058] The maintenance port 241 is for sending and receiving information related to generate and maintain the topology. The information is sent to the topology generator 151 via the maintenance port 241 so the topology generator 151 may generate and maintain the topology. Examples of information sent via the maintenance port 241 to the managements system 150 shown in FIG. 1 generating the topology may include setParent(parent), insertChild(child), insertChild(child, afterChild), removeChild(child). For example, after a control plug-in receives information that a new child control plug-in is inserted, such as in response to subscribing to that event, the control plug-in notifies the management system 150 which updates the topology.

[0059] The property port 242 is for sending and receiving the property information 233. Examples of instructions sent via the property port 242 may include setProperty(key, value), getProperty(key), and getKeys( ). The management system 150 may receive property information sent via the property port 242 and store and display the property information if the display is requested by a system administrator.

[0060] The input function 220 may include functions for sending and receiving the relationship information described above with respect to the ports 240-242, which is used to generate and maintain the topology. Also, the assessment function 221 may include functions for generating the data structure 230 holding the information described above.

[0061] The structure 200 may include other ports not shown for transmitting and receiving information to and from control plug-ins in the hierarchy. The information may include instructions transmitted to lower-level control plug-ins for controlling the allocation of computer resources for an application or instructions transmitted to the resource manager 112 shown in FIG. 1. The information may also include monitoring data from the data monitoring system 113 shown in FIG. 1 or event information subscribed to by a higher-level control plug-in. These ports are described in U.S. patent application serial number (TBD, attorney docket no. 200400592-1), entitiled “Hierarchal Control System For Controlling The Allocation Of Computer Resources”, which is incorporated by reference in its entirety.

[0062] A control plug-in may include software and the functions 220-222 of the structure 200 may be implemented as software modules. The ports can be software ports for sending and receiving the information described above.

[0063] FIG. 3 illustrates a hierarchal control system 300, which includes a control system having control plug-ins arranged in a hierarchy. In the example shown in FIG. 3, servers are the pool of computer resources 111 of FIG. 1 that are released from or added to server groups running applications. It will be apparent to one of ordinary skill in the art that other types of resources may also be used. Also, control plug-ins are represented using the functions of the structure 200 shown in FIG. 2. The ports 240-242 and the data structure 230 are not shown but are included for each control plug-in. The data flows of input data to control plug-ins are illustrated using dotted lines and the data flow of output data from control plug-ins are illustrated using solid lines.

[0064] As shown in FIG. 3 and described in detail below, control plug-ins at intermediate levels may receive input data from lower level control plug-ins to make an assessment decision on whether to adjust parameters in a lower-level control plug-in, which may result in adjusting the allocation of servers for a server group.

[0065] The system 300 includes control plug-ins at different levels of the hierarchy. The control plug-ins may be stored and executed by the resource assessment computer system 120 of FIG. 1, which may include multiple servers running one or more of the control plug-ins. In the example shown in FIG. 3, control plug-ins 310a-c at the lowest level of the hierarchy in the system 300 are each responsible for flexing server groups, such as increasing or decreasing the number of servers in the server groups 311a-c based on control functions and parameters in the control plug-ins of the system 300. As the server groups 311a-c are flexed, servers are either released to or acquired from the server free pool 312. The server free pool 312 includes the servers currently not utilized by any server groups. A single server free pool is shown in FIG. 3. However, a server free pool is provided for each application type. For example, web server application types utilize servers from a web server pool, and database application types utilize servers from a database server pool. Also, the server free pool 312 is part of the pool of resources 111 shown in FIG. 1 and is monitored by the data monitoring system 113.

[0066] The input data for the control plug-ins 310a-c may include monitoring data from the data monitoring system 113 shown in FIG. 1 measuring metrics for each server group. The output data may include instructions for flexing a server group if needed.

[0067] One or more server groups running an application are referred to as a farm. For example, the server groups 311a-b run instances of the application A and are in the farm A. The server group 311c is in the farm B and runs instances of the application B. The control plug-ins 320a-b are in an intermediate level of the hierarchy in the system 300 and include control functions for controlling farms. The control plug-ins 320a and 311a-b include control functions for allocating servers for the farm A, and the control plug-ins 320b and 310c include control functions for allocating servers for the farm B. The control plug-ins 320a-b receive data assessment information from respective lower level control plug-ins and may output adjustment data including adjusted parameters for the control functions of the respective lower level control plug-ins. Adjusted parameters may include new parameter values for a parameter set. The adjustment data may include but is not limited to revised thresholds for the parameters for the lower level plug-ins such that unutilized servers in the server free pool 312 are allocated or to re-allocate servers to server groups that need them.

[0068] The control plug-in 320 is at the customer level and includes control functions for administering policies, for example, for customer C. For example, customer C may desire better performance for application A, and thus new parameters may be output to the control plug-in 320a resulting in an increased number of servers being allocated for the server groups 311a-b. The control plug-in 340 is at the data center level and may include control functions for administering data center policies. For example, the servers in the server groups 311a-c and the server free pool 312 may be provided in a data center and subject to the policies of the data center.

[0069] FIG. 4 illustrates the control plug-ins of the system 300 shown in FIG. 3 communicating with the management system 150 shown in FIG. 1 to generate a topology 160 of the control plug-ins. The control plug-ins are organized in the hierarchy as shown. In FIG. 4, the data structure 230 and the ports 240-242 shown in FIG. 2 are shown for each of the control plug-ins.

[0070] The management system 150 includes links 420 to the root control plug-in as well as to the remaining control plug-ins in the hierarchy. The links 420 may be used for communicating relationship and property information to the topology generator 151 to generate and maintain the topology 160.

[0071] For example, via the links 420, the topology generator 151 receives at least one of a reference to a parent and a child control plug-ins for each control plug-in in the hierarchy. This information is used to generate the topology 160. The topology generator 150 also receives event information associated with changes in topology, such as when a control plug-in is added to or removed from the hierarchy or suspended or resumed. This information is used to synchronize changes in the hierarchy with changes in the topology 160. Also, if a control plug-in is removed, the topology generator 151 may automatically remove all the children as well as the node corresponding to the removed control plug-in from the topology 160.

[0072] A system administrator may use the console 152 to inspect the state of the control plug-in hierarchy based on information that is present in the topology 160. The topology generator 151 may generate a visual representation of the topology 160, such as shown in FIG. 4, including nodes and links mirroring the hierarchy of the control plug-ins. This visual representation may be inspected by the system administrator via the console 152 to perform management operations. The system administrator can select a node of the topology 160 and issue a control operation for that node which is propagated to the corresponding control plug-in, such as shown at 410. Examples of such control operations are suspend or resume the control function of the control plug-in or start or stop the control plug-in. The system administrator may also modify parameters or control functions of a control plug-in to implement a new or revised policy.

[0073] FIG. 5 illustrates a flow chart of a method 500 for generating a topology of control plug-ins organized in a hierarchy, according to an embodiment. The method 500 is described with respect to FIGS. 1-4 by way of example and not limitation.

[0074] At step 501, the management system 150 receives information from the control plug-ins organized in a hierarchy. This may include relationship and property information. At step 502, the management system 150 determines relationships between the control plug-ins based on the received information. Relationships may include parent-child relationships. At step 502, the management system 150 generates the topology 160 based on the determined relationships.

[0075] The steps of the method 500 may be implemented as software embedded on a computer readable medium, such as a computer readable medium in the management system 150 of FIG. 1.

[0076] The steps may be embodied by a computer program, which may exist in a variety of forms both active and inactive. For example, they may exist as software program(s) comprised of program instructions in source code, object code, executable code or other formats for performing some of the steps. Any of the above may be embodied on a computer readable medium, which include storage devices and signals, in compressed or uncompressed form. Examples of suitable computer readable storage devices include conventional computer system RAM (random access memory), ROM (read only memory), EPROM (erasable, programmable ROM), EEPROM (electrically erasable, programmable ROM), and magnetic or optical disks or tapes. Examples of computer readable signals, whether modulated using a carrier or not, are signals that a computer system hosting or running the computer program may be configured to access, including signals downloaded through the Internet or other networks. Concrete examples of the foregoing include distribution of the programs on a CD ROM or via Internet download. In a sense, the Internet itself, as an abstract entity, is a computer readable medium. The same is true of computer networks in general. It is therefore to be understood that those functions enumerated below may be performed by any electronic device capable of executing the above-described functions.

[0077] While the embodiments have been described with reference to examples, those skilled in the art will be able to make various modifications to the described embodiments without departing from the true spirit and scope. The terms and descriptions used herein are set forth by way of illustration only and are not meant as limitations. In particular, although the methods have been described by examples, steps of the methods may be performed in different orders than illustrated or simultaneously. Those skilled in the art will recognize that these and other variations are possible within the spirit and scope as defined in the following claims and their equivalents.