Method and system for processing application performance data ouside of monitored applications to limit overhead caused by monitoring

[0001] The present invention claims priority under 35 USC section 119 and based upon provisional application with a Ser. No. of 60/917,947 which was filed on May 15, 2007.

[0002] Present invention is directed to systems for software application performance monitoring, which acquire performance measurement data inside the monitored application and provides a method for aggregation, correlation and evaluation of said performance measurement data outside the monitored application to keep the overhead caused by monitoring related tasks low.

[0003] Current bytecode instrumentation based performance monitoring tools provide performance data at a fine grained level, down to data describing single method calls. Information at this detailed level is a great improvement to find and eliminate performance problems more quickly and efficiently.

[0004] The acquisition of such high-quality information requires the placement of a large quantity of bytecode instrumentations for performance monitoring which are injected into the bytecode of the monitored application. This multitude of monitoring sensors also creates a large amount of measurement data which has to be aggregated and analyzed.

[0005] Current byte code instrumentation based performance monitoring tools perform instrumentation of application bytecode inside the monitored application and also aggregate measurement data within the memory of the monitored application. Although this approach eases interpretation of the measurement data because correlation between acquired measurement data and the sensor that created the data and its context can be easily performed, it creates considerable overhead in terms of processing time and memory consumption.

[0006] Especially capturing of SQL statements and monitoring the execution of said SQL statements creates significant memory overhead, because SQL statements typically consist of long strings and capturing those strings requires a considerable amount of memory. In the worst case, excessive capturing of SQL statements may cause an out of memory error which may lead to a crash of the monitored application.

[0007] Consequently, an alternative approach is required which transfers measurement data out of the monitored application after acquisition to keep memory overhead low and to guarantee that a maximum memory overhead is not exceeded at any time. Additionally, the new approach should allow performing bytecode instrumentation outside of the application to reduce the processing overhead caused by bytecode instrumentation.

[0008] Current approaches which transfer measurement and logging data from a monitored application to a centralized server which aggregates and correlates the measurement data, like distributed logging systems or the monitoring framework of TPTP, a plug-in to the well known eclipse development environment, generate significant overhead in terms of network traffic. The reason for this overhead is redundant context data sent together with measurement data because those approaches retransfer context information required to interpret that measurement data with every captured measurement.

[0009] An alternative approach should also reduce the network overhead generated by transferring the measurement data, by storing context information required to interpret captured measurement data, and reuse this stored context information whenever this is possible instead of retransferring it every time measurement data is sent.

[0010] A method for dynamically injecting an agent into an application may include the steps of connecting a bootstrap agent to the management server from a virtual machine, the management server being external to the virtual machine, initiating connections with the management server, requesting a handshake between the management server and the virtual machine by the bootstrap agent in response to the connections, determining matching in process agent binaries in response to the handshake and installing the in process agent binaries to the application of the management server.

[0011] The step of verifying that the version of the connected bootstrap agent may be supported by the management server, and the step of initiating connections includes the step of verifying that the version of the virtual machine may be supported by the management server.

[0012] The version of the bootstrap agent and the version of the virtual machine may be used to determine the matching in process agent binaries, and the format of the game process agent binaries may be determined.

[0013] The format of the in process agent binaries may be byte code, and the format of the in process agent binaries may be native code.

[0014] The byte code may be a single byte code in process agent binary and the native code may be a single native code in process agent binary.

[0024] The present invention provides a system and method to dynamically inject monitoring bytecode (said monitoring bytecode is further referred to as sensor) into the bytecode of an application. The injection of the sensors is performed outside of the application to keep the overhead caused by monitoring low.

[0025] A method to process application performance data may include the steps of injecting an agent into a monitored application, the agent is capturing byte code the monitored application is going to load; the agent is sending the captured byte code of the monitored application to a monitoring server external to the monitored application. The monitoring server may generate instrumented byte code from the captured byte code by augmenting the captured byte code with sensor byte code and send the instrumented byte code back to the agent. The agent may provide the instrumented byte code to the monitored application, which loads the instrumented byte code instead of the captured byte code. During instrumentation of a specific portion of byte code, metadata describing the instrumented byte code portion, like class name or method name, together with information about the injected sensor byte code may be created, assigned with an unique sensorId, and stored in a sensor metadata repository located at the external monitoring server. The injected sensor byte code may be parameterized in a way that it tags each measurement data item with the unique sensorId that identifies the portion of byte code to which it is deployed.

[0026] When instrumented byte code is executed, one or more sensors may be executed and generate measurement data.

[0027] The measurement data generated by the sensors may be inserted into event records, together with the unique sensorIds of the executed sensors.

[0028] The sensors may write the event record into an event buffer of fixed size. After the event records are stored, execution of byte code of the monitored application is continued, no blocking of the execution occurs.

[0029] The event buffer may be implemented as a ring buffer of fixed size.

[0030] While sensor may write sensors write event records to the ring buffer, the agent may concurrently and asynchronously read event records from the ring buffer and may remove read out event records from the ring buffer.

[0031] If a sensor writes an event record to the ring buffer while it is full, the ring buffer may discard the received event record to guarantee predictable maximum memory overhead.

[0032] The monitoring server may receive the sensor events sent by the agent, and forward them to the event collector which buffers received event records.

[0033] The event correlation module may cyclically poll the event collector, correlate the received sensor events with the sensor metadata identified by the sensorIds of the received sensor events and put the correlated sensor event/sensor metadata pairs into the measurement buffer.

[0034] An analysis module may process the sensor event/sensor metadata pairs and perform various anaylsis tasks based on the data stored in the measurement buffer.

[0035] The measurement data may include sensor data, and the measurement data may include a unique ID.

[0036] The measurement data may include sensor data and a unique ID, and the sensor data and the unique ID is combine to obtain a performance measurement, and the data set of fixed size may include a ring buffer.

[0037] The data set may be adapted to be asynchronous read out, and the sensor data may include counting sensor data.

[0038] The sensor data may include timing sensor data, and the agent may capture the byte code by interrupting the loading of the byte code.

[0039] The method may include the step of extracting metadata from the byte code, and the metadata may be used to select a method.

[0040] The method may include a comparing method, and the method may include a rule-based method.

[0041] An event record may be placed within the database of fixed size, and the monitoring server may collect the event record.

[0042] The monitoring server may compare the sensor ID of the event record with stored sensor ID in order to determine if the event is to be measured, and the event record may be discarded if the sensor ID of the event record does not match the stored sensor ID.

[0043] The method may include the step of bypassing placing the measurement data into the data set if the capacity of the data set has been exceeded.

[0044] The measurement data generated by those sensors is stored in a buffer of fixed size which makes the monitoring caused memory consumption predictable and limits the size of the memory overhead to the size of the buffer. Said buffer is cyclically and asynchronously read out, and the contained measurement data is transferred to a monitoring server outside the application to perform all monitoring related calculation, correlation and analysis tasks. Asynchronous reading of the buffer decouples acquisition of measurement data from its further processing. Decoupling of measurement data acquisition and processing in turn allows processing of the measurement data outside of the monitored application, and prevents that the monitored applications gets blocked by transportation or processing of the measurement data.

[0045] The monitoring related network traffic is reduced by a redundancy less design of the protocol that conveys measurement data from the application to the monitoring server. Data describing the context of placed sensors is extracted during the placement of the sensor and stored at the monitoring server. A unique ID is assigned to each stored context data entity. Measurement data is tagged with this unique ID, which identifies the context data of the sensor that generated the measurement data and thus allows the reconstruction of the context of the measurement data, without repetitively sending sensor context data additionally to measurement data.

[0046] An agent is injected into the monitored application which intercepts loading of bytecode, captures the bytecode and sends the captured bytecode to a monitoring server for instrumentation. The monitoring server inserts sensor byte code into the received bytecode and extracts and stores context information related to the inserted sensors. The instrumented bytecode is sent back to the agent, which forwards it to the application runtime. The application in turn loads the instrumented bytecode instead of the original bytecode.

[0047] If instrumented bytecode is executed, the injected sensors perform measurements and store the acquired measurement data, together with an unique ID that identifies the context of the sensor in a buffer which is provided by the injected agent. The capacity of said buffer is limited to a predefined size to guarantee that memory overhead remains below a certain limit.

[0048] The agent asynchronously and cyclically sends measurement data accumulated in said buffer to the monitoring server. The monitoring server receives the sent measurement data and uses the ID received with the measurement data to find the context information related to the sensor that generated the measurement. Measurement data, together with the sensor context information is used to perform various performance analyses.

[0049] Acquisition of application performance data has always been an important but also difficult task, because efforts to gain said performance data may change the behavior of the monitored application which also impacts the acquired performance data. In the worst case, performance monitoring may cause malfunction or even a crash of the application.

[0050] Bytecode based software platforms like Sun Microsystems' Java or Microsoft's .NET framework provide interfaces to intercept class loading events and to alter the bytecode of the loaded class before it is loaded into memory. Additionally those frameworks enable restricted altering of bytecode that is already loaded into the runtime environment. Open and documented bytecode formats enable analysis of bytecode and selective altering said bytecode. These features enable the creation of monitoring systems that instrument application code with performance monitoring data on the fly, without the need to alter application source code.

[0051] Such systems ease the task of application performance monitoring because relatively little preparation of the monitored application is required. On the other hand, such monitoring systems create significant monitoring related overhead in terms of processing time required to perform bytecode instrumentation. Additionally, storage of acquired measurement data consumes memory inside the monitored application.

[0052] The described exemplary embodiment provides a system that performs instrumentation of application bytecode and aggregation of performance measurement data outside of the monitored application and performs aggregation and all computation of measurement data outside of the monitored application to keep the overhead caused by monitoring low.

[0053] Referring now to FIG. 1, therein an approach is shown using prior art solutions to place sensor code and to aggregate measurement data, as it is used by most current monitoring systems. An instrumentation engine 102 is loaded by the application which injects sensor bytecode 103 into application code. The injected sensors 104 generate measurement data, which is aggregated in a measurement buffer 105 which resides in the process of the monitored application 101. An external monitoring server 106 with an analysis module 107 cyclically polls the measurement buffer 105 to obtain data for performance analyses.

[0054] This approach shows various shortcomings. First, the instrumentation engine which is deployed to the application must provide functionality to parse and analyze bytecode of the application and to inject sensor bytecode into bytecode of the application 101. Code representing this functionality must be deployed to and executed in the monitored application. Additionally, the bytecode representing all sensors must be known by the instrumentation engine and thus must also be deployed to the application. These requirements result in a fat instrumentation engine, which performs all instrumentation tasks within the process of the monitored application 101, generating significant processing overhead.

[0055] The measurement buffer 105 which resides in the memory of the monitored application 101 is another source of overhead caused by this approach because said measurement buffer requires a considerable amount of memory. Additionally, the measurement buffer 105 may cause erroneous behavior of the monitored application 101, up to an application crash, because peak memory consumption of the measurement buffer 105 is not predictable. If the load handled by the application increases, the amount of memory required by the measurement buffer also rises, as the increased load causes more activity of the monitored application which in turn causes more acquired measurement data. The increased memory consumption caused by the measurement buffer may in the worst case lead to a crash of the application, due to an out of memory error.

[0056] Although the monitoring server 106 cyclically reads out and clears the measurement buffer 105, the probability of an application crash caused by a growing measurement buffer 105 can not be excluded. Increasing the polling frequency of the monitoring server 106 may reduce the probability of an application crash, but it can not eliminate the possibility of an application crash caused by monitoring overhead. An alternative solution that limits the memory consumption caused by monitoring processes to a predictable maximum is required.

[0057] Referring now to FIG. 2, therein a block diagram of the present invention is depicted. An agent 202 which is deployed to the application monitors and intercepts loading of bytecode, captures the original bytecode 203 and sends said original bytecode 203 to a monitoring server 215 for instrumentation. The monitoring server 215 forwards the original bytecode 203 to an instrumentation engine 214 which parses the original bytecode 203 to extract metadata from said original bytecode 203 and to inject sensors 205 into the original bytecode 203. Each injected sensor is assigned a unique sensorId 302, and the metadata extracted from the original bytecode is stored within a sensor metadata node 301, which is assigned the same sensorId 302.

[0058] Said sensor metadata node 301, which allows identifying injected sensors and measurement data generated by those sensors, is inserted into a sensor metadata repository 216. The instrumented bytecode 204 is sent back to the agent 202, which forwards it to the virtual machine to finalize loading of bytecode. The instrumented bytecode 204 is loaded instead of the original bytecode 203, and the injected sensors 205 start monitoring application performance.

[0059] The agent 202 additionally provides a fixed size event buffer 207 which may be implemented as a ring buffer, to temporarily buffer measurement data generated by sensors in the application 201. Sensors 205 acquire measurement data 403 and encapsulate it in sensor events 401, together with a sensorId 402, which is later used to reconstruct the context of the sensor 205 that generated the sensor event. The generated sensor event is written 206 into the ring buffer 207 of the agent 202. If the capacity of a fixed size event buffer 207 is reached, new sensor events are discarded; the memory consumption of the fixed size event buffer 207 is not increased. Said ring buffer 207 is cyclically and asynchronous read out 208 by the agent 202 and the buffered sensor events are sent 209 to the monitoring server 215.

[0060] The monitoring server 215 forwards received sensor events to the event collector 210 which forwards the sensor events 401 to the event correlation module 211. The event correlation module uses the sensorIds 403 contained in the received sensor events 401 to correlate measurement data enclosed in said sensor events with sensor metadata stored in the sensor metadata repository 216 to reconstruct the semantic of received measurement data. The correlated measurement data is placed in a measurement buffer 212, which is used by the analysis module 213 to analyze received measurement data.

[0061] FIG. 3 shows a sensor metadata node 301, which stores data describing instrumented sensors. Sensor metadata nodes are stored in the sensor metadata repository and are used to correlate incoming sensor events with the sensors 205 that generated the events. A sensor metadata node 301 contains a sensorId 302, which uniquely identifies an instrumented sensor 205.

[0062] Additionally a sensor metadata node provides sensor metadata 303 which contains but is not limited to a sensor type, identifying the type of the sensor which may e.g. be a timing sensor type, measuring execution times or a counting sensor type, counting the executions of a method or suchlike; the name of the class which contains the sensor; the name of the method the sensor is injected to; the signature of said method; and the line number of the source code where the sensor 205 is placed.

[0063] A sensor metadata node 301 also contains metadata describing the measurements generated by the referred sensor 304, which contains but is not limited to the type of the measurements, and the unit of the measurement.

[0064] An event record 401, which is used to convey measurement data acquired by instrumented sensors 205 from the application 201 to the monitoring server 215, is displayed in FIG. 4. An event record 401 contains the field sensorId 402 which uniquely identifies the sensor 205 that generated the event record and which is used to correlate the measurement data included in the event record 401 with metadata describing the sensor that generated the event record 401. The measurement data 403 of an event record 401 contains the measurement value acquired by an instrumented sensor 205 at a specific measurement. The measurement data contains but is not limited to execution counters, reflecting the number of times a method or function has been executed, captured argument values of a specific method or function invocation, CPU time required for a specific invocation, or execution duration of a specific method or function invocation.

[0065] FIG. 5 depicts the process of instrumenting original bytecode 203 received from the application 201, with sensors 205 and loading the instrumented bytecode 204 instead of the original bytecode 203. Agent 202 and instrumentation engine 214 collaborate to perform the instrumentation of the bytecode.

[0066] FIG. 5a shows the process of capturing original bytecode 203, and requesting instrumentation of the bytecode by the instrumentation engine 214, from the monitoring server 215. When the virtual machine of the application 201 initiates loading of bytecode, the agent 202 intercepts the loading process and captures the original bytecode 203 before it is loaded. The captured original bytecode 203 is sent to the monitoring server 215 for instrumentation. The agent 202 waits until it receives the instrumented bytecode 204 from the monitoring server 215 and forwards the received instrumented bytecode 204 to the bytecode loading process of the virtual machine of the application 201, which loads the instrumented bytecode 204 instead of the original bytecode 203.

[0067] FIG. 5b shows the tasks performed by the monitoring server 215 to instrument sensors 205 into received original bytecode 203. If the monitoring server 215 receives original bytecode 203, it forwards the bytecode to the instrumentation engine 214 for instrumentation. The instrumentation engine 214 first extracts metadata from the received bytecode which is required to identify the class and methods represented by the received original bytecode 203. The extracted metadata is used for selection of sections of the original bytecode which should be instrumented. Selection might be performed by comparing extracted class name and method names with user defined class and method names, or it may use a rule based system which selects methods matching a set of user defined rules.

[0068] Parameterized sensor code is inserted into the bytecode of the selected methods in a subsequent step, which is described in detail in FIG. 6. After the sensors 205 are placed, the instrumented bytecode 204 is sent back to the agent 202.

[0069] The preferred embodiment performs instrumentation of original bytecode 203 during loading of the bytecode, but instrumentation of original bytecode may also be performed on user request by redefining bytecode during application runtime if the underlying virtual machine supports bytecode redefinition.

[0070] The process depicted in FIG. 6 inserts sensor bytecode into received original bytecode and generates information required to correlate received measurement data with the sensor that created the measurement data. Correlation allows to reconstruct the semantic of the received measurement data, like e.g. class or method of the monitored application into which the sensor that was injected. First, a sensor metadata node 301 with a unique sensorId 302 is created. The sensor metadata 303 of the sensor metadata node 301 is initialized with the metadata previously extracted from the received original bytecode 203, as depicted in FIG. 5b. The stored sensor metadata 303 allows to identify the type of the placed sensor, as well as its location within the application 201 bytecode, which is required for later correlation of measurement data created by the sensor. Additionally, metadata describing the measurement data generated by the sensor 304 is stored in the sensor metadata node 301. The initialized sensor metadata node is inserted into the sensor metadata repository. In a subsequent step, the bytecode representing the sensor to instrument is selected and parameterized with the previously created unique sensorId 302. The sensorId 302 is stored within the bytecode of the sensor and used to initialize event records 401 created by the instrumented sensor 205. The sensorId 302, which identifies the sensor metadata node 301 describing the sensor 205 that created the event record 401, allows to correlate received event records 401 with the sensor metadata node 301 which describes the sensor 205 that generated and sent the event records 401. The parameterized sensor bytecode is injected into the received original bytecode 204.

[0071] FIG. 7 schematically shows the execution of a sensor 205. If instrumented bytecode is executed by the application 201, bytecode of one or more instrumented sensors 205 is executed. On execution, a sensor 205 first creates an event record 401 and sets the sensorId 402 of the event record to the value assigned to the sensor during instrumentation, as depicted in FIG. 6. In a subsequent step, the sensor retrieves measurement data and stores it in the previously created event record 401. The initialized event record is placed in the fixed size event buffer 207 provided by the agent 202 and execution of the monitored bytecode continues.

[0072] The process depicted in FIG. 8 shows the sending of event records 401 stored in the fixed size event buffer 207 of the agent 202 to the monitoring server 215. The agent 202 cyclically starts a process that sends event records 401 stored in the fixed size event buffer 207 to the monitoring server 215. Event records 401 which have been sent to the monitoring server 215 are deleted from the event buffer 207 in a subsequent step.

[0073] FIG. 9 depicts the process of correlating received event records 401 with the metadata describing the sensors 205 that created the event records. After an event record 401 is received by the instrumentation server 215, it is inserted into the event collector 210 which forwards the event record to the event correlation module 211. The event correlation module 211 first reads the sensorId 402 of the received event record 401 and queries the sensor metadata repository 216 for a sensor metadata node 301 with a matching sensorId 302. If a matching sensor metadata node 301 is found in the sensor metadata repository 216, a reference to the matching sensor metadata node 301 is added to the received event record 401 and the event record 401 is inserted into the measurement buffer 212. If no matching sensor metadata node 301 is found for a received event record 401, the event record is discarded.

[0074] The analysis module 213 reads the event records 401 stored in the measurement buffer 212 and uses the reference to the matching sensor metadata node 301 added during event correlation to reconstruct the semantic of the analyzed event records.