Method and system for automated analysis of the performance of remote method invocations in multi-tier applications using bytecode instrumentation

[0001] The present invention claims priority under 35 USC section 119 based on provisional application Ser. No. 60/597,576 filed on Dec. 12, 2005.

[0002] The present invention relates to computer software and more particularly to methods, systems, computer program products, and methods of doing business whereby programmatically-generated bytecode insertion is used to perform run-time analysis of performance of remote method invocations.

[0003] Object oriented and bytecode based software development platforms, including Sun Microsystems's Java and Microsoft's NET platform, have gained Inventors: Bernd Greifeneder and Christian Schwarzbauer wide acceptance for developing Internet and Enterprise class software applications. Bytecode based software provides cross-platform and cross-language compatibility and eases the networked integration of software applications.

[0004] The above platforms further provide frameworks, including Sun Microsystems's RMI and Microsoft's NET Remoting, for invoking methods of remote objects and computers transparently. Remote method invocation frameworks hide the complexity and mechanics for preparing request and response messages, serializing arguments and return values, setting up and managing network connections, transporting the message over network links, and dispatching method invocations based on the request message. While the frameworks greatly simplify programming remote method invocations and stow away complexity, programmers frequently oversee costly usage of said remote invocations, causing poor performance and scalability.

[0005] Application performance is often a critical business factor and as such subject to optimization. Remote method invocations can significantly contribute to poor performance. Therefore, monitoring and diagnosing performance of remote method invocations is required to optimize source code, software architecture, and configuration of networked software applications.

[0006] There are several known types of monitoring remote method invocations. One of them is sniffing packets at the network level. Such network sniffing tools see due to their nature applications as a black-box and consequently lack application context information, which is required to relate remote method invocations to application internals. Another limitation of network sniffers is that they cannot alter the remote method invocations message, which prevents adding trace tags to the remote method invocation message. Also network sniffers cannot see remote method invocations if they are sent over encrypted communication channels.

[0007] Another type of monitoring tool is based on remote management protocols, including but not limited to SNMP, JMX, WBEM. Such remote management protocols are used to query aggregated performance information by use of monitoring agents. Monitoring agents require source code modifications for instrumenting the application. Due to the generic nature, performance metrics provided through such management interfaces are aggregated over different types and occurrences of remote method invocations. Furthermore, available performance metrics are pre-built into the application and cannot be changed at run-time. These metrics cannot be associated to particular application transactions that are on-the fly.

[0008] Another known type of monitoring remote method invocations is to enable remote method invocation logging of said application development frameworks. These log messages are intended for diagnosing remote method invocation errors rather than for diagnosing remote method invocation performance. As such, they lack required performance information including but not limited to message size, serialization cost information. Furthermore, log events are restricted to built-in events of the applications runtime platform.

[0009] A manual approach to capturing performance information of remote invocation calls is to add generation of log messages to the application source code. Modifying application source code requires deep programming and performance measurement knowledge, which may not be available in all situations where performance measurement is required. Altering the source code can introduce undesired application defects. Furthermore, access to source code is often not available. Altered source code of applications must be recompiled and redeployed. Redeployment may also require an undesired restart of the application, which in turn may increase application downtime.

[0010] Accordingly, a need exists for overcoming these shortcomings of the prior art.

[0011] The present invention is directed to a system and method for capturing performance metrics of distinct remote method invocations in networked software applications to enable performance diagnosis and performance bottleneck root-cause analysis in production and load test environments. Its objective is to capture the performance metrics for single remote method calls.

[0012] The present invention captures the performance metrics in a manner that provides minimal disruption to the execution characteristics of the application.

[0013] The present invention does not require access to source code for capturing remote method invocation performance metrics from bytecode based software applications.

[0014] Another object of the present invention is to provide techniques for programmatically instrumenting the bytecode of applications during load-time and run-time so that performance metrics about distinct remote method invocations will be captured.

[0015] The above performance metrics preferably include but are not limited to bytes send, bytes received, objects serialized, objects deserialized, objects visited for serialization, objects visited for deserialization, remote method invocation response time at the stub, remote method invocation response time at the dispatcher.

[0016] Instrumenting software applications for the purpose of performance diagnosing remote method invocations using a first method preferably includes: intercepting load requests for original bytecode at run-time; programmatically altering original bytecode by inserting additional bytecode (the altered and added bytecode is further called sensors) to identified methods; loading for each intercepted load request the altered bytecode in place of the original bytecode.

[0017] Alternatively, instrumenting software applications for the purpose of performance diagnosing remote method invocations using a second method preferably includes: loading original bytecode at run-time; programmatically altering loaded original bytecode by inserting additional bytecode to methods that implement remote method invocation interfaces; and to methods that send/receive remote method invocation over network streams; and to methods that serialize/deserialize objects; redefining original bytecode by altered bytecode at run-time.

[0018] When instrumented remote methods are invoked, the altered bytecode (sensors) gets an existing or creates a new remote performance diagnostics thread-local-storage (RPD-TLS) object for the current thread in which the method is running. A thread-local-storage object is a global variable, of which the accessibility scope is restricted to the single thread in which it has been created, allowing multiple threads to create their own copy of the same variable type. The RPD-TLS object preferably includes variables for holding performance metrics; variables for correlating the invoked method with the message send/receive streams, variables for correlating the invoked method with serialization/deserialization streams, variables for control and status information.

[0019] During the execution of the remote method call, the altered bytecode (sensors) collects performance metrics and writes them to the RPD-TLS of the current thread. The RPD-TLS may contain further control and status information that sensors can evaluate and use to change behavior and method call correlation.

[0020] On completion of the remote method invocation, the collected performance metrics are preferably stored for further analysis. Storing performance metrics may include transfer over network connections, writing to file, and storing in memory.

[0021] The technique may further include selectively deactivating performance measurement of one or more instrumented remote method invocations.

[0022] Capturing performance information of remote method invocations may occur on client as well as on the server side of the remote communication.

[0034] Historically, it has been difficult to provide tools to diagnose the root-cause for poor performing remote invocation calls, since most existing tools rely on network based information, which lack an application context. The present invention, on the other hand, discloses a relatively lightweight approach to diagnosing the performance of remote method invocations, which is applicable also in load-testing and production scenarios.

[0035] In the Java platform's distributed object model, a remote object is one whose methods can be invoked from another Java virtual machine, potentially on a different host. An object of this type is described by one or more remote interfaces which are interfaces written in the Java programming language that declare the methods of the remote object.

[0036] Remote method invocation (RMI) is the action of invoking a method of a remote interface on a remote object. Most importantly, a method invocation on a remote object has the same syntax as a method invocation on a local object.

[0037] Referring now to FIG. 1, therein is shown a principle diagram for a remote method invocation using Java RMI between a client application 101 and a server application 106. The client application 101 and server application 106 can be of any bytecode based application type, including but not limited to a rich-client, a web-server, an application server. From the perspective of the application, the remote method invocation is similar to a local invocation. Thus, the client application 101 performs logically a direct method invocation on the remote server application 106. FIG. 1 illustrates further the layers of the client application 101 and server application 106 that are involved into the remote method invocation. Code of the client application 101 transparently invokes a stub method 102 with the same syntax as the actual remote method that will be invoked. The stub 102 serializes method arguments and deserializes method return values by making use of the (de)serialization/(un)marshalling layer 103, which in turn uses the transport layer 104 to send a remote method invocation request containing the method arguments. The request is physically transferred preferably over a TCP/IP network connection 105 to and from the server application. The clients transport layer 104 now blocks the current thread until the server replies with a response or a network failure or timeout occurs. The server application 106 runs a TCP/IP listener 110 capable of accepting incoming remote method invocations request from client application 101. Incoming requests are escalated to the transport layer 109 which handles the request and forwards the request stream to the (de)serialization and (un)marshalling layer 108. The (de)serialization and (un)marshalling layer 108 then reconstructs the method arguments. The dispatcher 107 finally calls the remote method using provided method arguments. The dispatcher 107 waits for the remote method, which is executed each in its own thread within the server application 106. Once the remote method execution completed, the dispatcher 107 makes use of the serializer and marshaller in layer 108 which in turn uses the transport layer 109 to send the response to the client application 101 using the TCP/IP connection 110. The TCP/IP layer 105 of the client application 101 forwards the response to the transport layer 104 which reads the response and deserializes the response data stream in layer 103 to reconstruct the remote methods return value. The above return value is then returned by the stub 102 to the calling client application 101.

[0038] FIG. 1 illustrates further the preferred locations for sensors 111 through 116 that capture remote method invocation performance information for each single remote method invocation call. The sensors may be used at the client application 101, at the server application 106, or concurrently at both the client application 101 and the server application 106. Sensors 111 and 114 are of a remote method invocation type of sensor. Sensors 112 and 115 are of a serialization type sensor. Sensors 113 and 116 are of a transport type sensor. Sensors are blocks of bytecode that are injected into the application layers using preferred instrumentation technique shown in FIG. 3 and FIG. 4.

[0039] The exemplary embodiment describes remoting performance diagnostics using Java RMI/JRMP terminology. The present invention is not limited to Java RMI/JRMP and may be applied to other remote invocation methods that are based on bytecode execution including but not limited to Java Web Services, JMS, NET remoting.

[0040] Client applications 101 may use threads to perform remote method invocations concurrently. Server applications 106 use typically thread pools to accept and dispatch remote method invocations concurrently. Server applications 106 will serialize remote method invocations in case the maximum number of dispatcher threads has been reached. On the server application 106 side, when a client application 101 connects to the server socket, a new thread is forked to deal with the incoming call. The original thread can continue listening to the original socket so that additional calls from other clients can be made.

[0041] The stub 102 hides the serialization of parameters and the network-level communication in order to present a simple invocation mechanism to the caller.

[0042] FIG. 9 illustrates the preferred embodiment of the remoting performance diagnostics thread-local storage (RPD-TLS) 901. The RPD-TLS 901 holds information on a per-thread basis and includes the groups 902 through 905. The remote method meta information 902 can hold meta information that is set by the instrumentation engine during instrumentation time. The above meta information 902 is used to identify a particular remote method invocation of the application. Stream correlation information 903 is used when transport or serialization streams overlap in the current thread during the remote method invocation. The above correlation information 903 is used in this case to identify the correct stream that is associated with the ongoing remote method invocation. Throughput performance metrics 904 include but are not limited to: number of serialized and marshaled objects; number of deserialized and unmarshalled objects; number of objects visited during serialization; number of objects visited during deserialization; number of bytes sent; number of bytes received; remote method invocation time at the stub 102 and at the dispatcher 107. Response time performance metrics 905 include but are not limited to: response time of the stub method invocation; response time of the dispatcher method invocation; elapsed time to serialize and deserialize the method arguments of the remote method call request; elapsed time to serialize and deserialize the method return value or exception of the remote method call response; elapsed exclusive time spent for serialization, excluding stream transport time.

[0043] FIG. 2 shows an exemplary method and system for configuring an instrumentation engine 204 using meta-information 201 through 203. The sensor locations 201 specify the method and code location for placing the sensors by preferably using predefined patterns. The instrumentation method 202 is an algorithm, preferably in the form of a software program, which defines how bytecode alterations have to be performed to correctly inject the sensors into the application layers. The sensor operations 203 define what the injected sensor needs to do in order to capture remote method performance information. The sensor operations preferably include bytecode targeted to the applications platform.

[0044] Referring to FIG. 3, instrumenting a client or server application 301 at load time involves in preferred embodiment following steps as shown in the flowchart of FIG. 10: interception 1002 of the native class loader 304 using an agent 303; capturing 1004 the original bytecode 306 and transferring 1006 the original bytecode 306 to an instrumentation engine 309; instrumenting 1008 the bytecode in the instrumentation engine using meta-information as shown in FIG. 2.; returning 1010 the instrumented bytecode 305 to the agent 303, which in turn forwards 1012 the instrumented bytecode 305 to the application at runtime instead of the original bytecode 306.

[0045] Referring to FIG. 4, instrumenting a client or server application 401 at run-time allows adding or removing the remoting performance diagnostics sensors at run-time, without the need for an application restart. The present invention is architected to support run-time instrumentation using an instrumentation engine 406, which is preferably hosted by an out-of-process instrumentation server 405. The preferred embodiment includes following steps as shown in the flowchart of FIG. 11: the original bytecode 408 is fetched 1102 from the runtime 402 by making use of the agent 403; alternatively, the original bytecode 408 may be retrieved 1104 from a repository 407; in the next step the instrumentation engine 406 instruments 1106 the original bytecode 408 to include sensors as defined in the meta information as shown in FIG. 2; the resulting instrumented bytecode 404 is then sent to the agent 403, which replaces 1108 the original bytecode by the instrumented bytecode.

[0046] Serialization is defined as the process of deconstructing objects with variables and references to other objects and to a serial stream of objects and primitive type values. Deserialization is defined as the reconstruction of a stream of objects and primitive type values to a single object.

[0047] Marshalling is defined as the process of encoding an object and primitive type values to a byte sequence for transport and persistency. Unmarshalling is defined as the process of decoding a byte sequence to objects and primitive type values.

[0048] The sensors 201, 202, 203 that are injected into the original bytecode by the instrumentation engine 204 are can be categorized into three types. The three types of sensors are shown in FIG. 8. The first sensor type is the remote method invocation sensor 803, for which bytecode is placed into all or selected method stubs. Furthermore, the bytecode for this sensor type is also placed into the dispatcher method on the server side. The bytecode for the second sensor type is the object serialization sensor 805, which is placed into the bytecode of methods for object serialization and object deserialization. The third sensor type, the transport sensor 804, is placed into write and read methods of the remote method invocation transport stream bytecode. All three sensor types communicate through the RPD-TLS 901. The remote method invocation sensor type is preferably after completion of a single remote method invocation responsible for transferring the captured performance metrics to a metric processing unit 807 for storage, analysis, and correlation. The metric processing unit 807 may be contained in-process of the application as well as out-of-process in a performance processing server.

[0049] Referring to FIG. 5, the instrumentation engine 204 injects bytecode of the remote method invocation sensors at the beginning of the remote method stub as well as at all exit points. Exit points include: the end of the method, return statements, and exceptions. As shown in FIG. 5a, the remote method invocation sensor bytecode at the beginning of the method call preferably performs following operations:

• • [0050] 1) checks 502 for the existence of the thread local storage variable that holds the RPD-TLS 901. If the RPD-TLS 901 does not yet exist within the current thread, a new one is created 504.
  • [0051] 2) resets 506, 508 the RPD-TLS 901 to initial values, which preferably assigns a remote method identifier, increases a remote method call serial number, clears all remote method call performance metrics that are on a per-call basis.
  • [0052] 3) enable capturing 510 of serialization/deserialization and transport streams by settings associated flags in RPD-TLS 901.
  • [0053] 4) store 512 the current timestamp as method entry timestamp in the RPD-TLS 901.

[0054] The remote method invocation sensor bytecode at the exit points of the methods preferably performs following operations (also shown in FIG. 5b):

• • [0055] 1) checks 522 for the existence of the thread local storage variable that holds the RPD-TLS 901. If the RPD-TLS 901 does not exist, skip remainder of the sensor bytecode.
  • [0056] 2) store 524 the current timestamp as method exit timestamp in the RPD-TLS 901; the difference of the method entry and exit timestamp represent the remote method call response time.
  • [0057] 3) disable 526 capturing of serialization/deserialization and transport streams by clearing associated flags in RPD-TLS 901.
  • [0058] 4) read 528 the performance metrics from RPD-TLS 901 and forward them to a performance metric collection unit, which may store, correlate, analyze, aggregate, externalize these performance metrics.

[0059] The instrumentation engine preferably places further transport sensor bytecode into the remote method invocation transport stream for capturing read bytes and sent bytes. The transport sensor bytecode preferably performs operations shown in FIG. 6 and listed below:

• • [0060] 1) checks 602 for existence of RPD-TLS 901; if RPD-TLS 901 is not available, skip remainder of sensor bytecode.
  • [0061] 2) checks 604 whether capturing flag in RPD-TLS 901 is enabled if the flag is not set, skip remainder of sensor bytecode
  • [0062] 3) checks 606 whether the current transport stream correlates with the remote method invocation by preferably comparing the transport stream object reference with an object reference stored in the RPD-TLS 901; if said stream reference does not match the one of the RPD-TLS 901, skip remainder of sensor bytecode.
  • [0063] 4) captures 608 the number of bytes sent or received, depending whether the sensor bytecode is added to a sending or receiving method;
  • [0064] 5) increments 610 the RPD-TLS transport read/write counters by captured value.
  • [0065] 6) captures 612 the current time at the beginning of the transport stream method, captures the current time at the end of the transport stream method, and increments the elapsed transport time counter in the RPD-TLS by the difference of said timestamps.

[0066] The instrumentation engine 204 preferably places further serialization sensor type bytecode into the object (de)serialization and (un)marshalling methods. The preferred operations this serialization sensor bytecode performs is shown in FIG. 7 and listed below:

• • [0067] 1) checks 702 for existence of RPD-TLS 901 if RPD-TLS 901 is not available, skip remainder of sensor bytecode.
  • [0068] 2) checks 704 whether capturing flag in RPD-TLS 901 is enabled if the flag is not set, skip remainder of sensor bytecode
  • [0069] 3) checks 706 whether the current object serialization/deserialization stream correlates with the remote method invocation by preferably comparing the (de)serialization stream object reference with an object reference stored in the RPD-TLS; if said stream reference does not match the one of the RPD-TLS, skip remainder of sensor bytecode.
  • [0070] 4) captures serialization/deserialization information as follows:
    • [0071] a. if this is a method that (de)serializes an object: insert bytecode that
    • [0072] captures 708 the current time at the beginning of the (de)serialization stream method, captures the current time at the end of the (de)serialization stream method, and increments the elapsed (de)serialization time counter in the RPD-TLS 901 by the difference of said timestamps.
    • [0073] b. determines 710 if this is a method for (un)marshalling objects:
    • [0074] increase 712 number of written/read objects in the RPD-TLS 901.
    • [0075] c. if this is a method that analyzes object references for the need of (un)marshalling during traversal of the object (un)serialization tree: increase 714 the number of visited objects in the RPD-TLS 910.

[0076] The present invention may be further extended to provide metrics about each serialized object for pinpointing the most expensive object of the serialization process.