Testing a component of a distributed system

[0001] Distributed software accounts for a large percentage of software programs produced today. Distributed software is typically implemented as multiple components interacting in a single computer system or in multiple computer systems interconnected by networks both large and small. Rigorous testing is often required in order to produce a robust, high-quality, and reliable distributed software system. However, testing of such distributed software systems remains difficult because the interaction of components with other components throughout the system or over multiple systems is often highly complex. The complexity is due, in large part, to the difficulty in determining if erroneous code lies within the components, within the other components, or within the interactions between the components, especially when the component interacts with a relatively large number of other components.

[0002] Conventionally, each component of the distributed software is individually tested to validate the internal logic of each component, in an effort to produce error-free code for each component. In addition, system integration testing is performed to focus on testing of component interactions only when it is determined that each component contains error-free code. This type of testing, however, is difficult and time consuming. It would therefore be desirable to have a testing system that does not suffer from the drawbacks and difficulties associated with conventional testing schemes.

[0003] A method for testing a component of a distributed system is disclosed herein. In the method, a first invocation request is sent to the component, where the first invocation request includes an identifier. A second invocation request is sent to a second component, where the second invocation request carries the identifier from the first invocation request. In addition, the second component switches behavior based upon the identifier, where the switching of behavior of the second component is employed to test the component.

[0012] For simplicity and illustrative purposes, various examples of systems and methods of the invention are described. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the examples. It will be apparent however, to one of ordinary skill in the art, that the examples may be practiced without limitation to these specific details. In other instances, well known methods and structures have not been described in detail so as not to unnecessarily obscure the examples.

[0013] Systems and methods for efficiently testing the interactions between a component under test and stub components are described herein. More particularly, various methods and systems for testing how the component responds to a variety of conditions as simulated by the stub components are described herein. In this regard, the stub components may be programmed to emulate various behaviors which may occur during a final implementation of the component. For instance, when a call is made from the component, the stub components may respond with one of the programmed behaviors depending upon an identifier appended to the call, as described in greater detail herein below. In one regard, therefore, the component may be tested in a relatively controlled environment and the tests may be performed without a centralized coordinator in the test execution environment.

[0014] As also described in greater detail below, the interactions between the component and the stub components may be tested without requiring additions or modifications to the code contained in the component. As such, the amount of time required to perform the testing may be substantially minimized and the component may be tested under final implementation scenarios.

[0015] With reference first to FIG. 1A, there is shown a block diagram of a system 100 for testing a component 104 of a distributed system. The system 100 includes a test program 102, a component under test 104 (hereinafter “component 104”), a plurality of stub components 106a-106n, and a run-time infrastructure 108. The ellipses (“ . . . ”) between the stub component 106a and the stub component 106n indicate that any reasonably suitable number of stub components 106a-106n may be included in the system 100. In addition, it should be understood that the system 100 may include additional elements and that some of the elements described herein may be removed and/or modified without departing from a scope of the system 100.

[0016] Generally speaking, the test program 102 may comprise a software program that implements test cases, which may belong to a test suite. Test cases may be defined as a set of inputs, execution preconditions, and expected outcomes, developed for a particular objective, such as to exercise a particular program path or to verify compliance with a specific requirement. The test suite may be defined as a collection of one or more test cases for the component 104.

[0017] The stub components 106a-106n may be defined as special-purpose implementations of the software components that the component 104 calls or otherwise is dependent on in a final integrated system. The stub components 106a-106n may emulate the functionalities of the corresponding software components either partially or fully.

[0018] A depiction of a manner in which the elements of the system 100 may interact is provided through the following example. In this example, a user may initialize one or more test cases on the component 104 through operation of the test program 102, in which the component 104 is designed to interact with an e-mail server and a document repository server. Using the examples described herein, the stub components 106a-106n may be programmed to emulate various behaviors of the e-mail server and the document repository server. In this regard, the stub components 106a-106n may be used to emulate a variety of possible responses from the e-mail server and the document repository server without having to modify an actual e-mail server or an actual document repository server.

[0019] The stub components 106a-106n in this example may be programmed to, for instance, not respond, respond with a very large work load, respond after a significant time delay, etc., thus testing how the component 104 reacts to a variety of different conditions. In addition, the interactions between the component 104 and the stub components 106a-106n may be tested without having to communicate remotely with other systems. In this regard, the interactions of the component 104 with remote components may be tested in a controlled environment without requiring the use of remote system resources.

[0020] In the example above and as used throughout the present disclosure, the component 104 may be tested through use of various identifiers. The identifiers may broadly be defined as instructions for the stub components 106a-106n to behave in a predefined manner. In other words, the stub components 106a-106n may be programmed to respond in a predefined manner according to the identifier received by the stub components 106a-106n. In the example above, the stub components 106a-106n may be programmed to not respond, respond with a large work load, etc., according to the respective identifier received by the stub components 106a-106n.

[0021] The identifiers may identify the particular test cases to be run on the component 104 and the stub components 106a-106n. In addition, the identifiers may include one or both of test case identifiers and test suite identifiers. As the names suggest, test case identifiers are identifiers associated with test cases and test suite identifiers are identifiers associated with test suites. When there are multiple test suites, and test cases contained in the multiple test suites, a unique test case may be identified by a particular test case identifier and a particular test suite identifier. In any respect, the test case identifiers and the test suite identifiers may comprise predetermined values chosen to represent certain conditions to be tested.

[0022] The run-time infrastructure 108 may be defined as a dynamically bindable package of one or more programs managed as a unit and accessed through documented interfaces that may be discovered at run-time. In addition, and as described in greater detail herein below, the run-time infrastructure 108 is configured to track interactions between the test program 102, the component 104, and the stub components 106a-106n, by tracking calls and responses between these elements. More particularly, for instance, the run-time infrastructure 108 may track and maintain a sequence of calls made between these elements.

[0023] When the test cases have both test case identifiers and test suite identifiers, the run-time infrastructure 108 may merge the two identifiers into a single identifier that is unique in the system 100, for the calls between the test program 102, the component 104, and the stub components 106a-106n. For example, the test case with the test case identifier of “1” and the test suite identifier of “2” may be merged into a single identifier of “00010002”.

[0024] Correspondingly, when a call reaches the stub components 106a-106n, the run-time infrastructure 108 may perform the reconstruction of the test case identifier and the test suite identifier, such that the stub components 106a-106n may retrieve these two identifiers and switch the stub component 106a-106n behavior based on the received test case identifier and test suite identifier.

[0025] A sequence may contain the list of calls, as well as the ordering of the calls and the inputs and outputs to the calls, in order to produce a call history, which may be used in a debugging process. For instance, it may be determined that a particular test case where the stub component 106a delays in responding for a particular time and the stub component 106b responds with an error code, causes the component 104 to crash upon receiving an expected response from the stub component 106c. Because the sequence of calls and the inputs and outputs to the calls are recorded by the run-time infrastructure 108, a user may determine that the inappropriate delay from the stub component 106a caused the component 104 to eventually crash. This discovery may not be possible if the sequence of calls and the inputs and outputs to the calls were not maintained by the run-time infrastructure 108.

[0026] An example of a run-time infrastructure configured to maintain a sequence of calls and pass messages from which call inputs and outputs may be reconstructed between components, may be found in co-pending and commonly assigned U.S. patent application Ser. No. 09/955,764 filed on Sep. 19, 2001, and entitled “Run-Time Monitoring in Component-Based System,” the disclosure of which is hereby incorporated by reference in its entirety.

[0027] An example of a manner in which identifiers are employed to perform a test on the component 104 is provided with respect to FIG. 1B. More particularly, FIG. 1B illustrates a call graph 150 created from a sequence of calls captured by the run-time infrastructure 108. As shown, the call graph 150 identifies a test suite 152 and a test case 154. In this example, the test suite 152 has an identifier equal to “1” and the test case 154 has an identifier equal to “2”. As such, the test program 102 is configured to perform the test case 154 identified by the test suite identifier “1” and the test case identifier “2”.

[0028] In addition, the test program 102 appends the identifiers of the test suite 152 and the test case 154 to call A 156 sent to the component 104. The run-time infrastructure 108 may merge the test case identifier and test suite identifier for the test case 154 into a single unique identifier. In response to the call A 156 and the appended identifier, the component 104 makes three calls, shown as call B1 158, call B2 160, and call B3 162, to the stub components 106a, 106b, and 106c, respectively. The identifier is also appended to the three calls B1 158, B2 160, and B3 162. In response to the respective calls and the appended identifier, the stub components 106a, 106b, and 106c may switch their behaviors in some respect, as described in greater detail herein below, after the stub components 106a, 106b, and 106c retrieve the test case identifier and the test suite identifier, from the appended identifier that propagates in the run-time infrastructure 108 along the call sequence. As an example, however, the stub component 106b may respond to the component 104 with calls C1 164 and C2 166. The component 104 may then perform a function based upon the response received from the stub component 106b. In this regard, the interactions between the component 104 and the stub components 106a-106n may be tested. As such, the calls depicted in the call graph 150 may be used by a programmer to determine if the programming interactions between two or more components are operating correctly.

[0029] With reference now to FIG. 2, there is shown a flow diagram of a method 200 for testing a component 104 of a distributed system. The following description of the method 200 is made with reference to the system 100 illustrated in FIG. 1A, and thus makes reference to the elements cited therein. The following description of the method 200 is one manner of a variety of different manners in which the system 100 may be implemented. In addition, it should be understood that the method 200 shown in FIG. 2 is not limited to being implemented by the elements shown in FIG. 1A and may be implemented by more, less, or different elements as those shown in FIG. 1A.

[0030] As shown in the method 200, the test program 102 sends a first invocation request, including an identifier, to the component 104 at step 202. In other words, the test program 102 sends a call to the component 104, as indicated by the arrows 120a and 120b, to perform a test on the component 104. Appended to the call is an identifier, such as the test case identifier and/or the test suite identifier described hereinabove. The identifier is henceforth propagated in the system 100, for instance, as shown and described above with respect to FIG. 1B.

[0031] More particularly, for instance, the test program 102 sends the call with the identifier to the run-time infrastructure 108, as identified by the arrow 120a. In addition, the call and the identifier are sent from the run-time infrastructure 108 to the component 104 as indicated by the arrow 120b. In this regard, the run-time infrastructure 108 may track the call and the identifier sent from the test program 102 to the component 104.

[0032] At step 204, the component 104 sends a second invocation request, which includes the identifier from the first invocation request sent from the test program 102, to a second component, such as, the stub component 106a, as indicated by the arrows 122a and 122b. More particularly, the arrows 122a and 122b indicate that the second invocation request is also sent through the run-time infrastructure 108. In one respect, therefore, the run-time infrastructure 108 may also track the second invocation request sent from the component 104 to the second component, such as, the stub component 106a.

[0033] In response to the second invocation request and the identifier, the second component switches its behavior at step 206. In other words, the second component switches its behavior as called for by the identifier. For instance, the second component may switch its behavior to emulate a predefined response associated with the identifier. In keeping with the example above, the second component may switch its behavior to not respond, to respond after a significant delay, to respond with a relatively large work load, etc. By switching the behavior of the second component, the component 104 may be tested under various different possible scenarios.

[0034] Turning now to FIG. 3, there is shown a flow diagram of a method 300 for testing a component of a distributed system according to another example. The following description of the method 300 is made with reference to the system 100 illustrated in FIG. 1A, and thus makes reference to the elements cited therein. The following description of the method 300 is one manner of a variety of different manners in which the system 100 may be implemented. In addition, it should be understood that the method 300 shown in FIG. 3 is not limited to being implemented by the elements shown in FIG. 1A and may be implemented by more, less, or different elements as those shown in FIG. 1A.

[0035] As shown in FIG. 3, a test suite with test cases including a test suite identifier and test case identifiers is provided to the test program 102 by, for instance, a user, at step 302. At step 304, the test program 102 may initialize a test case on the component 104. The test initialization may be performed prior to execution of a test case and may include sending of an identifier into a storage area of the run-time infrastructure 108 via a utility function call supported by a testing utility library. To substantially avoid conflict due to concurrent thread execution, the storage area of the run-time infrastructure 108 may be private to the thread that is running the test program 102.

[0036] As a result of the test initialization, the identifier, which may be the merging of the test suite identifier and test case identifier, is sent to the run-time infrastructure 108 at step 304. At step 306, the test program 102 invokes a call to the component 104, and the run-time infrastructure 108 appends the stored identifier to the call request message, as indicated by the arrows 120a and 120b. More particularly, in the run-time infrastructure 108, the call from the test program 102 is translated into a call request message. In addition, the identifier is retrieved from the storage area of the run-time infrastructure 108 and appended to the call request message sent to the component 104.

[0037] The identifier is propagated in the system 100, following the call chain that occurs between the component 104 and the stub components 106a-106n. As the identifier propagates through the system 100, the run-time infrastructure 108 may maintain a record of the relationships between the identifier and the calls both from and to the component 104. The run-time infrastructure 108 may also track communications between the component 104 and the stub components 106a-106n.

[0038] In response to the call invoked by the test program 102, the component 104 sends a call to another component, the stub component 106a, as indicated by the arrows 122a and 122b, at step 308. As shown by the arrows 122a and 122b, the call is sent to the run-time infrastructure 108 prior to being sent to the stub component 106a. The run-time infrastructure 108 appends the call from the component 104 to the stub component 106a with the identifier at step 310, and the call is sent to the stub component 106a with the identifier.

[0039] At step 312, the stub component 106a switches behavior based upon the identifier retrieved from the run-time infrastructure 108. At step 314, the stub component 106a returns a predetermined response in accordance with the switch in behavior to the component 104, as indicated by the arrows 124a and 124b, to test the component 104 based upon the switched behavior. The response to the component 104 may include, for instance, responses designed to emulate various conditions that may occur between the component 104 and various other components. For example, the stub component 106a may return an error code, an expected or unexpected value, a relatively large file, etc. In certain situations, however, the identifier may cause the stub component 106a to not respond until a predetermined period of time has elapsed. This situation may simulate a condition where a program running on a remote server is unavailable for a period of time. These responses are thus designed to test the ability of the component 104 to respond under various possible conditions.

[0040] In addition, at step 316, after the response is returned from the stub component 106a, the component 104 responds to the response returned from the stub component 106a based on the implementation of the component 104 and sends back a response to the test program 102, as indicated by the arrows 132a and 132b. The test program 102 then evaluates the response from the component 104 resulting from the switch in behavior of the stub component 106a, at step 318.

[0041] With reference now to FIG. 4, there is shown a flow diagram of a method 400 for testing a component of a distributed system, according to a further example. The following description of the method 400 is made with reference to the system 100 illustrated in FIG. 1A, and thus makes reference to the elements cited therein. The following description of the method 400 is one manner of a variety of different manners in which the system 100 may be implemented. In addition, it should be understood that the method 400 shown in FIG. 4 is not limited to being implemented by the elements shown in FIG. 1A and may be implemented by more, less, or different elements as those shown in FIG. 1A.

[0042] As shown in the method 400, a call, with an identifier, is sent to the component 104 by the test program 102 at step 402. As described above with respect to FIG. 3, the call from the test program 102 and the identifier may be sent through the run-time infrastructure 108. The identifier is propagated in the system 100 following a call chain that occurs between the component 104 and the stub components 106a-106n. More particularly, for instance, the component 104 may respond to the call by sending a call to a second component, in this case, the stub component 106a, as indicated by the arrows 122a and 122b. The call from the component 104 may be received by the run-time infrastructure 108 as indicated at step 404 and the run-time infrastructure 108 may append the call with the identifier at step 406. In addition, the call and the identifier may be sent to the stub component 106a as indicated by the arrow 122b.

[0043] At step 408, the stub component 106a switches behavior based upon the identifier. In addition, the stub component 106a returns a predetermined response in accordance with the switch in behavior to the component 104, as indicated by the arrows 124a and 124b, at step 410. The component 104 may respond to the response from the stub component 106a by sending a call to a third component, in this case, the stub component 106b, as indicated by the arrows 126a and 126b. The call from the component 104 may be received by the run-time infrastructure 108 as indicated at step 412 and the run-time infrastructure 108 may append the identifier to the call at step 414. In addition, the call and the identifier may be sent to the stub component 106b as indicated by the arrow 126b.

[0044] At step 416, the stub component 106b switches behavior based upon the identifier. In addition, the stub component 106b returns a predetermined response in accordance with the switch in behavior to the component 104, as indicated by the arrows 128a and 128b, at step 418. In addition, at step 420, the response by the component 104 to the switch in behavior of the stub components 106a and 106b is determined based upon the implementation of the component 104. In addition, a response from the component 104 is returned to the test program 102, as indicated by the arrows 132a and 132b. The test program 102 then evaluates the response to determine how the component 104 responds to the switch in behavior of the stub component 106a and the stub component 106b, as indicated at step 422.

[0045] Although not shown, the component 104 may also respond to the response from the stub component 106b by sending a call to a fourth component, and so on. This process may continue with any number of test cases in a test suite and may be scaled to take into account any reasonably suitable number of stub components 106a-106n. The detailed representation of the identifier may be updated, for instance, when the test program 102 determines that execution of the current test case has been finished. In addition, the test program 102 may initiate the execution of a new test case with a new identifier following completion of the current test case.

[0046] FIG. 5 shows a flow diagram of a method 500 for implementing stub components for a test suite and building a testing system for testing a component in a distributed system. The following description of the method 500 is made with reference to the system 100 illustrated in FIG. 1A, and thus makes reference to the elements cited therein. The following description of the method 500 is one manner of a variety of different manners in which the system 100 may be implemented. In addition, it should be understood that the method 500 shown in FIG. 5 is not limited to being implemented by the elements shown in FIG. 1A and may be implemented by more, less, or different elements as those shown in FIG. 1A.

[0047] In the method 500, the component 104 to be tested is specified along with the stub components 106a-106n associated with the component 104, at step 502. More particularly, at step 502, a user may specify which component 104 is under test and which stub components 106a-106n are associated with or are otherwise configured to interact with the component 104. In addition, the user may provide test-oriented implementations for the stub components 106a-106n to generally enable free-will implementations of the stub components 106a-106n. In this regard, for instance, a relatively broad range of testing scenarios may be validated, independent of the actual implementations of the stub components 106a-106n.

[0048] At step 504, helper classes, proxies, and stubs are produced using a compiler. The compiler may be employed to automatically produce the helper classes, proxies, and stubs to substantially enable remote accessibility between the component 104 and the stub components 106a-106n. At step 506, a test suite having identifiers associated with the test suite is provided. The test suite may be associated with a test plan for the test program 102 and may thus include test cases with identifiers.

[0049] At step 508, stub component 106a-106n implementations are provided for the test suite and test cases. The implementations of the stub components 106a-106n may be provided by the user. In addition, in each implementation of the stub components 106a-106n, the user may use the application program interface (API) determined from a run-time monitoring to retrieve the test suite identifiers and the test case identifiers. The implementations of the stub components 106a-106n may be provided to coordinate various behaviors of the stub components 106a-106n with these identifiers.

[0050] At step 510, the testing system, which may be represented as at least one executable code, is built based upon the manually implemented code and the code generated by the compiler. The testing system is then run and the identifier propagates through the run-time infrastructure 108 to the component 104 and each stub component 106a-106n called by the component 104 at step 512, as described in greater detail herein above with respect to FIGS. 2-4.

[0051] Some or all of the steps illustrated in the methods 200, 300, 400, and 500 may be contained as a utility, program, subprogram, in any desired computer accessible medium. In addition, the methods 200, 300, 400, and 500 may be embodied by a computer program or a plurality of computer programs, which may exist in a variety of forms both active and inactive in a single computer system or across multiple computer systems. For example, they may exist as software program(s) comprised of program instructions in source code, object code, executable code or other formats for performing some of the steps. Any of the above may be embodied on a computer readable medium, which includes storage devices.

[0052] Examples of suitable computer readable storage devices include conventional computer system RAM (random access memory), ROM (read only memory), EPROM (erasable, programmable ROM), EEPROM (electrically erasable, programmable ROM), and magnetic or optical disks or tapes. Concrete examples of the foregoing include distribution of the programs on a CD ROM It is therefore to be understood that those functions enumerated below may be performed by any electronic device capable of executing the above-described functions.

[0053] FIG. 6 illustrates a block diagram of a computer system 600 that may implement some of the methods shown in FIGS. 2-5. The computer system 600 includes one or more processors, such as processor 602, providing an execution platform for executing software. The processor 602 may also execute an operating system (not shown) for executing the software in addition to performing operating system tasks.

[0054] The computer system 600 also includes a main memory 604, such as a Random Access Memory (RAM), providing storage for executing software during run-time and mass storage 606. The mass storage 606 may include a hard disk drive 608 and/or a removable storage drive 610, representing a floppy diskette drive, a magnetic tape drive, a compact disk drive, or a nonvolatile memory where a copy of software or data may be stored. Applications and resources may be stored in the mass memory 606 and transferred to the main memory 604 during run time. The mass memory 606 may also include ROM (read only memory), EPROM (erasable, programmable ROM), EEPROM (electrically erasable, programmable ROM).

[0055] A user interfaces with the computer system 600 with one or more input devices 612, such as a keyboard, a mouse, a stylus, or any other input device and views results through a display 614. A network interface 616 is provided for communicating through a network 618 with remote resources 620. The remote resources 620 may include servers, remote storage devices, data warehouses, or any other remote device capable of interacting with the computer system 600.

[0056] What has been described and illustrated herein are examples of the systems and methods described herein along with some of their variations. The terms, descriptions and figures used herein are set forth by way of illustration only and are not meant as limitations. Those skilled in the art will recognize that many variations are possible within the spirit and scope of these examples, which intended to be defined by the following claims and their equivalents in which all terms are meant in their broadest reasonable sense unless otherwise indicated.