Method and System for Testing Graphical User Interface Programs

A portion of the disclosure of this patent document contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone for purpose of review of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent files or records, but otherwise reserves all copyrights whatsoever.

1. Field of the Invention

This invention relates generally to the field of software testing and, more particularly, to the field of testing graphical user interface programs.

2. Description of the Related Art

One aspect of the field of software testing focuses on the testing of graphical user interface programs (GUI programs). A GUI program is a program that provides functionality which is not accessible from a command line but is accessible through the program's graphical user interface. Typically, GUI program testing takes place during a product development cycle of a new program. GUI program testing may also take place during the development of a new version of an existing GUI program.

GUI program testing often relies on a test engineer manually trying different user input combinations and noting anomalies as they occur. After the engineer changes the software to address the anomalies, the manual testing process is repeated until the test engineer verifies that the anomalies have been fixed. Manual testing has several drawbacks. For example, it is prone to error because it relies too heavily on human involvement in the testing process.

Due to the inadequacies of manual testing techniques, automated testing techniques have been developed. FIG. 1 illustrates a computer system 100 for testing a GUI program 101 using automated testing techniques found in the prior art. The computer system 100 includes a display device 103, a storage device 105, an input device 107, and a computer 109. The computer includes an interface 111, a processor 113, and a computer memory 115, which stores the GUI program 101. The GUI program includes a GUI component 119 and an engine component 121. The GUI component updates and retrieves information from a GUI 117. In short, the GUI component has program code to control changes to the GUI. Typically, the code is generated automatically from GUI builders. Engine code is code that implements the program's functionality. In other words, engine code generates the information to be displayed on the GUI.

Prior art automated testing methods simulate input events (such as mouse clicks) in order to test the same data processing paths that are invoked in response to user input. In this way all functionality in a GUI program is tested through the program's GUI component. The prior art methods begin the testing process by having a user enter data through a GUI 117 using the input device 107. For example, the inputs may indicate that a "File Open" command should be invoked. The method records the user inputs and saves them to a separate file. The recorded inputs are used as a test script 123 later in the testing process. A test script engine 124 drives the test script 123. In response to the inputs, the GUI 117 changes its state. For example, if the inputs indicate that a "File Open" operation should be invoked, then the method opens a file and displays it on the display device 103. In response to the changes in the GUI the method saves the new GUI state as a bitmap in a raster file (also called saving a "snapshot" of the GUI). The bitmap may also be saved in any other type of image format.

The snapshot acts as a baseline for comparing the program's response to user inputs at an earlier point in the development process with the program's response to the same user input at a later point in the development process. For example, as the development process continues, changes are made to the program code underlying the GUI component 119 and the engine component 121. In order to test the changes using automated techniques, the prior art method replays the user inputs recorded earlier. In response to replaying the user input, the GUI component 119 invokes the engine component 121 which in turn instructs the GUI component to redraw the GUI. In this way the engine component 121 is invoked through the GUI component 119. The method then takes a snapshot of the changes to the GUI 117. Finally, the earlier snapshot is compared to the later snapshot. If the snapshots are identical then the method concludes that the changes made to the program work properly. Non-identical snapshots may indicate that a bug has been introduced into the program.

There are several disadvantages associated with the automated testing techniques used in the prior art. The maintainability of the "test scripts" (i.e., the recorded user inputs) is low. In other words, as the development cycle progresses, the GUI 117 typically undergoes a wide variety of changes. Therefore, the user inputs made at an early stage of development may be meaningless at a later stage of development. One example could involve recording a user selecting a pull-down menu on a first screen and then invoking a "File Open" command from the pull-down menu. If the "File Open" command was removed from the pull-down menu of the first screen during the development cycle then the test script would be invalid when testing the revised pull-down menu. In this case a comparison of GUI snapshots would be impossible.

Another disadvantage associated with the prior art method involves the difficulty involved in porting the test scripts to different platforms. If the "baseline" snapshot is taken on a first platform and subsequently compared to a snapshot taken on a second platform then the bitmaps will not match if any of the system parameters change between the platforms. For example, snapshots taken on an Intel.TM. platform are often more condensed than snapshots taken on a Sparc.TM. platform. Therefore, the snapshots will not match even if the test script causes the same actions to occur to the GUIs of both platforms.

Another disadvantage associated with the method of the prior art is the amount of storage space required to store the snapshots. Each snapshot requires a significant amount of storage space. Often, over 300 snapshots are saved in order to fully test one GUI program. Such storage requirements are expensive and therefore provide a disincentive to using the prior art method.

Finally, even if the prior art testing method concludes that a bug was introduced into the GUI program, it is still difficult to determine whether the bug is in the GUI component 119, or the engine component 121, or some combination of the two components.

In a preferred embodiment, the present invention provides an improved method and system for testing a graphical user interface program. The preferred embodiment provides improved performance over prior methods because it decouples the testing of a program's engine component from the testing of its GUI component. The preferred method begins the testing process by invoking test functions that send data to the program's engine component. The data simulates user action on the graphical user interface of the program. In response to receiving the data, the engine component processes the simulated user action as flit had been sent from the graphical user interface component. The preferred method captures the results of this processing and uses the results to determine whether the engine component is performing properly.

FIG. 1 illustrates a computer system for testing a GUI program using automated testing techniques found in the prior art.

FIG. 2 illustrates a computer system for testing a GUI program using the automated testing techniques of the present invention.

FIG. 3 illustrates a flow diagram that provides an overview of the processing that takes place when practicing an embodiment of the present invention.

FIG. 4 is a flow diagram of the steps taken by a No Hands process when practicing an embodiment of the present invention.

FIG. 5 illustrates the programming interface for NhFunction in a language independent format.

FIG. 6 illustrates the programming interface for NhSymbol in a language independent format.

FIG. 7 illustrates the programming interface for NhPointerToObject in a language independent format.

FIG. 8 illustrates the programming interface for NhObjectToPointer in a language independent format.

FIG. 9 is a flow diagram that illustrates the steps taken by a test function when practicing an embodiment of the present invention.

Appendix A is a No Hands manual.

In a preferred embodiment, the present invention provides an improved method and system for testing a graphical user interface program. The present invention overcomes problems in the prior art by decoupling the testing of the program's engine component from the testing of the program's GUI component.

The preferred embodiment is typically used during the development of a new product. Not all components of a product move through the development cycle at the same pace. It has been observed that the design of an engine component of a GUI program typically stabilizes before the design of a graphical user interface component. This observation is important to understanding why the prior art testing methods fail to efficiently test GUI programs. The prior art methods test each component of the graphical user interface program through the GUI component. It is inefficient to test a GUI program through a component whose design is typically unstable well into the product development cycle.

The preferred embodiment provides improved performance over prior methods because it tests a program's engine component without invoking the engine component through the GUI component. The preferred method begins the testing process by invoking test functions that send data to the program's engine component. The data simulates user action on the graphical user interface of the program. In response to receiving the data, the engine component processes the simulated user action as if it had been sent from the graphical user interface component. The preferred method examines the results of this processing to determine whether the engine component is performing properly. By testing the engine component without invoking the graphical user interface component, the preferred embodiment increases the portability and maintainability of the test scripts while reducing the overall time spent testing the GUI program.

FIG. 2 illustrates a computer system 200 which provides an environment for practicing the preferred method of the present invention. The computer system 200 includes a display device 203, a storage device 205, an input device 207, and a computer 209. The display device 203 includes a graphical user interface (GUI) 211 for facilitating the input and output of data in the system 200. The computer includes an interface 213, a processor 215, and a computer memory 217. The computer memory stores a number of items including a GUI program 219, a test script 221, and a test script engine 222. The GUI program and the test script are preferably implemented as distinct processes. The test script 221 includes a series of commands which invoke routines that simulate user action on a user interface of the GUI program. The simulated action is used to test the GUI program. Test scripts can be manually developed from design specifications of the GUI component and the engine component while both components are under development. The test scripts can also be created automatically through use of a computer program developed for that purpose. The commands are retrieved from the test script and sent to the GUI program via a component 223. In the preferred embodiment, the component 223 is the No Hands software product from SunSoft, a division of Sun Microsystems, Inc..sup.1 The No Hands product is described in the No Hands manual attached hereto as Appendix A and incorporated herein by reference. The No Hands program is preferably implemented as a distinct process and communicates with the test script and the GUI program through any well-known interprocess communication mechanism (not shown). The computer memory also includes an operating system 231.

In one embodiment of the present invention the GUI program includes a GUI component 225, an engine component 227, and a test function component 229. The GUI component updates and retrieves information from the GUI 211. The engine component implements the program's underlying functionality. The test function component is invoked from routines in a No Hands library 233. Those of ordinary skill will understand that the No Hands library can be linked, either dynamically or statically, into the GUI program. The test function sends simulated interactions between program components to the program's engine component. The test function component then returns a value to the test script 221 or the test script uses the No Hands component 223 to retrieve information to determine whether the engine component is functioning properly.

FIGS. 3-9 are flow diagrams which illustrate a preferred method for practicing the present invention. FIG. 3 is a flow diagram that provides an overview of the processing that takes place in the system 200 when the preferred embodiment tests a GUI program. In step 301, the method invokes a test script. The test script then launches No Hands in step 303. No Hands retrieves from the test script a command which instructs the GUI program to run a test function. The No Hands process then sends the command to the GUI program. In step 305, the GUI program receives the command and invokes the test function indicated by the command. The test function invokes the engine component and then No Hands or the test script compares the result produced by the engine component with an expected result to determine whether the engine component works as expected.

FIG. 4 is a flow diagram of the steps taken by the No Hands process. Before processing begins in this method, the system opens the test script through a call to the operating system. The test script then launches No Hands through a call to the operating system. In step 401, No Hands determines whether unprocessed commands still remain in the test script. If unprocessed commands do remain, No Hands retrieves the next command from the test script (step 403). The command instructs the GUI program to invoke a test function which indicates whether the engine component of the program is working properly. In one embodiment, the preferred command is a No Hands NhFunction command. The NhFunction command allows a user of the command to cause the calling of any library routine within the GUI program. The routine may, for example, exist within a shared, linked library that can be named or in a binary program with a corresponding symbol table. In some embodiments, the NhFunction command calls an NhSymbol command which looks up symbols out of a shared library. The symbols returned may be used in calls generated by NhFunction. The NhFunction command may also call NhPointerToObject or NhObjectToPointer. NhPointerToObject converts a widget pointer from the GUI program to a corresponding object name maintained by No Hands. When used in conjunction with NhFunction, return values are converted to objects that can be addressed in the test scripts. NhObjectToPointer converts the No Hands' object name to the widget pointer recognized by the GUI program. When used in conjunction with NhFunction, function calls can be made within the GUI program on any object whose name is known by No Hands. FIG. 5 illustrates the programming interface for NhFunction in a language independent format. FIG. 6 illustrates the programming interface for NhSymbol in a language independent format. FIG. 7 illustrates the programming interface for NhPointerToObject in a language independent format. FIG. 8 illustrates the programming interface for NhObjectToPointer in a language independent format.

Returning to the discussion of FIG. 4, in step 405, No Hands examines the command it retrieved from the test script. If the command does not instruct the system to invoke a routine internal to the GUI program then normal processing occurs in step 407. However, if the command (e.g., the NhFunction command) does instruct the GUI program to invoke an internal routine (e.g., an internal test function) then No Hands executes the command to invoke the routine. Upon return from step 409, No Hands compares the result captured by the test function with an expected result (step 411). In the preferred embodiment a set of expected results is determined before testing begins and is maintained in a separate file. If the result from the engine component matches the expected result (step 413), then the engine component passes this test (step 415). If the results do not match then the engine component fails this test (step 417). Those of ordinary skill will understand that the comparison could also be carried out after performing a series of invocations of the engine component instead of after performing a single invocation of the engine component. Those of ordinary skill will also understand that the analysis of the captured results could alternatively be performed by the test script. Upon completion of step 415 or 417, processing continues with step 401. In step 401, if all the commands have been processed then processing stops in the test script (step 419).

FIG. 9 is a flow diagram that illustrates the steps taken by the test function. In step 901, the test function invokes the engine component of the GUI program. The test function typically simulates interactions between program components, which are represented by a set of callbacks invoked by the GUI component in response to an event on the user interface. A callback is a routine that is automatically invoked in response to user input on a GUI object. For example, a set of callback routines could be invoked when the left mouse button is pressed and released while inside a button of a dialog box. Therefore, the test function typically invokes those parts of the engine component which would have been invoked by the callback routines associated with a given event. Those of ordinary skill will understand that test functions are not limited to simulating interactions between program components but could instead be written to simulate any action(s) processed by the program. At times, no actual data is sent via the test function, rather, the invocation of the test function is the data itself (e.g., instructing a debugger to perform a step). Test functions can also be written to provide access to internal data for consistency checking and to provide additional functionally that aids the testing process but is not exposed to the user.

In step 903, the test function captures the result of the data processing performed by the engine component. For example, the test function could instruct the GUI program to write its results to a separate file for later examination or the test function itself could have a return value that indicates the result of the processing. After performing the test, the test function returns to the No Hands process, indicating that the test function has completed its processing and the GUI program is ready to receive a new command.

By following the preferred method illustrated in FIGS. 3-9, a test engineer is able to test an engine component of a GUI program without invoking the engine component through the GUI component. After the design of the GUI component has stabilized, the test engineer can begin testing the GUI component. Since the test engineer knows that the engine component works properly, the test engineer need only make sure that the GUI appears correctly on the display device and that the proper callbacks are made in response to user action on the GUI.

The present invention provides a number of advantages over methods used in the prior art. First, the total time spent testing the GUI program decreases. In the prior art a bug was found by comparing snapshots. However, this technique did not tell the test engineer whether the bug was in the GUI component or the engine component. It was up to the test engineer to search both components looking for the bug. Using the techniques of the present invention this search time is minimized because the two components are tested separately.

Second, the present invention makes it easier to port the test scripts and test results. As was mentioned above, the prior art methods ran into problems when comparing snapshots of GUIs across different platforms. If any system parameters changed between platforms then the snapshots could give the false indication of a bug when the programs actually worked as expected. By creating test scripts which test the underlying functionality of the engine component, the present invention provides scripts which are easily ported because the underlying functionality of the GUI program typically remains substantially the same across platforms.

In short, the preferred embodiment provides complete flexibility by isolating testing of the GUI component from testing of the engine component.

From the foregoing it will be appreciated that, although specific embodiments of the invention have been described herein for purposes of illustration, various modifications may be made without deviating from the spirit and scope of the invention. For example, while the test functions have been shown and discussed as residing in the GUI program, those of ordinary skill in the art will appreciate that the test functions could alternatively be incorporated in a library and linked (either dynamically or statically) to the GUI program. While the embodiment discussed above uses the No Hands product to provide synchronization and inter-process communication, those of ordinary skill will understand that this functionality could be provided directly by the test script. While the embodiment discussed above illustrates one aspect of the invention using a GUI program and a test script, those of ordinary skill will understand that the invention can be used to provide more effective testing of any two software components.

While the interprocess communication channels have been discussed above as two-way communication links, those of ordinary skill will understand that one-way communication links could be substituted for some or all of the two-way communication links.

Another embodiment operates in a way substantially similar to the preferred embodiment, except that the test functions are used to test the GUI component 225 without invoking the engine component 227. In this way testing of the GUI component is decoupled from testing of the engine component.

Those of ordinary skill will understand that when test functions are included (i.e., instrumented) in the GUI program, the test functions can later be used to perform diagnostics on the program after it is operational in the field.

While the embodiment discussed above describes simulating user actions in the test functions, those of ordinary skill will realize that these test functions can be used to simulate and report on non-GUI events affecting the engine component, such as in a distributed or client/server application. For example, a query to a remote database could be reported on or even simulated by test functions. ##SPC1##