Mechanism for graphical test exclusion

[0001] This Application is related to Application No. 10/767,849, entitled “Automated Test Execution Framework with Central Management,” filed 29 Jan. 2004.

[0002] 1. Field of the Invention

[0003] This invention relates improvements in software and hardware design verification. More particularly, this invention relates to methods and systems for centrally managing the composition and execution of test suites for verification of different hardware and software.

[0004] 2. Description of the Related Art

[0005] The meanings of acronyms and certain terminology used herein are given in Table 1. The terms Sun, Sun Microsystems, Java, J2EE, J2ME, J2SE, and the Sun logo are trademarks or registered trademarks of Sun Microsystems, Inc., in the United States of America and other countries. All other company and product names may be trademarks of their respective companies.

[0006]

	TABLE 1
	API	Application programming interface
	CLDC	Connected, limited device configuration. CLDC is
		suitable for devices with 16/32-bit RISC/CISC
		microprocessors/controllers, having as little
		as 160 KB of total memory available.
	HTTP	HyperText Transfer Protocol
	ID	Identifier
	IP	Internet Protocol
	J2EE	Java 2 Enterprise Edition
	J2ME	Java 2 Micro Edition
	J2SE	Java 2 Standard Edition
	JAD	Java application descriptor
	JAR	Java archive
	JDTS	Java Device Test Suite execution framework
	MIDlet	A MIDP application
	MIDP	Mobile information device profile. A set of Java
		APIs, which, together with the CLDC, provides a
		complete J2ME application runtime environment
		targeted at mobile information devices.
	Platform	An underlying system on which application
		programs can run on a computing device. It
		typically includes an operating system, and can
		include supporting hardware, which may be either
		standardized, or customized for a particular
		computing device.

[0007] Tools have been developed in recent years to aid in the design verification of hardware and software systems, for example software suites, hardware circuitry, and programmable logic designs. In order to assure that the design complies with its specifications, it is common to generate a large number of input or instruction sequences to assure that the design operates as intended under a wide variety of circumstances. In general, test systems produce a report indicating whether test programs have been passed or failed, and in some cases may even indicate a module that is estimated to be faulty.

[0008] Conventionally, test systems employing complex test suites use a computer-implemented testing framework for computing devices, such as mobile information devices, and for testing software designed to run on such devices. A developer submits a computing product under development, typically a computing device or software that is designed to run on the device to the test system, which runs a selected battery of test programs on the product while monitoring its behavior. Each product under test requires an instance of an execution test harness, or the use of a stand-alone test execution API. The latter is described in commonly assigned application Ser. No. 10/347,748, entitled “Generating Standalone MIDlets from a Testing Harness”, which is herein incorporated by reference.

[0009] In environments where testing of a product is ongoing, different aspects may be tested by different teams. As test results are evaluated, it is often necessary to revise the product under test, or to modify the test suites themselves. In such an environment, communicating such revisions to the different testing teams, maintaining version control, synchronization among the teams, and generally coordinating the testing activities is a difficult management problem. Errors could result in inappropriate testing and the wastage of valuable time and testing resources. Indeed, failure of coordination could result in the release of an inadequately tested product into the marketplace. A related problem when many test suites are being concurrently executed is the effort of setting up each test suite with its own test harness and environment. Bundling the test harness with the test suite is not a good solution, as the effort in maintaining up-to-date versions becomes formidable as the number of concurrently operating test suites increases and when the product or the test suites are frequently modified by different test teams.

[0010] There is also a need to further customize the composition of established test suites in a particular session, and in all sessions and clients. For example, a bug may have been discovered in a test or group of test programs comprising a test suite. The system administrator may therefore wish to exclude such test programs from performance by the clients of the test execution harness.

[0011] A centralized system for centrally managing test suites is disclosed in commonly assigned Application No. (STC File No. 47900), entitled “Automated Test Execution Framework with Central Management”, which is herein incorporated by reference. In this arrangement, a central repository contains a management unit, available test suites and a single test execution harness or framework. Using the management unit, a system administrator establishes active versions of the various test suites and their individual configurations. End users install clients of the central repository, using a system-provided installer program. In each client, an execution script is created, which downloads the harness and a local configuration file. Then, when the harness is executed at the client, it loads with all designated test suites already installed, configured and ready for execution. The client always has the most current versions of all test suites. Advantageously, all necessary information is obtained from a single central location. A modification of this and similar systems is the subject of the instant invention.

[0012] The invention provides methods and systems for verification of a hardware or software design. A test suite is executed in conjunction with a test framework for centrally managing the simultaneous or sequential execution of multiple test suites on one or more devices or simulators. Inventive techniques for optimally configuring the test suite initially involve a step of administrative qualification or disqualification of test programs in the suite as optionally excludable from performance. Using a graphical user interface that hierarchically displays the test suite as a tree, an administrator is able to select test programs or entire divisions of the tree in order either to disqualify selected test programs, to designate them such that they cannot be excluded by clients, or to qualify them for client exclusion. The results of the operation are compactly stored in a table of excludable records, as pointers to nodes of the tree. Using the same or a different graphical user interface, the client sees only those elements of the tree that have not been disqualified or designated as non-excludable by the administrator, and may select from among them to prevent one or more test programs of the test suite from being executed.

[0013] The invention provides a method for managing the composition and execution of test suites in testing a computing product, which is carried out by storing a suite of test programs for execution using the computing product, wherein the suite is organized in a hierarchy. The method is further carried out by hierarchically displaying the suite, and selecting qualified test programs from the suite, the qualified test programs being optionally excludable from performance. The method is further carried out by transmitting a pointer to a node of the hierarchy to a client, wherein all of the test programs that are located in a subtree headed by the node are included in the qualified test programs, and executing at least a portion of the test programs using the computing product responsively to the pointer.

[0014] An aspect of the method includes hierarchically displaying only the qualified test programs to the client, the qualified test programs being selectable by the client for performance thereof.

[0015] In another aspect of the method a hierarchical display to the client limited to the qualified test programs is achieved by disposing a representation of a subtree of the suite on a graphical user interface responsively to the pointer.

[0016] An additional aspect of the method includes selecting one of the qualified test programs via the graphical user interface as an excluded test program, and while executing the test programs, avoiding performance of the excluded test program.

[0017] Another aspect of the method includes selecting one of the qualified test programs via a master exclude list as an excluded test program, and while executing the test programs, avoiding performance of the excluded test program.

[0018] The invention provides a computer software product, including a computer-readable medium in which computer program instructions are stored, which instructions, when read by a computer, cause the computer to perform a method for managing the composition and execution of test suites in testing a computing product, which is carried out by storing a suite of test programs for execution using the computing product, wherein the suite is organized in a hierarchy. The method is further carried out by hierarchically displaying the suite, and selecting qualified test programs from the suite, the qualified test programs being optionally excludable from performance. The method is further carried out by transmitting a pointer to a node of the hierarchy to a client, wherein all of the test programs that are located in a subtree headed by the node are included in the qualified test programs, and executing at least a portion of the test programs using the computing product responsively to the pointer.

[0019] The invention provides a verification system for managing the composition and execution of test suites in testing a computing product, including a server, a memory in the server for storing a suite of test programs for execution using the computing product, the suite being organized in a hierarchy. The system further includes a terminal for hierarchically displaying the suite, an editor adapted for selection of qualified test programs from the suite, wherein the qualified test programs are optionally excludable from performance, and a communication facility for transmitting a table of the qualified test programs to a client, wherein the table of the qualified test programs includes a pointer to a node of the hierarchy, and wherein all of the test programs that are located in a subtree headed by the node are included in the qualified test programs.

[0020] The invention provides a method for managing the composition and execution of test suites in testing a computing product, which is carried out by receiving a table including qualified programs of a suite of test programs, the qualified programs being qualified for exclusion from performance, wherein the suite is organized as a hierarchy of the test programs, and the table includes a pointer to a node of the hierarchy, and wherein all of the test programs that are located in a subtree headed by the node are included in the qualified programs. The method is further carried out by hierarchically displaying the qualified programs, selecting one of the qualified programs to be excluded from performance to define an excluded test program, executing the suite of test programs using the computing product, and while executing the suite of test programs, avoiding performance of the excluded test program.

[0021] An aspect of the method executing the suite of test programs includes executing unqualified test programs of the suite.

[0022] The invention provides a method for managing the composition and execution of test suites in testing a computing product, which is carried out by providing a central repository holding a suite of test programs, an execution test harness, an installer, and a graphical user interface for modifying the suite, displaying the suite as a hierarchy on the graphical user interface, selecting qualified test programs from the suite via the graphical user interface, the qualified test programs being excludable from performance by a client of the execution test harness, and transmitting a record of the qualified test programs to the client.

[0023] According to one aspect of the method, the record of the qualified test programs includes a pointer to a node of the hierarchy, wherein all of the test programs in a subtree headed by the node are included in the qualified test programs.

[0024] A further aspect of the method includes downloading the installer to the client, and responsively to an execution of the installer in the client, downloading and installing at least a portion of the test programs from the central repository to the client for use by the client in testing the computing product, and avoiding performance of any of the excluded test programs.

[0025] The invention provides a computer software product, including a computer-readable medium in which computer program instructions are stored, which instructions, when read by a computer, cause the computer to perform a method for managing the composition and execution of test suites in testing a computing product, which is carried out by receiving a table including qualified programs of a suite of test programs, the qualified programs being excludable from performance, wherein the suite is organized as a hierarchy of the test programs, and the table includes a pointer to a node of the hierarchy, and wherein all of the test programs that are located in a subtree headed by the node are included in the qualified programs. The method is further carried out by hierarchically displaying the qualified programs, selecting one of the qualified programs to be excluded from performance to define an excluded test program, executing the suite of test programs using the computing product, and while executing the suite of test programs, avoiding performance of the excluded test program.

[0026] The invention provides a computer software product, including a computer-readable medium in which computer program instructions are stored, which instructions, when read by a computer, cause the computer to perform a method for testing a computing product, which is carried out by providing a central repository holding a suite of test programs, an execution test harness, an installer, and a graphical user interface for modifying the suite, displaying the suite as a hierarchy on the graphical user interface, using the graphical user interface to select qualified test programs from the suite, the qualified test programs being excludable from performance by a client of the execution test harness, and transmitting a record of the qualified test programs to the client.

[0027] The invention provides a verification system for managing the composition and execution of test suites in testing a computing product, including a client terminal that receives a table including qualified programs of a suite of test programs from a server, the qualified programs being qualified for exclusion from performance. The suite is organized as a hierarchy of the test programs, and the table includes a pointer to a node of the hierarchy, wherein all of the test programs that are located in a subtree headed by the node are included in the qualified programs. The system includes a graphical user interface in the client terminal that hierarchically displays the qualified programs for selection thereof. The client terminal is adapted, responsively to a selection via the graphical user interface of an excluded test program, to execute the suite of test programs using the computing product, and to avoid performance of the excluded test program.

[0028] The invention provides a verification system for managing the composition and execution of test suites in testing a computing product, including a server, a memory in the server that stores a central repository holding a suite of test programs, an execution test harness, an installer, and a graphical user interface for modifying the suite, a terminal displaying the suite as a hierarchy on the graphical user interface, qualified test programs from the suite being selectable via the graphical user interface. The system includes a communications facility for transmitting a record of the qualified test programs, and a client terminal linked to the server for hierarchically displaying only the qualified test programs responsively to receiving the record from the server. The qualified test programs are selectable as excluded test programs.

[0029] According to an aspect of the verification system, the server is adapted to download the installer to the client terminal. Responsively to an execution of the installer, the server is further adapted to download and install at least a portion of the test programs from the central repository to the client terminal for execution using the computing product, while avoiding execution of any of the excluded test programs.

[0038] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be apparent to one skilled in the art, however, that the present invention may be practiced without these specific details. In other instances well-known circuits, control logic, and the details of computer program instructions for conventional algorithms and processes have not been shown in detail in order not to unnecessarily obscure the present invention.

[0039] Software programming code, which embodies aspects of the present invention, is typically maintained in permanent storage, such as a computer readable medium. In a client/server environment, such software programming code may be stored on a client or a server. The software programming code may be embodied on any of a variety of known media for use with a data processing system, This includes, but is not limited to, magnetic and optical storage devices such as disk drives, magnetic tape, compact discs (CD's), digital video discs (DVD's), and computer instruction signals embodied in a transmission medium with or without a carrier wave upon which the signals are modulated. For example, the transmission medium may include a communications network, such as the Internet.

System Architecture.

[0040] Turning now to the drawings, reference is initially made to FIG. 1, which is a high level block diagram of a system 10 for centrally managing the simultaneous or sequential execution of multiple test suites on different platforms for verification of different hardware and software designs in accordance with a disclosed embodiment of the invention. The heart of the system 10 is a central repository 12, which can reside on a local or a remote server, and which contains data structures necessary for multiple clients (test consoles) of the central repository 12 to perform testing simultaneously, or at different times. The central repository 12 holds data structures that define platforms 14, test suites 16, an execution test harness 18, and an installer 20. The installer 20 creates a script for launching the framework. The script includes paths to binary files of the test execution framework. The binaries themselves are located in only one place, the central repository 12. Centrally locating the binary files is highly advantageous, as only one instance of each binary file need be updated. Furthermore, each user is guaranteed to see the most current version of the framework. Because distribution of local copies of the binaries is avoided, users need not be concerned about having outdated software. The execution test harness 18 may be implemented as a modification of the test framework “Java Device Test Suite” (JDTS) (version 1.0 or higher), available from Sun Microsystems, Inc., which employs MIDP.

[0041] Typically the installation is packaged in an archive, such as a JAR file. The system 10 is capable of being controlled by a single operator 22, using a client terminal 24. However, in many environments it is desirable that a superuser, or administrator 26, manages the central repository 12. This is done using a platform editor 28, which acts as a management unit. Using the platform editor 28, the administrator 26 is capable of reconfiguring the platforms 14, test suites 16, execution test harness 18, and installer 20. An end user 30 interacts with the central repository 12 via a client terminal 32. The end user 30 launches an execution package, which is first downloaded and installed from the central repository 12 using conventional communication facilities. The end user 30 and the terminal 32 may be co-located with the other components of the system 10, or alternatively may be located remotely and connected to the central repository 12 via a data network. Although only one instance of the end user 30 is shown, it will be understood that many end users can interact with the central repository 12 concurrently, or at different times, using the same or different platforms and the same or different test suites. The end user 30 and the terminal 32 are referred to hereinbelow as a client 34.

[0042] The system 10 is suitable for use in many development environments, including MIDlet development using J2ME. It may be adapted to J2EE and J2SE environments using programming techniques known to the art.

System—Functional Organization.

[0043] Continuing to refer to FIG. 1, the central repository 12 contains test parameters, platform configuration parameters, framework parameters and the test programs themselves. The end user 30 makes use of the central repository 12 in a “session”. In an implementation of a session for a framework application in the execution state, contents of the central repository 12 are stored and loaded. The disclosure of the system implementation is common to the platform editor 28, which manages platforms by using the central repository 12, and by the execution framework, including the installer 20, which creates an execution script and local configuration files. The execution test harness 18 is not downloaded. Its binary files remain on the central repository 12.

[0044] Reference is now made to FIG. 2, which is a high level functional block diagram of an implementation of the platform editor 28 (FIG. 1) in accordance with a disclosed embodiment of the invention. A class PlatformSessionManager 36 has methods for loading and saving platforms and sessions.

[0045] A class Platform 38 encapsulates all platform specific information. An instance of the class Platform 38 includes such information as available test suites, and their respective properties, as well as other platform-specific information. The test suites and their properties are managed by a class TestSuitesManager 40. Platform-specific information is managed by a class EnvironmentView 42. Test programs in the test suites can be excluded from performance by a class ExcludedTestsView 44.

[0046] A class Session 46 encapsulates all information specified by the client 34 (FIG. 1). Such client specific information includes the basic properties of the test suites and the platform that applies to a particular session of the client 34.

[0047] The class Platform 38 and the class Session 46 interact with the class TestSuitesManager 40, the class EnvironmentView 42, and the class ExcludedTestsView 44.

[0048] The class TestSuitesManager 40 interacts with a table TestSuiteTable 48, which contains records of the different test suites, as indicated by a class TestSuiteTable 50.

[0049] The class EnvironmentView 42 interacts with a table EnvironmentTable 52, which contains records of known platform environments, as indicated by a representative table EnvironmentRecord 54.

[0050] The class ExcludedTestsView 44 interacts with a table ExcludedTestsTable 56, which contains records of excluded test programs, as indicated by a representative class ExcludedTestsRecord 58.

[0051] Records originating from the class TestSuiteTable 50, the table EnvironmentRecord 54, and the class ExcludedTestsRecord 58, are included in a class record 60, and initially written to an interface table 62, before being ultimately transferred to an appropriate one of the table EnvironmentTable 52, the table EnvironmentRecord 54, or the table ExcludedTestsTable 56.

[0052] A class XMLAccessManager 64 accepts requests from the class TestSuitesManager 40, the class EnvironmentView 42, and the class ExcludedTestsView 44 for read or write operations to and from the class Platform 38 and the class Session 46. The class XMLAccessManager 64 manages queues of read requests 66, write requests 68, and executes them sequentially.

[0053] A class XMLAccess 70 contains a single access point to XML files that represent platforms and sessions. It has methods for read/write access to these XML files.

System Operation—Platform Editor.

[0054] Reference is now made to FIG. 3, which is a block diagram illustrating the modification of records in the class record 60 (FIG. 2), in which framework components 72, 74 may be extracted from the table TestSuiteTable 50, the table EnvironmentTable 52, and the table ExcludedTestsTable 56.

[0055] For example, assume that a test suite x has a parameter that points to its classes directory. This parameter should be written as follows:

• • [0056] TestClassesDir=PARENT_—DIR/bin/classes.

[0057] A system component replaces the parameter PARENT_—DIR according to an entry in a local configuration file, local.configuration, which contains the path to the directory TestClassesDir.

Master Exclude List.

[0058] In some applications, it is desirable to remove tests from test suites temporarily. Formerly, this could be done only by recompiling the entire test suite.

[0059] A facility for removal of test suites without use of a graphical user interface is provided, using a file that contains a master list of excludable tests, herein referred to as a “master exclude file”. An example of a master exclude file is shown in Listing 1. Each test suite having tests available for exclusion using the master exclude file has an entry that is prefixed “TS=”, followed by the full name of the test suite.

[0060] On a line below this entry is found the path to an excludable node. Thus, in Listing 1, the nodes com.sun.midp.midp1_—0, and com.sun.midp.midp2_—0.functional.tests.midlet can be excluded from the test suite MIDP 2.0 Tests. The nodes com.sun.wma.functional.Tests.connection.cbs.CBSClose.closeConnection, and com.sun.wma.functional.Tests.security.sms.client can be excluded from the test suite WMA 1.1 Tests.

[0061] To remove a node from the master exclude list, it is only necessary to edit the master exclude file, and re-execute the test framework.

[0062] The master exclude list is built using a static block or constructor inside a class ExcludedTestsViewImpl, which provides a session view of excluded test records. The master exclude list is loaded only once, when the ExcludedTestsViewImpl is loaded. The class ExcludedTestsViewImpl is disclosed more fully in the above-noted Application No. (STC File No. 47900). The class ExcludedTestsViewImpl has been further modified by the addition of a method is InMasterExclude (String name, String tsName), which returns the value True if a specified node appears in the master exclude list. If so, then the node is not shown to the user. Another method in the class ExcludedTestsViewImpl, is Excluded (String name, String tsName), tests for the existence of a specified test in the excluded test list of the current session. This method has been modified so that it also determines whether a specified node is in the master exclude list. This ensures that the user sees neither nodes in the master exclude list nor nodes that were conventionally excluded by the administrator in the currently loaded profile.

System Operation—Client.

[0063] With continued reference to FIG. 1 and FIG. 2, after the client 34 loads a platform from the central repository 12, it creates a session object, that is an instance of the class Session 46, which is based on the platform object, that is the class Platform 38. This session object contains only those basic properties that can be modified by the client 34. The session and platform objects contain all the information taken from the central repository 12, using the path replacement mechanism disclosed above. The client 34 can then operate as if a conventional local installation of the test suite and harness had occurred.

[0064] Further details of the system 10 (FIG. 1) and the data structures shown in FIG. 2 are disclosed in the above noted Application No. (STC File No. 47900).

Excluded Test Programs—Server Side.

[0065] According to the invention the test suites 16 (FIG. 1) are represented as a hierarchy of test programs. No limitations are placed on the hierarchy, which can have an arbitrary depth, and can be a binary tree, or a tree of any order, and may even vary in the number of branches arising from different nodes.

[0066] Reference is now made to FIG. 4, which illustrates a test suite 80 having ten test programs, elements of which may be administratively qualified for exclusion from performance by a client in accordance with a disclosed embodiment of the invention. The test suite 80 is represented as a multilevel tree having a root node 82, in which nodes represent divisions or subgroups of the test suite 80, and leaves represent test programs of the test suite 80. The root node 82 includes pointers to other nodes. Except for the trivial case of a test suite consisting of a single test program, in general the root node 82 does not contain actual test information. The test suite 80 is divided into three primary divisions 84, 86, 88. The primary division 84 has three secondary divisions 90, 92, 94. The secondary division 90 has three test programs 96, 98, 100. The secondary division 92 has two test programs 102, 104. The secondary division 94 consists of a single test program. The primary division 86 has two test programs 106, 108. The primary division 88 has two test programs 110, 112.

[0067] An administrative operator builds an instance of the table ExcludedTestsTable 56 (FIG. 1), using the table TestSuiteTable 48 as a reference, and a graphical user interface that provides a convenient hierarchical display of the elements of the test suite 80. The nodes and leaves of the hierarchy can be visited in any order, and can be selected or deselected using the graphical user interface. As will be seen from the discussion below, selection or deselection of a node can affect the parent and the entire subtree of that node.

[0068] Reference is now made to FIG. 5, which is a flow chart illustrating a method of qualifying test programs in a test suite for exclusion by a client in accordance with a disclosed embodiment of the invention. The purpose of the method is to administratively differentiate those test programs of the test suite that are optionally excludable by a client, from those whose performance is mandatory. Recalling that low end computing devices typically have limited storage, according to the method the configuration of excludable test programs is presented to the client using minimal memory resources, while still displaying the hierarchical relationships of the elements of the test suite. In preparing the table, elements of the test suite hierarchy can have three statuses:

• • [0069] (1) not flagged (not qualified for exclusion);
  • [0070] (2) flagged (at least one element in the subtree headed by the flagged element is qualified as excludable by a client); and
  • [0071] (3) added to table of excluded records (the element and its children are all qualified as excludable by the client).

[0072] The method begins at initial step 114. Here a current test suite is accessed. As noted above, this is typically, but not necessarily, accomplished using a graphical user interface, which displays the test suite as a tree according to its predefined hierarchical relationships.

[0073] Next, at step 116 an element, which can be either a node or leaf of the tree (FIG. 4), is accessed. This is normally done manually by an operator having administrative privileges. Alternatively, the order of visiting the nodes and leaves of the tree may be accomplished using a “master exclude list”. This list is embodied in a text file having a particular format in a special format. Consulting this list enables exclusion of specified nodes from the test suite without interaction with the graphical user interface. However, when the master exclude list is employed, an excluded node is no longer available for recall using the graphical user interface.

[0074] Control now proceeds to decision step 118, where it is determined if the test suite element corresponding to the element that was accessed in step 116 is to be qualified as optionally excludable by the client. If the determination at decision step 118 is negative, then control proceeds to decision step 120, which is disclosed below.

[0075] If the determination at decision step 118 is affirmative, then control proceeds to decision step 122, where it is determined if the parent of the element that was accessed in step 116 has been previously flagged as having at least one excludable element in its subtree or has been previously added to the table of excluded test programs, details of which are disclosed below. If the parent generation of that node becomes fully selected, more remote ancestral nodes will be affected recursively, as will be evident from the disclosure hereinbelow. If the determination at decision step 122 is negative, then control proceeds to decision step 124, which is disclosed below.

[0076] If the determination at decision step 122 is affirmative, then control proceeds to step 126. The subtree headed by the element that was accessed in step 116 is flagged as excludable and a pointer to the element stored.

[0077] Control now proceeds to decision step 124, where it is determined if all siblings of the element that was accessed in step 116 are now flagged as excludable. For example, if the test program 98 (FIG. 4) was selected in step 116, the inquiry in decision step 124 would extend only to the test programs 96, 100, as these are the only elements having a common parent with the test program 98, which is the secondary division 90.

[0078] Referring again to FIG. 5, if the determination at decision step 124 is affirmative, then control proceeds to step 128, which is disclosed below.

[0079] If the determination at decision step 124 is negative, then control proceeds to step 130. The element that was accessed in step 116 is added to the table of excluded test programs. Control then proceeds to decision step 132.

[0080] At decision step 132 a determination is made whether more nodes or leaves are to be accessed. If the determination at decision step 132 is negative, then control proceeds to final step 134, and the method terminates. Otherwise, control returns to step 116.

[0081] Step 128 is performed if the determination at decision step 124 is affirmative. Here the element that was accessed in step 116 and its siblings are all removed from the table of excluded test programs. The previous status of the node and its siblings is memorized, however, in order to retain a trace of previous accesses. This is essential, should it become necessary to reverse an action taken during performance of the method.

[0082] Next, at step 136, the parent of the element that was accessed in step 116 is added to the table of excluded test programs. The result of performing step 116, together with step 136, is a much smaller table of excluded test programs than would be the case if all flagged elements were simply added to the table of excluded test programs. Control next proceeds to decision step 132, which has been described above. Should the parent of the newly excluded node and its sibling all become excluded, there is automatic recursion to the grandparent node.

[0083] Decision step 120 is performed when the determination at decision step 118 is negative. Here it is determined whether the access of an element at step 116 was intended to cancel a previous action to exclude that element. If the determination at decision step 120 is negative, then control proceeds to decision step 132. This could occur, for example, if the access in step 116 was in error.

[0084] If the determination at decision step 120, then a sequence of steps is undertaken that reverses the operations of the sequence of steps beginning with decision step 122 and ending at decision step 132. Control proceeds to step 138, where the element selected in step 116 is removed from the table of excluded test programs if it had previously been entered.

[0085] Next, at step 140, the flag that was previously set in step 126 is reset.

[0086] Control now proceeds to decision step 142, where it is determined if the parent of the element selected in step 116 is in the table of excluded test programs. If the determination at decision step 142 is negative, then control proceeds to decision step 132, which has been described above.

[0087] If the determination at decision step 142 is affirmative, then control proceeds to step 144. The parent element is removed from the table of excluded test programs It should be noted that there is no automatic recursion to ancestral nodes in this case. If the parent was in the exclude table, then the grandparent node was not in the table. Only the node in the highest level of the hierarchy having all its children selected are found in the exclude table.

[0088] Next, at step 146 previously selected siblings of the element selected in step 116 are added to the table of excluded test programs. It will be recalled that the status of the siblings was memorized in step 128 in order to provide for the eventuality that step 146 might need to be performed. Control then proceeds to decision step 132, which has been described above.

[0089] Reference is now made to FIG. 6, which is a hierarchical screen display 148, which presents a test suite to the administrative operator following performance of the method disclosed with reference to FIG. 5. The test suite may comprise large numbers of test programs; however only those, which were administratively qualified as excludable by the client, are flagged on the display 148 by an open square icon. While a particular display scheme is presented in the disclosure of FIG. 6, it will be evident to those skilled in the art that many variations of the display 148 could be implemented to apply the principles of the instant invention. The left pane of the display 148 includes a folder 150 that contains excludable test programs. The folder 150 is expanded in the right pane. A division 152 of the test suite, entitled “Audiobuilding” is further expanded in the right pane. The division 152 is associated with a hatched square, indicating that it contains at least one test program that will not be visible to the client, and at least one qualified test program that will be visible. Another division 154, entitled “Benchmark_—test programs”, will be completely invisible to the client. The division 152 has only one division 156, entitled “com”, at the next lower level of the hierarchy. The division 156 is also associated with a hatched square and an “X”, the latter indicating that at least one element of its subtree has been disqualified by the administrator, and will be invisible to the client, and at least one test program that will be visible. The subsidiary divisions 158, 160, entitled “sun.midp” and “functional”, respectively are associated with different types of square icons. In the case of the division 158, there is a hatched square icon. The division 158 has an edge leading to a test program 162, entitled “stress”, which has a clear square icon. The test program 162 has not been disqualified by the administrator; it will be visible to the client, who may choose to exclude it from performance. The division 160 is the parent of yet another division 164, entitled “test programs.sound”. The division 164 comprises four test programs 166, associated with clear square icons having an “X”, indicating that they have been disqualified from exclusion by the administrator. None of the test programs 166 will be seen by the client.

Excluded Test Programs—Client Side

[0090] Referring again to FIG. 1, it is now assumed that the table ExcludedTestsTable 56 has been prepared using the method disclosed with reference to FIG. 5. The table ExcludedTestsTable 56 and the table TestSuiteTable 48 form the basis for presenting test programs of a test suite that have been administratively qualified as excludable by a client. The same graphical user interface that was used to create the table ExcludedTestsTable 56 may be used to display the qualified test programs, with such appropriate modifications, as will be evident to those skilled in the art. Alternatively, a different graphical user interface may be used. The display of the test suite is hierarchical. Subtrees having at least one excludable element, and test programs in the subtrees that are excludable are associated with distinctive indicia of their status.

[0091] The graphical user interface remains aware of the hierarchical structure of a current test suite, referencing the table TestSuiteTable 48 (FIG. 1). The user display is constructed “on-the-fly”, based on the minimal information contained in the table ExcludedTestsTable 56 (FIG. 1).

[0092] Reference is now made to FIG. 7, which is a flow chart illustrating a method of presenting test programs of a test suite that have been administratively qualified for exclusion to a client in a hierarchical format, in accordance with a disclosed embodiment of the invention. For purposes of FIG. 7, it is assumed that the method disclosed with reference to FIG. 5 has been performed, and that a table of excludable records has been constructed. It will be recalled from the disclosure of FIG. 5, that economy in data storage is achieved by saving a pointer to the highest level in the tree under which all test programs were administratively selected as being excludable by the client, without saving information as to the individual test programs thereunder. The process of building a hierarchical tree and presenting it to the client, using a graphical user interface, begins with initial step 168. A test suite table containing details of the test suite and its hierarchical relationships, for example, the table TestSuiteTable 48 (FIG. 1), is accessed and examined, in order to provide an awareness of the structure of the test suite. A reference hierarchical tree of the test suite is constructed for use by the graphical user interface, based on the test suite table.

[0093] Next, at step 170, an entry in the table of excludable records is accessed.

[0094] Next, at step 172, a node in the reference hierarchical tree corresponding to the entry that was accessed in step 170 is visited.

[0095] Control now proceeds to decision step 174, where it is determined if there are children of the node visited in step 172 to be processed. Decision step 174 is performed recursively, so those nodes at lower levels of the subtree headed by the node visited in step 172 are accessed in turn. If the determination at decision step 174 is affirmative, then control proceeds to step 176. Otherwise, control proceeds to decision step 178, which is described below.

[0096] At step 176 a child node of the node that was visited in step 172 is accessed.

[0097] Next, at decision step 180, the table of excludable records is examined, and a determination is made whether the child node that was accessed in step 176 appears in this table. If the determination at decision step 180 is affirmative, then the display presented to the client via the graphical user interface may be updated to show this node and its subtree. Control returns to decision step 174.

[0098] If the determination at step 176 is negative, then control proceeds to step 182, where the element of the test suite corresponding to the child node that was accessed in step 176 is added to the table of excludable records, and the graphical user interface updated.

[0099] Decision step 178 is performed when the determination at decision step 174 is negative. A determination is made whether there are more entries on the original table of excludable records to be evaluated.

[0100] If the determination at decision step 178 is affirmative, then control returns to step 170.

[0101] If the determination at decision step 178 is negative, then control proceeds to final step 184. At this point, the display presented via the graphical user interface is complete. The client selects actual test programs to exclude. A field of the table of excludable records is updated to reflect the status of these selected test programs. The procedure terminates. The test suite can now be executed using the design-under-test, while avoiding performance of the selected test programs.

[0102] It will be appreciated by persons skilled in the art that the present invention is not limited to what has been particularly shown and described hereinabove. Rather, the scope of the present invention includes both combinations and subcombinations of the various features described hereinabove, as well as variations and modifications thereof that are not in the prior art, which would occur to persons skilled in the art upon reading the foregoing description.

Computer Program Listings.

• [0103] TS=MIDP 2.0 Tests
• [0104] com.sun.midp.midp1_—0
• [0105] com.sun.midp.midp2_—0.functional.tests.midlet
• [0106] TS=WMA 1.1 Tests
• [0107] com.sun.wma.functional.Tests.connection.cbs.CBSClose.close
• [0108] Connection
• [0109] com.sun.wma.functional.Tests.security.sms.client