Method and system for supporting negative testing in combinatorial test case generators

[0001] The invention relates generally to computer systems, and more particularly to testing computer programs.

[0002] Given a large enough market, software consumers will find numerous bugs in software programs, which can reflect poorly on the developer. Thus, an important part of software development is testing to eliminate bugs in software, and to have the software designed to otherwise handle unusual circumstances. That is one reason why Beta testing is used, so that by sheer numbers, many users (who understand that bugs are likely early in the development process) can help to debug a product before it is sold to consumers.

[0003] Beta testing is only one way software is tested, and can only be done when the product under development is reasonably stable and safe enough to give to those who will use it in the real world. To get to this point, and also to find bugs that even large numbers of Beta testers may not find, software producers also run their programs through pre-arranged tests. Such pre-arranged testing is often referred to as black-box testing, in which parameters of a domain under test are varied to evaluate how the program behaves.

[0004] Black box testing can find many bugs, however there is no way for developers to realistically anticipate each of the possible combinations of parameters that can cause a bug. Testing all combinations is not practical; by way of an example to point out the difficulties in exhaustive testing, consider the “Font” dialog in the Microsoft® Word word processing program. The first “Font” tab of the “Font” dialog has lists of possible parameters for font, style, size, color, underline style, and underline color. Checkboxes are available for effects, including strikethrough, superscript, subscript, shadow, small caps, and others. On this first “Font” tab alone of the “Font” dialog, there are over 1,500,000 parameter combinations that can be tested in order to exhaustively evaluate this tab of the dialog. Moreover, there are many such dialogs and tabs. For example, there is a “Paragraph” dialog, which also has two tabs and a large number of parameters.

[0005] As can be readily appreciated, with limited computing resources and time, exhaustive testing of all such parameter combinations in a reasonable time is basically impossible for all but a few programs having very simple sets of parameters. Moreover, the ability to comprehensively test a program becomes even more impractical when each test case takes relatively long time to perform, such as when testing possible variations in installing a program, formatting a disk, and so on, where each test might take on the order of minutes to perform. As a result, even with relatively small domains, exhaustive testing is ordinarily not realistically possible.

[0006] Thus, black-box testing for all combinations, one parameter at a time, is simply not a sufficient solution to software testing. One approach is domain modeling, whereby parameters are defined and appropriate values for each of them are chosen by a tester. Then, a suite of test cases is devised out of the parameters. A simple way to test is just to cover a set of values for each parameter, without considering how the parameters combine. In general, based on the tester's own experience, the tester devises test cases that he or she believes covers the most likely interactions between parameters of the domain. As can be appreciated, while this can find some bugs, domain modeling is usually a long and tedious process, and highly dependent on the tester's ability to construct test cases. It also cannot easily be automated and often causes maintenance problems.

[0007] Another, more formal approach referred to as “default testing” makes explicit assumptions about how values will be most often combined with one another in typical usage. A tester varies one parameter per test, leaving all others with their default values, and observes how the program behaves. As can be appreciated, this type of test suite is very easy to generate, but like domain modeling is not very effective in finding bugs.

[0008] Research has shown that many bugs can be found by simply testing combinations of two parameters. This can significantly reduce the amount of resources needed to test, and thus pair-wise testing has been attempted in various ways. However, at present, such pair-wise testing, and other multiples (e.g., testing with triplets) is not a very developed technology, and is often done manually, which is time consuming and dependent on the skill of the tester. For example, not all pairs are valid, and the tester needs to recognize such constraints when testing.

[0009] In addition to testing valid combinations, referred to as “positive testing,” it is often desirable for the tester to test with invalid values, to make sure the program handles such errors properly. Such “negative testing” causes problems when testing with combinations, however, because of the way in which programs are written, namely to take some failure action upon the first error detected. More particularly, a problem known as input masking can occur with negative testing, in which one invalid input prevents another invalid input from being testing.

[0010] For example, negative testing of a function that can only handle positive values with a combination of parameter values such as {a=3, b=−1, c=−2} will not completely test these parameters a, b and c (tested as pair-wise combinations) due to input masking, because the first error condition detected, (e.g., −1) will end the testing when applied, before the other error condition (−2) can be tested. Given an efficiently generated set of test cases, there may be no other test case that tests for c=−2, and in such an event, any program bug that corresponds to a failure to recognize an error when parameter c=−2 will be not be found in such black box testing.

[0011] Known combinatorial test case generators do not properly handle negative testing. What is needed is a way to efficiently generate combinatorial test cases that supports negative testing in a manner that avoids input masking, while still properly handling positive test cases.

[0012] Briefly, the present invention provides a system and method for testing software with N-way combinations of parameter values that takes as input an original model containing valid and invalid values, and outputs a resultant test suite that tests all valid combinations, does not violate any user constraints, and does not have any negative test case with more than one invalid value. Thus, a full suite of positive and negative test cases are produced while avoiding the problem of input masking.

[0013] To this end, an original model comprising valid and invalid values (which are distinguishable from one another in some way), is input along with a number N to specify the order of the N-way combinations to test and any defined constraints, also may be input. The original model is modified in a first phase, by generating exclusions (constraints) for every pair of invalid values in the model such that not more than one invalid value is allowed per test case, while leaving intact any other exclusions. A first suite of test cases is then generated from the modified model, with any test cases that do not have an invalid value eliminated, creating a test suite with only negative tests.

[0014] In a second phase, (which may be performed in parallel with the first phase), the original model is modified into a second model by eliminating all invalid values from the original model. A second suite of test cases is then generated from the modified model. Because no invalid values were used, the output from the generator comprises a suite of test cases with only positive test cases that test all N-way combinations. Merging the first and second test suites provides a single, full suite of positive and negative test cases that tests all valid N-way combinations of valid values in positive test cases, does not violate any constraints, and has no negative test cases with more than one invalid value, thereby solving the problem of input masking.

[0015] Other advantages will become apparent from the following detailed description when taken in conjunction with the drawings, in which:

Exemplary Operating Environment

[0023] FIG. 1 illustrates an example of a suitable computing system environment 100 on which the invention may be implemented. The computing system environment 100 is only one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of the invention. Neither should the computing environment 100 be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary operating environment 100.

[0024] The invention is operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well known computing systems, environments, and/or configurations that may be suitable for use with the invention include, but are not limited to: personal computers, server computers, hand-held or laptop devices, tablet devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like.

[0025] The invention may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and so forth, which perform particular tasks or implement particular abstract data types. The invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in local and/or remote computer storage media including memory storage devices.

[0026] With reference to FIG. 1, an exemplary system for implementing the invention includes a general purpose computing device in the form of a computer 110. Components of the computer 110 may include, but are not limited to, a processing unit 120, a system memory 130, and a system bus 121 that couples various system components including the system memory to the processing unit 120. The system bus 121 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus also known as Mezzanine bus.

[0027] The computer 110 typically includes a variety of computer-readable media. Computer-readable media can be any available media that can be accessed by the computer 110 and includes both volatile and nonvolatile media, and removable and non-removable media. By way of example, and not limitation, computer-readable media may comprise computer storage media and communication media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can accessed by the computer 110. Communication media typically embodies computer-readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of the any of the above should also be included within the scope of computer-readable media.

[0028] The system memory 130 includes computer storage media in the form of volatile and/or nonvolatile memory such as read only memory (ROM) 131 and random access memory (RAM) 132. A basic input/output system 133 (BIOS), containing the basic routines that help to transfer information between elements within computer 110, such as during start-up, is typically stored in ROM 131. RAM 132 typically contains data and/or program modules that are immediately accessible to and/or presently being operated on by processing unit 120. By way of example, and not limitation, FIG. 1 illustrates operating system 134, application programs 135, other program modules 136 and program data 137.

[0029] The computer 110 may also include other removable/non-removable, volatile/nonvolatile computer storage media. By way of example only, FIG. 1 illustrates a hard disk drive 141 that reads from or writes to non-removable, nonvolatile magnetic media, a magnetic disk drive 151 that reads from or writes to a removable, nonvolatile magnetic disk 152, and an optical disk drive 155 that reads from or writes to a removable, nonvolatile optical disk 156 such as a CD ROM or other optical media. Other removable/non-removable, volatile/nonvolatile computer storage media that can be used in the exemplary operating environment include, but are not limited to, magnetic tape cassettes, flash memory cards, digital versatile disks, digital video tape, solid state RAM, solid state ROM, and the like. The hard disk drive 141 is typically connected to the system bus 121 through a non-removable memory interface such as interface 140, and magnetic disk drive 151 and optical disk drive 155 are typically connected to the system bus 121 by a removable memory interface, such as interface 150.

[0030] The drives and their associated computer storage media, discussed above and illustrated in FIG. 1, provide storage of computer-readable instructions, data structures, program modules and other data for the computer 110. In FIG. 1, for example, hard disk drive 141 is illustrated as storing operating system 144, application programs 145, other program modules 146 and program data 147. Note that these components can either be the same as or different from operating system 134, application programs 135, other program modules 136, and program data 137. Operating system 144, application programs 145, other program modules 146, and program data 147 are given different numbers herein to illustrate that, at a minimum, they are different copies. A user may enter commands and information into the computer 110 through input devices such as a tablet, or electronic digitizer, 164, a microphone 163, a keyboard 162 and pointing device 161, commonly referred to as mouse, trackball or touch pad. Other input devices not shown in FIG. 1 may include a joystick, game pad, satellite dish, scanner, or the like. These and other input devices are often connected to the processing unit 120 through a user input interface 160 that is coupled to the system bus, but may be connected by other interface and bus structures, such as a parallel port, game port or a universal serial bus (USB). A monitor 191 or other type of display device is also connected to the system bus 121 via an interface, such as a video interface 190. The monitor 191 may also be integrated with a touch-screen panel or the like. Note that the monitor and/or touch screen panel can be physically coupled to a housing in which the computing device 110 is incorporated, such as in a tablet-type personal computer. In addition, computers such as the computing device 110 may also include other peripheral output devices such as speakers 195 and printer 196, which may be connected through an output peripheral interface 194 or the like.

[0031] The computer 110 may operate in a networked environment using logical connections to one or more remote computers, such as a remote computer 180. The remote computer 180 may be a personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to the computer 110, although only a memory storage device 181 has been illustrated in FIG. 1. The logical connections depicted in FIG. 1 include a local area network (LAN) 171 and a wide area network (WAN) 173, but may also include other networks. Such networking environments are commonplace in offices, enterprise-wide computer networks, intranets and the Internet. For example, in the present invention, the computer system 110 may comprise source machine from which data is being migrated, and the remote computer 180 may comprise the destination machine. Note however that source and destination machines need not be connected by a network or any other means, but instead, data may be migrated via any media capable of being written by the source platform and read by the destination platform or platforms.

[0032] When used in a LAN networking environment, the computer 110 is connected to the LAN 171 through a network interface or adapter 170. When used in a WAN networking environment, the computer 110 typically includes a modem 172 or other means for establishing communications over the WAN 173, such as the Internet. The modem 172, which may be internal or external, may be connected to the system bus 121 via the user input interface 160 or other appropriate mechanism. In a networked environment, program modules depicted relative to the computer 110, or portions thereof, may be stored in the remote memory storage device. By way of example, and not limitation, FIG. 1 illustrates remote application programs 185 as residing on memory device 181. It will be appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers may be used.

Efficient Test Suite Generation

[0033] The present invention is, in part, directed towards a choosing an appropriate set of test cases from combinations of parameters and values, including positive and negative test cases, taking into consideration possible constraints on those values. Although the present invention will primarily be described with examples of pairs, it will be understood that the example mechanism described herein will work for any combinations of N parameters (N-way combinations) that can be covered, that is, N would equal 2 for pairs, 3 for triplets, and so on. The first part of the description herein is directed to explaining one way in which test suites can be efficiently generated, with the second part of the description being directed to explaining how the test suite can support positive and negative test cases while solving the problem of input masking.

[0034] The following is a simplified example that explains how a storage volume formatting program may be tested with three parameters, with desired values for those parameters. The test suite may cover all pairs of values among the parameters, regardless of the parameters to which those values belong. For simplicity herein consider that only the parameters set forth in the following table are to be tested:

[0035]

	Size:	10, 1000, 5000, 40000
	Format:	Quick, Slow
	FileSystem:	FAT, FAT32, NTFS

[0036] Note that because this example is so simple, it would be reasonably possible to exhaustively test all parameter combinations, (there would be twenty-four tests required to test exhaustively), however it can be readily appreciated that more complex domains quickly become impractical to exhaustively test, because exhaustive test suites grow exponentially while combinatorial (e.g., pair-wise) test suites only grow quadratically with the number of parameters and logarithmically with the number of values in a parameter. For example, with a disk formatting program, it would be possible to test many other sizes, other file systems, and other cluster sizes. Further, other parameters, such as compressed versus non-compressed formatting, may also be tested. Thus, in an actual storage volume formatting program, exhaustive testing would not be very efficient.

[0037] The above example table results in twenty-six unique pairs:

• [0038] {10, Quick}; {10, Slow}; {10, FAT}; {10, FAT32}; {10, NTFS}; {1000, Quick}; {1000, Slow}; {1000, FAT}; {1000, FAT32}; {1000, NTFS}; {5000, Quick}; {5000, Slow}; {5000, FAT}; {5000, FAT32}; {5000, NTFS}; {40000, Quick}; {40000, Slow}; {40000, FAT}; {40000, FAT32}; {40000, NTFS}; {Quick, FAT}; {Quick, FAT32}; {Quick, NTFS}; {Slow, FAT}; {Slow, FAT32}; and {Slow, NTFS}.
  Note that generation of these pairs is straightforward, even in complex cases, as each value for each type of parameter is simply paired once with each other parameter of a different type, and then that value is eliminated from further pairing. For N-way combinations where N>2, generation is also straightforward. For purposes of the present example, it can be assumed that all of these pairs are valid, (however as described below with respect to handling constraints, not all of these pairs actually are valid because some file systems have size limitations).

[0039] In keeping with the present invention, multiple combinations (pairs in this example) can be tested at the same time, in a single test, by efficiently selecting the combinations to test. For example, the table below shows that every pair can be tested in only twelve tests:

[0040]

	Test Case	Size	Format	FileSystem
	1	1000	Slow	FAT32
	2	40000	Quick	NTFS
	3	10	Slow	NTFS
	4	10	Quick	FAT32
	5	5000	Slow	FAT
	6	1000	Quick	FAT
	7	5000	Slow	NTFS
	8	40000	Slow	FAT
	9	5000	Quick	FAT32
	10	40000	Quick	FAT32
	11	10	Slow	FAT
	12	1000	Quick	NTFS

As can be readily appreciated, the reduced number of tests is possible because in this example, up to three different pairs can be tested in one test, that is, not only is “1000” and “Slow” tested in the first test case listed above, but “1000” and “FAT32” is also tested, as is “Slow” and “FAT32.” In other words, each single test case covers three combinations, e.g., test case number 11 {10, slow, FAT} actually covers three pairs: {10, slow}, {10, FAT} and {slow, FAT}. Note that some pairs (e.g., {10, Slow}) are tested more than once, however that is not typically significant, and is generally necessary so that all pairs can be tested at least once since some values for the other parameters need to be present. Further, note that even in this very simple example, there is a fifty percent reduction in the number of test cases, with only twelve test cases required, versus twenty-four test cases that would be required to test exhaustively. For larger models having more parameters, the savings is significantly larger.

[0041] FIG. 2 shows the general architecture for generating the test cases in this manner. The test cases, which together can be thought of as a test suite, are used for testing a domain under test 200, such as the software program for formatting a volume described above, or any appropriate domain. In general, a tester (e.g., a person or process) 202 produces a model 204 that may be maintained in one or more files and which reflects the characteristics of the domain-under-test 200. The model 204 generally includes a set of parameters and values for those parameters. For example, the tester may review the program and determine the parameters and values to test, and/or the program developer may provide a list of parameters and/or values to test along with the version of the program to be tested. The creation of the model is represented in FIG. 2 by the arrows labeled with circled numerals (1) and (2). In addition to the parameters and values for those parameters, the model may include a set of constraints (also referred to as exclusions) that identify what parameters and/or values should not be tested, and act as limitations applied to the model. For example, a FAT storage volume cannot exceed a certain size, and thus one such constraint would specify that FAT cannot validly be paired with a size above a certain limit. Note that the set of constraints may be empty. Constraints are further described below.

[0042] As further represented in FIG. 2, the model (file) 204 is provided (represented by the arrow labeled 3a) as an input to a test case generation mechanism 206, an example one of which may be based on an algorithm comprising one or more processes as described below with reference to FIGS. 4-7. As represented by the arrow labeled 3b, the test case generation mechanism 206 may also receive information (e.g., from the tester 202) as to what N-way combinations should be produced, ranging from two to the total number of parameters, that is, whether pairs (N=2), triplets (N=3) and so forth should be produced for combining into tests in the test suite.

[0043] As described below, with this input, the test case generation mechanism 206 outputs a set (or suite) of test cases 208₁ (e.g., to a medium such as a storage or transmission medium). This suite of test cases 208₁ generally has two properties, namely that all valid N-way combinations are covered at least once, and that no test case violates any of the given constraints. These properties also apply when negative testing with invalid values, as described below, however with negative testing, a third property is also present, namely that no test case has more than one invalid parameter value, thereby avoiding input masking. The arrow labeled four (4) represents these test cases 208₁ being generated, and the dashed arrow between instances of the test cases 208₁ and 208₂ representing the storage and/or transmission medium. Note that an error message is produced if any problem was encountered.

[0044] Thereafter, the instance of the test suite 208₂ is used by the tester 202 or another (e.g., manually or as an input to an automation procedure) to exercise the test cases from the suite against the domain under test 200. Note that the tester 202 may vary the tests with factors other than those in the model/test cases. For example, the tester 202 may run the test cases on a machine when memory is plentiful, and a second time when memory is scarce. Additional parameters may be varied.

[0045] Although the above approach greatly simplifies testing, in real-world scenarios, constraints need to be handled. By way of example, formatting a storage volume has an implicit limitation which says that the FAT file system cannot be applied on volumes larger than about 4 gigabytes (4000 megabytes). For that reason, any test case that covers combination of values cannot be run, (although of course a test could be separately run to make sure the formatting program properly prevents this from even being attempted). The test case generation takes this into account and for example might not allow the pair {4000, FAT} to appear in any test case, because doing so might prevent one or more other pairs from being properly tested.

[0046] Note that excluding a given invalid pair from the list of pairs to be covered, and then generating the tests cases, will not always satisfy the given constraints. For example, there are cases when a test is invalid because a certain combination of three or more parameter values makes an invalid combination, however in pair-wise testing, only pairs are known at the time before test case production starts.

[0047] Further, the problem cannot be overcome by removing all test cases which violate constraints after generation of the test suite is completed, because along with the invalid combination, one or more valid ones would be removed, which would mean that the “all valid combinations covered” requirement would be violated. For example, removing {40000, Slow and FAT} would not only remove the invalid pairing of {40000 and FAT}, but also would remove {40000 and Slow}, and {Slow and FAT}, which are valid pairs that may not be covered in another test case. As described below, in keeping with the present invention, proper test case generation assures that the properties of a comprehensive test suite (all valid combinations covered, but no invalid combinations covered) are satisfied, by providing a generation process that considers parameters and constraints throughout, as part of the overall generation process.

[0048] FIG. 3 is a representation of the general operation of the test cases generation mechanism 206. In general, the model file 204 (generally comprising the parameters and values to test and any constraints), and any additional parameters 312 (such as comprising the number N, such as N=2 for pairs) are fed to a model file reader and constraints parser component 314 of the test cases generation mechanism 206. The data is then initialized by a data initialization component 316. For example, the pairs are generated, and any invalid pairs are removed, as described below.

[0049] Components 318 and 320 represent a process within the test cases generation mechanism that generates test cases until all required combinations are covered in at least one test case. As represented by these components, the process repeats until all required combinations are covered, whereby output data in the form of test suite 208 is provided. Any invalid combinations (e.g., of three or more parameters when pairs are being tested, or four or more when triplets are being tested) can be handled in a manner that makes sure that no invalid combination is tested while a valid combination remains untested, generally by not allowing an parameter that is invalid to be used in a test case as that test case is being generated, as described below. Also, note that after generation, further processing may be done on the test suite 208, e.g., the tests in the suite may be reordered so that faster tests can be run first and the results analyzed while slower tests are running, or processed for some other reason.

[0050] FIG. 4 shows the processes performed by the test case generation mechanism 206 in more detail, beginning at step 400 where the model file 204, which contains parameters and their respective values, and optionally a set of constraints given as logical predicates, is read into the mechanism 206. As described above, the mechanism 206 also may be given (N) to specify the N-way combinations to be produced, although if not given, a default, such as N=2 for pair-wise generation, may be used.

[0051] FIG. 8 provides an example of a model file 800 comprising parameters, values for those parameters and constraints, and the set of pairs 802 that may be generated therefrom, given a pair-wise test. Note that the general data structures of FIG. 8 are only example for purposes of readability, and that the actual structures may take virtually any feasible form. Further, note that the examples in FIGS. 8-11 will be used in conjunction with the flow diagrams of FIGS. 4-7.

[0052] Step 402 represents the parsing of any constraints, as described below. For now, the result of parsing constraints may be considered as a syntax tree that represents the constraints in a ready-to-use form. Step 404 represents the creation of the needed data structures (e.g., data arrays), including a structure that holds the combinations to be covered, e.g., (for N=2, one structure may hold the pairs, while another may hold the test cases).

[0053] At step 406, based on the constraints, any combinations that are definitely invalid and cannot be covered are removed from the list of combinations. Note that this is only a first stage of applying constraints, and another stage of applying constraints will take place during actual test case generation. This first-stage removal is based on the observation that only those constraints which involve a number of different parameters equal to N or less may be applied, whereas other constraints need to remain unprocessed until later. By way of example, when working with pairs, given a constraint such as, if [A] equals “a1” then [B] cannot equal “b1,” step 406 can remove the {a1, b1} pair from the combinations to be covered, before any pairs are used to make the combinations, because the constraint applies to a pairing. However, such removal cannot be done with a constraint such as, if [A] equals “a1” AND [C] equals “c1” then [B] has to equal “b1” because such a constraint involves three parameters, and only pairs are available at this time (given this pair-wise example). Note that if instead N equaled three for triplet combinations, constraints based on combinations of two and three parameters may be applied to remove invalid combinations, but not those involving four or more parameters.

[0054] FIG. 9 provides an example of such first stage removal at step 406. In FIG. 9, the constraints in the model file 800 indicate that for this formatting program, the FAT file system cannot be used with compression, and that NTFS cannot be used with compression with a cluster size of less than 4K. Another way of stating this is that for these given values, {FAT, On}, {NTFS, 1K, ON} and {NTFS, 2K, ON} are invalid with respect to their usage in test cases. However, at this time only pairs are present, and thus only the invalid pair {FAT, On} can be removed, as represented by the crossed-out, curvy line in the modified structure 902. Note that removing this pair is not like eliminating a valid pairing once it has been used in a test case, because valid, used pairs may have to be re-used, as will become apparent below, whereas invalid pairs are never used.

[0055] As described below with reference to FIGS. 5-7, in general, following first stage removal, the main part of the test case generation process loops in an attempt to bundle as many still uncovered combinations as possible into a test case currently being built from parameter values. Any remaining constraints are also applied.

[0056] FIG. 5, step 500 essentially begins the process wherein an iteration counter is initialized, such as to zero or one. The iterations to build up test cases then begin, essentially by shuffling the parameters, e.g., randomly or pseudo-randomly, selecting one, and selecting a list of valid values for that selected parameter. These steps are represented in FIG. 5 via steps 502, 504 and 506.

[0057] It should be noted that to obtain the list at step 506, the possible values are compared with the test case being built with respect to the constraints, with any value which violates a constraint being removed from (or not added to) the list, thereby providing the second stage removal (actually prevention of using invalid values) against the actual test set values. For example, if the test set being built was {100, NTFS, On, ______}, the list of valid values would equal only 4K, and would not include 1K or 2K. Thus, in the present pair-wise example, not only are invalid pairs removed at step 406, but invalid combinations involving three or more parameters are not allowed based on the constraints.

[0058] The list of valid values also may be shuffled at step 512. Note, however, that there is no requirement that the parameters and/or values be shuffled randomly, however the parameters and/or values may need to be rearranged in some manner in the event that a particular parameter ordering prevents the test case from being generated. More particularly, if when building a test case, as a result of constraints there are no valid values for a selected parameter at step 508, the current iteration is discarded via step 510 and the parameters reshuffled so that a different test case can be assembled. Although not shown, a counter can be provided in this particular loop to exit in the event that a number of reshuffling attempts do not solve the problem.

[0059] Continuing with the example represented in FIGS. 8-11, at this time assume no test cases have been generated, and that the parameters have been shuffled into size, file system, compression and cluster size at step 502. If size is selected as the parameter at step 504, then the list of valid size values at step 506 is the set {10, 100}. As the test case being built is currently empty, these are both valid values at this time, and the process continues to step 600 of FIG. 6 to select a value from the list of valid values.

[0060] A number of ranked tests are used to select a best fitting value, with a general criteria for “best” being the value, if any, that when added to a test case will eliminate or most likely eliminate the most remaining combinations, thereby tending to minimize the number of required test cases. Note that although terms such as “best,” “minimize” and “maximize” are used herein, there is no intention to define such terms in an absolute sense, but rather such terms should be interpreted as (at least) approaching a desired goal, but not necessarily guaranteeing a minimum number of test cases. For example, it may be possible that given a certain set of parameters, values and constraints, an alternative process could figure out a lower number of test cases relative to the process described herein. However, the present mechanism has been found to work reliably and efficiently in practice, and provides significant gains needed to efficiently test software in real-world testing situations.

[0061] At step 600 of FIG. 6 a first test is performed to determine whether the test case being built (the partial test case) has less than N−1 elements therein. In the present example, N=2 and thus N−1=1, however the test case is empty, having zero elements. Thus, the test at step 600 is satisfied, and given the set {10, 100}, the value equal to 10 is chosen at step 610. This is added to the partial test case, e.g., {10, ______, ______, ______} as represented by step 612, which then returns to step 514 of FIG. 5. Because there is at least one other parameter to add to the test case, step 514 returns to step 504 to attempt find a value for the next parameter.

[0062] For purposes of the present example, consider the next selected parameter (step 504) as comprising the file system type, whereby step 506 will obtain the list of values {FAT, NTFS}. Note that these are both valid values with respect to the chosen size and the constraints, and since at least one exists, step 508 branches to step 512 to shuffle those values before continuing on to step 600 of FIG. 6 to select a value from the set.

[0063] This time, however, step 600 determines that because the test case has at least N−1 elements (has at least one element when pair-wise testing is active), a more intelligent selection mechanism is to be used. Step 602 represents a first way to determine an appropriate value, namely by looking for one that if chosen, would maximize the number of yet uncovered combinations which will be covered by the test case. In other words, if two values were available, for example, and choosing one would eliminate two pairs from the set 902 while choosing the other would eliminate only one pair, the one that eliminated the two pairs would be chosen at step 608. As can be readily appreciated, step 602 uses the knowledge of the past choices in an attempt to maximize the number of still-uncovered combinations, in keeping with a general goal of efficiently building test cases. Note that once a partial test case has been generated, the effect of choosing a certain value for a current parameter is determined.

[0064] In the present example, choosing FAT or NTFS would each only eliminate {10, FAT} or {10, NTFS} from the set of pairs 902, and thus there is a tie, whereby step 602 branches to step 604. Note that any value or values that did not tie for the maximum are eliminated from contention, e.g., if two values eliminated two pairs and that was the most of any, any values that only eliminated one pair would not be evaluated in further tests, generally because eliminating pairs is an important goal.

[0065] Step 604 further tests those values that tied at step 602, by evaluating whether any value appears in the largest number of still uncovered combinations, when compared to the other potential candidate value or values. In the present example, FAT appears uncovered in six valid pairings, while NTFS is appears uncovered in seven valid pairings. Note that {FAT, On} is not considered a valid pairing. Thus, NTFS is chosen at step 608, added at step 612, and the partial test case comprises {10, NTFS, ______, ______}. As can be readily appreciated, step 604 acts as a first tie-breaker, with a general goal of maximizing the opportunity for future values to produce a test case which will cover more values. Note that it is likely better to choose a value that appears most often on the “uncovered list” because doing so improves the chances for selecting values for the next parameters that will result in covering still untouched combinations.

[0066] Step 514 loops back for the process to handle the next parameter, which is compression in this example (selected at step 504), having two possible values, On or Off, at step 506. Because the constraints allow {10, NTFS, On, ______} and {10, NTFS, Off, ______}, both of these values are valid, and step 508 branches to step 512, where On and Off are shuffled.

[0067] As described above, because the test case has at least one element and pair-wise testing is being developed, step 600 will branch to step 602. Step 602 considers how many still uncovered pairs would be eliminated (covered) if Off is chosen, and how many if On is chosen. In the Off case, {10, Off} and {NTFS, Off} would be eliminated. In the On case, {10, On} and {NTFS, On} would be eliminated. Thus, there is a tie, and step 602 branches to step 604.

[0068] At step 604, the number of uncovered “Off” values are compared against the number of uncovered “On” values, and since Off appears in seven instances, while “On” appears in six instances, Off is chosen. The partial test case is thus {10, NTFS, Off, ______}. This state 1002 is shown in FIG. 10, with the partial test case shown via structure 1010. Note that the curved line through {FAT, On} indicates invalid and essentially removed.

[0069] As can be understood from the above description, the remaining parameter is size at step 504, with valid values comprising {1K, 2K, 4K}. Note that had compression been On, the constraints would have eliminated 1K and 2K from the list at step 506, however the shuffling and testing (FIG. 6) did not happen to work that way in the present example. As is further understood from the above description, the process continues to step 600 and on to step 602 with the values {1K, 2K, 4K}.

[0070] At step 602, the possible test cases comprise {10, NTFS, Off, 1K}, {10, NTFS, Off, 2K} and {10, NTFS, Off, 4K}. If any of these values was chosen, three currently uncovered pairs (those with 10 and the chosen size, NTFS and the chosen size and Off and the chosen size) would be eliminated, and thus there is a tie and step 602 branches to step 604. This time, the three values are equal at step 604, because each appears six times in the list of uncovered pairs 1102. Step 604 thus branches to step 606.

[0071] Step 606 looks at the covered pairs in the actual test cases (instead of the uncovered ones in the set to cover), and looks for the minimum number rather than the maximum number. At this time, no test cases are complete, and thus 1K, 2K and 4K are considered equal at step 606. Step 606 thus branches to step 610 where one of the values, e.g., 4K, is chosen (the values were randomly shuffled at step 512), and added to the test set at step 612. As can be readily appreciated, step 606 is another (second) tie-breaker, based on the knowledge that with either value from a thus far “tied” set, selection likely will do equally well in terms of certain and potential fitness. Step 606 helps spread values more equally among the test cases. For example, without having any weighing mechanism that would determine that for parameter A, a value a1 should appear twice as often as a value of a2 or a3, step 606 helps reduce the number of test cases by preventing large differences in the frequencies of occurrence for any value of any one parameter.

[0072] At this time, there are no parameters needed to complete the test case, and thus step 514 continues to FIG. 7. FIG. 7 essentially tests the progress of the process.

[0073] Step 700 represents testing for whether at least one new combination was covered by the most-recently built test case. If so, as in the present example, at step 702 the latest test case is added to the set that will be output when all combinations have been covered. Any combinations that were used in this test case are marked as eliminated from the list of pairs. In the present example, six pairs {10, NTFS}, {10, Off}, {10, 4K}, {NTFS, Off}, {NTFS, 4K} and {Off, 4K} are eliminated from the list of uncovered pairs, and a first test case 1110 {10, NTFS, Off, 4K} is complete, as represented in FIG. 11.

[0074] Step 706 of FIG. 7 represents a test for whether all of the valid combinations remaining on the list (e.g., the list 1102) have been covered by at least one test case. If so, the test case generation process will end, after outputting the test cases as a test suite (such as to a file), as represented by step 708. In the present example, there are other combinations, and thus the process returns to step 500 of FIG. 5 (entry point A) to build more test cases. Note that it not necessary to reset the iteration counter if the maximum number of allowed iterations is large enough to account for successful iterations.

[0075] Returning to step 700, if not at least one new combination was covered, the latest test case did not add any new pairs to be tested, and can be discarded at step 710. Note that only after the test case is built are the combinations (pairs) of that test case removed from the list (at step 704, described above). This is because if the test case is discarded at step 710, those combinations remain available during the next attempt.

[0076] The iteration counter is increased at step 712, and tested against a maximum (e.g., some fairly large number such as one-hundred) at step 714. If the maximum is reached, then there is a problem, such as an unusual combination that cannot seem to be covered given the current constraints. The process thus may end, giving the tester partial results and some indication (e.g., a list of the uncovered pairs) that an unusual situation has been encountered, perhaps allowing the tester to manually add the test case or cases needed to test any uncovered combinations, or try additional iterations. Otherwise another iteration is attempted by returning to FIG. 5, entry point B, (which does not re-initialize the iteration counter).

[0077] For brevity, because generation of the remaining test cases can be understood by simply following the operations of the test generation mechanism that have been already described herein, the present example will not be described to completion. Suffice it to state that following the rules of the operations, all valid combinations will be covered, and no constraints violated, although some of the pairs will be tested more than once.

[0078] The following is an example of a more complex model for formatting a volume, with a partial output shown therefor:

[0079] Example of an input file to the implementation:

[0080]

	TYPE,Primary,Logical,Single,Span,Stripe,Mirror,RAID-5
	SIZE,10,100,500,1000,5000,10000,40000
	FORMAT,quick,slow
	FSYSTEM,FAT,FAT32,NTFS
	CLUSTER,512,1024,2048,4096,8192,16384,32768,65536
	COMPRESSION,on,off
	if [FSYSTEM] = “FAT” then [SIZE] <= 4096;
	if [FSYSTEM] = “FAT32” then [SIZE] <= 32000;
	if [FSYSTEM] in {“FAT”, “FAT32”} or
	([FSYSTEM] = “NTFS” and
	[CLUSTER] > 4096) then [COMPRESSION] = “off”;

[0081] Example of the output the implementation produces:

[0082]

					COMPRES-
TYPE	SIZE	FORMAT	FSYSTEM	CLUSTER	SION
RAID-5	1000	slow	NTFS	4096	off
Single	10	quick	FAT32	1024	off
Logical	10	slow	FAT	32768	off
Span	5000	quick	NTFS	8192	off
Mirror	1000	slow	FAT32	16384	off
Primary	1000	quick	FAT	2048	off
Stripe	10	slow	NTFS	65536	off
RAID-5	10	quick	FAT	16384	off
Primary	5000	slow	FAT32	512	off
Mirror	1000	quick	FAT	65536	on
Stripe	5000	quick	FAT32	4096	off
Single	40000	slow	NTFS	2048	on
Span	1000	quick	FAT	512	off
. . .	. . .	. . .	. . .	. . .	. . .

[0083] As shown in the above examples, there is provided a constraint language, which in general defines constraints by a set of logical predicates. The general syntax of the predicates was chosen such that they are intuitive for developers and testers to understand and use, and so that any limitations of the domain can be specified, that is, the various possible dependencies between parameters can be expressed in this language.

[0084] In general, predicates are arranged as IF<condition> THEN <final_term> statements, stating what values are chosen (final_term) when a specific condition is met. A condition is generally a sentence comprising simple logical terms, joined by operators OR and AND. Parentheses are allowed to change the default priority of operations (which is AND before OR), and logical terms determine what values are allowed for specific parameter. The following table represents the predicate syntax:

[0085]

Constraints	::=
Constraint
| Constraint Constraints
Constraint	::=
IF Condition THEN Term;
| ParameterName Relation ParameterName;
Condition	::=
Clause
| Clause LogicalOperator Condition
Clause	::=
Term
| ( Condition )
| NOT Condition
Term	::=
ParameterName Relation Value
| ParameterName LIKE PatternString
| ParameterName IN { ValueSet }
ValueSet	::=
Value
| Value, ValueSet
LogicalOperator	::= AND | OR
Relation	::=
=
| <>
| >
| >=
| <
| <=
ParameterName	::= [String]
Value	::= “String” | Number
String	::= whatever is typically regarded as a string of
	characters
Number	::= whatever is typically regarded as a number
PatternString	::= string with embedded special characters
	(wildcards):
	* a series of characters of any length (can be zero)
	? any one characters
	# any one digit

[0086] The following is a predicate example:

• • [0087] 1. IF [p1]=1 AND [p2]=“a” THEN [p3]=“X”;
  • [0088] 2. IF ([p1]=1 OR [p1]=2) AND NOT [p2]=“a” THEN [p3]=“X”;
  • [0089] 3. IF NOT ([p1] IN {1, 2}) AND ([p2] LIKE “a*” OR [p2] in {“b”, “c”}) THEN [p3]=“X”;
  • [0090] 4. [p1]< >[p2];

[0091] One general operation of the constraints algorithm comprises syntax parsing, which takes as input the constraints as text, and outputs a list of syntax tokens. In other words, when parsing, a syntax parser goes through the text and recognizes tokens. Semantics parsing takes as input a list of tokens, and outputs a syntax tree. Type checking is performed, as well as prioritization of logical operators.

[0092] In one implementation, tokens may be ordered using Reverse Polish Notation to eliminate the parentheses and arrange the suitable operator prioritization. The syntax tree is the built made by walking through the tokens in the Reverse Polish Notation form.

[0093] Semantic checking for each term checks that values are of the same type, that is, the value types are the same as the parameter type. Note that there are auxiliary operators comprising NOT_LIKE and NOT_IN used internally, and are not used in user-defined predicates. To type check, LIKE is only for string type and only for comparison with value, while IN is only for value sets.

[0094] NOTs are removed from the syntax tree to facilitate the next step, combinations creation. To this end, the syntax tree is walked through, moving down and removing NOTs using de Morgan laws. In the following, in which A and B are branches of a syntax tree, the rules symbolically show for AND and OR (two branches as they have two operands) and NOT (one branch), how NOTs are moved to the bottom of the tree where they can be eliminated:

• • [0095] 1. NOT (A AND B)==NOT A OR NOT B.
  • [0096] 2. NOT (A OR B)==NOT A AND NOT B.

[0097] Additionally:

• • [0098] 3. NOT (A)==NOT A.

[0099] What remains are NOTs just before the term (e.g., NOT [p1]=1), which are removed by changing term's operation according to following table, where NOT A==B:

[0100]

	A	B
	parameter = value	parameter <> value
	parameter <> value	parameter = value
	parameter > value	parameter <= value
	parameter >= value	parameter < value
	parameter < value	parameter >= value
	parameter <= value	parameter > value
	parameter LIKE value	parameter NOT_LIKE value
	parameter IN {value_list}	parameter NOT_IN {value_list}

[0101] The next step, combinations creation, takes the syntax tree without NOTs, and outputs a list of combinations to be excluded. To this end, each term has a final set of matching values assigned. The syntax tree is walked from the bottom up, determining what values terms satisfy, and summing them to an output list (for OR operation) or performing a cross product on them with existing items on the list (for AND operation). The result is an output list based on the constraints set forth in the above examples.

[0102] To this end, both OR and AND have two operands (branches of a syntax tree), and because the tree is evaluated from the bottom up, before OR or AND are evaluated the results of both branches are already known. For example, if the left branch has N_L elements and the right one has N_R, the result of OR is a sum of those results, that is, they are simply put into one collection and returned. The result of an OR has N_L+N_R elements. The result of AND is a cross-product of results coming from both branches, in which each element of the left branch is combined with each element of the right branch (or vice versa). The resulting set has N_L×N_R elements.

Negative Testing

[0103] In accordance with an aspect of the present invention, there is provided a method and system in which at least one test case includes an invalid value such that the program under test can be evaluated as to whether it properly returns an error, while not violating the rules for testing valid combinations and not allowing input masking. In general, an invalid value is one which is outside of the valid range of a parameter, which the program should detect and handle as an error. As used herein, a positive test case is one that contains only valid values, and when tested with a positive test case, the program is expected to return meaningful results.

[0104] As previously mentioned, prior test case generators do not properly handle invalid values, because they are able to generate test cases having more than one invalid value. As a result, input masking can occur. By way of example, consider the following function to be tested:

[0105]

	float SumSquareRoots( float a, float b )
	{
	if ( a < 0 ) throw error;	// [1]
	if ( b < 0 ) throw error;	// [2]
	return ( sqrt( a ) + sqrt( b ));
	};

[0106] As is readily apparent, valid values for each parameter need to be numbers greater than or equal to zero, while invalid values are any numbers less than zero. A test case that provides any parameter with a number less than zero results in the function throwing an error, which is important to test.

[0107] However, when verifying whether the function handles invalid values properly, the following test case pairing cannot be the only one that covers the invalid values:
a=−1, b=−1

[0108] This is because with the parameters {−1, −1}, the test case will only get to check [1] and never to check [2] in the function, as a result of the error being thrown from parameter a's invalidity before parameter b can be applied, which is input masking. Indeed, the second line of code “if (b<0) throw error” was missing, there would be a bug in the code, however the test case pairing {−1, −1} would not find it due to the input masking.

[0109] Thus, in keeping with the present invention, to avoid input masking, a rule of negative testing that cannot be violated is that each test case can at most contain only one invalid value. At the same time, the other rules for valid test cases, namely all valid combinations covered, but no combination that violates a constraint is covered, cannot be violated. For example, a 2-way combination with a three-parameter test case such as {−1, 2, 3}, where −1 is an invalid value to test, may prevent the valid {2, 3} pairing from being tested, and thus this pairing must be present in another test case.

[0110] By way of example, consider a defined a set of parameters and their values:

[0111]

	P1: V11, ˜V12, ˜V13, . . . , V1N1
	P2: V21, ˜V22, ˜V23, . . . , V2N2
	P3: V31, ˜V32, ˜V33, . . . , V3N3
	. . .
	Pm: Vm1, ˜Vm2, ˜Vm3, . . . , VmNm

[0112] where the first subscript of each value identifies the parameter and the second subscript is used to differentiate the desired value to test from others, and wherein values starting with “˜” are invalid.

[0113] In general, combinatorial generators may directly use a list of combinations to be excluded, and thereby ensure that not more than one invalid value is in any combination. Thus, if a generator can use exclusions (a raw list of excluded combinations) directly, the list is used.

[0114] If a list is not available, the parameters and their values can be converted to a suitable means of limiting the generation domain. For example, existing tools may not have the ability to produce such a list, and thus producing predicates (or another high-level means of specifying explicit limitations) is performed when negative testing is performed using an existing test case generator.

[0115] By way of example, to fulfill the requirement of having only one invalid value in any test case, constraints are added to exclude each pair of those invalid values. To this end, the invalid values are put together in one set:

• • [0116] {˜V₁₂, ˜V₁₃, ˜V₂₂, ˜V₂₃, ˜V₃₂, ˜V₃₃, . . . , ˜V_m2, V_m3}

[0117] From this set, all possible pairs of the invalid values are created (the order is not important). Note that in this step, pairs of values are created, regardless of the value of N in the N-way combinations being actually generated:

[0118]

{˜V12, ˜V13}
{˜V12, ˜V22}, {˜V13, ˜V22}
{˜V12, ˜V23}, {˜V13, ˜V23}, {˜V22, ˜V23},
{˜V12, ˜V32}, {˜V13, ˜V32}, {˜V22, ˜V32}, {˜V23, ˜V32}
{˜V12, ˜V33}, {˜V13, ˜V33}, {˜V22, ˜V33}, {˜V23, ˜V33}, {˜V32,
˜V33}
. . .
{˜V12, ˜Vm2}, {˜V13, ˜Vm2}, {˜V22, ˜Vm2}, {˜V23, ˜Vm2}, {˜V32,
˜Vm2}, {˜V33, ˜Vm2}
{˜V12, ˜Vm3}, {˜V13, ˜Vm3}, {˜V22, ˜Vm3}, {˜V23, ˜Vm3}, {˜V32,
˜Vm3}, {˜V33, ˜Vm3}, {˜Vm2, ˜Vm3}

[0119] This set of pairs may be used to produce constraints, to provide the needed exclusions in the model. For efficiency, pairs that deal with values from the same parameter (having the same first subscript in the notation used herein, e.g., {˜V₁₂, ˜V₁₃}) can be safely removed, as by definition a parameter can only have one value in any given test case, resulting in the reduced set:

[0120]

{˜V12, ˜V22}, {˜V13, ˜V22},
{˜V12, ˜V23}, {˜V13, ˜V23}, ,
{˜V12, ˜V32}, {˜V13, ˜V32}, {˜V22, ˜V32}, {˜V23, ˜V32}
{˜V12, ˜V33}, {˜V13, ˜V33}, {˜V22, ˜V33}, {˜V23, ˜V33},
. . .
{˜V12, ˜Vm2}, {˜V13, ˜Vm2}, {˜V22, ˜Vm2}, {˜V23, ˜Vm2}, {˜V32,
˜Vm2}, {˜V33, ˜Vm2}
{˜V12, ˜Vm3}, {˜V13, ˜Vm3}, {˜V22, ˜Vm3}, {˜V23, ˜Vm3}, {˜V32,
˜Vm3}, {˜V33, ˜Vm3}

[0121] The constraints produced based on this set, may for example, be represented as in the following table of if—then statements:

[0122]

if [P1] in {˜V12, ˜V13} then [P2] <> ˜V22;
if [P1] in {˜V12, ˜V13} then [P2] <> ˜V23;
if [P1] in {˜V12, ˜V13} or [P2] in {˜V22, ˜V23} then [P3] <> ˜V32;
if [P1] in {˜V12, ˜V13} or [P2] in {˜V22, ˜V23} then [P3] <> ˜V33;
if [P1] in {˜V12, ˜V13} or [P2] in {˜V22, ˜V23} or [P3] in {˜V32, ˜V33}
then [Pm] <> ˜Vm2;
if [P1] in {˜V12, ˜V13} or [P2] in {˜V22, ˜V23} or [P3] in {˜V32, ˜V33}
then [Pm] <> ˜Vm3;

[0123] Applying these constraints during test case generation results in a suite of test cases that have no more than one invalid value per test case. However, when performing N-way (e.g., pair-wise) generation according to the above constraints, most of the test cases will have an invalid value in them, thus making mostly negative tests. This creates a problem with respect to high-quality positive testing, because an invalid value can result in valid pairs not being tested. In particular, during pair-wise generation only pairs of values are tested, not all possible combinations. For example, consider the following test case:

• • [0124] ˜V₁₂ V_2N2 V₃₁ . . . V_m1
    which covers at least following pairs of values:
  • [0125] {˜V₁₂, V_2N2} {˜V₁₂, V₃₁} {˜V₁₂, V_m1} {V_2N2, V₃₁} {V_2N2, V_ml} {V₃₁, V_ml}.

[0126] Note that each pair may be covered in only one test case, which poses a problem for positive pairs such as {V_2N2, V₃₁}, {V_2N2, V_m1} and {V₃₁, V_m1}, because the test code will likely never apply them in positive test cases, as the application under test will likely stop execution before they are tested, when the invalid values are applied.

[0127] Thus, in accordance with another aspect of the present invention, to solve this problem a multi-stage process (represented in FIG. 12) is performed by a mechanism 1200 that generates valid and invalid combinations that comply with the requirements for test cases described herein, namely that all valid combinations are tested, any constraints are applied such that no invalid parameter combinations are in the test cases, and no negative test case has more than one invalid value. In other words, the mechanism 1200 generates test cases that comply with the rules for both valid and invalid N-way combinatorial test case testing.

[0128] In general, and as also described below with reference to FIG. 13, the mechanism 1200 includes a splitting function 1202 that splits the test generation for a given model M into two separate phases. Then, via differing pre-processing functions 1204 and 1210, the mechanism 1200 converts each model M into differing models M1 and M2, generally one model arranged to produce invalid test cases, and another model arranged to produce valid test cases. More particularly, the pre-processing performed for the (mostly) invalid test cases adds the generated exclusions for invalid pairings to any other constraints, to create a the modified model M1 so that mostly negative test cases will generated from that model M1, as described above, while the pre-processing performed for the valid test cases removes invalid values creating the modified model M2 so that only positive test cases will be generated from that model M2.

[0129] Using an appropriate test cases generation mechanism such as the above-described test cases generation mechanism 206, (e.g., separate instances 1206 and 1212 thereof), and inputting the differing models M1 and M2, the mechanism 1200 generates differing suites of N-way combination test cases, S1 and S2 respectively, in the manner described above. Note that other suitable test case generation mechanisms may be alternatively used, as long as they support the ability to specify combinations that are to be excluded.

[0130] Following generation of the test suites S1 and S2, via a post-processing function 1208 the mechanism 1200 (optionally) post-processes the mostly negative test suite S1 into a further modified test suite S1*, essentially by removing any fully-positive (valid combinations only) test cases, since those test case are already present in the test suite S2 and would otherwise be unnecessarily duplicated test cases. A concatenator 1210 concatenates the results, and the result is a single suite of test cases S that meets the rules for testing valid and invalid parameters via N-way combinations. Note that it is essentially equivalent to simply test the program separately via the separate test suites S1* (or even S1) and S2, however concatenation provides the benefit of having a single suite of negative and positive test cases, which may be easier to manage, store, sort and so forth.

[0131] To summarize, the mechanism 1200 takes as input an original model M containing valid and invalid values, in which the valid and invalid values are distinguishable from one another in the model. Note that the model may include user-defined constraints which are input, and that the number N for N-way combinations also may be input. The original model is preserved, and in a first phase, for every pair of invalid values in the model, exclusions are generated and attached to a modified model M1, leaving intact any other exclusions (constraints). A suite of test cases S1 is then generated from the modified model M1, with any test cases that do not have an invalid value eliminated, creating a reduced test suite S1* having only negative test cases therein.

[0132] In a second phase, (which may be performed in parallel with the first phase), the original model M is modified into a different model M2 by eliminating all invalid values from the original model M. A suite of positive test cases S2 is then generated from the modified model M2. The output comprising a suite of negative and positive test cases S is the merged outputs of the two suites, S1* and S2.

[0133] FIG. 13 provides an explanation of the overall process in the context of an example model. Consider the following model arranged as {parameter: values}, in which invalid values are preceded by the character (˜), and wherein constraints may be present but are not shown for purposes of clarity:

[0134]

	A: a1, ˜a2, a3
	B: ˜b1, b2
	C: c1, ˜c2, ˜c3
	D: d1, ˜d2

[0135] At step 1300 of FIG. 13, this model is input into the negative and positive test suite generation mechanism 1200, possibly along with a value for N (equal to two in this example). Step 1302 splits the model into two stages, essentially by creating a copy of the model and providing each state with its own copy. Steps 1304, 1306 and 1308 represent the processes that provide the negative test cases, while steps 1310 and 1312 represent the processes that provide the positive test cases.

[0136] Step 1304 represents the generation of additional exclusions, which are set forth below for the current example model:

[0137]

	if [A] in {“˜a2”} then [B] in {“b2”};
	if [A] in {“˜a2”} or
	[B] in {“˜b1”} then [C] in {“c1”};
	if [A] in {“˜a2”} or
	[B] in {“˜b1”} or
	[C] in {“˜c2”, “˜c3”} then [D] in {“d1”};

[0138] When the model above is generated with these additional exclusions, as represented by step 1306, the following suite of test cases results:

[0139]

	A	B	C	D
	a3	b2	c1	d1
	a1	b2	˜c2	d1
	a3	˜b1	c1	d1
	a1	˜b1	c1	d1
	˜a2	b2	c1	d1
	a3	b2	˜c3	d1
	a3	b2	˜c2	d1
	a1	b2	˜c3	d1
	a3	b2	c1	˜d2

[0140] Note that the only positive test case is {a3, b2, c1, d1}, which is removed at step 1308, leaving the set of negative test cases:

[0141]

	A	B	C	D
	a1	b2	˜c2	d1
	a3	˜b1	c1	d1
	a1	˜b1	c1	d1
	˜a2	b2	c1	d1
	a3	b2	˜c3	d1
	a3	b2	˜c2	d1
	a1	b2	˜c3	d1
	a3	b2	c1	˜d2

[0142] As represented by step 1310, the positive test cases are generated by eliminating the invalid values, leaving the following modified model M2:

[0143]

	A: a1, a3
	B: b2
	C: c1
	D: d1

[0144] Step 1312 generates the suite of positive test cases from this (very simple) model:

[0145]

	A	B	C	D
	a3	b2	c1	d1
	a1	b2	c1	d1

[0146] Note that first test case in this model is identical to the one removed at step 1306, which is not necessarily true for every model. However, regardless of the actual test cases that are generated, all valid N-way combinations are present in the positive test suite following step 1312.

[0147] Step 1314 represents merging the two outputs produced at steps 1308 and 1312, resulting in a complete positive and negative test suite S at step 1316:

[0148]

	A	B	C	D
	a1	b2	˜c2	d1
	a3	˜b1	c1	d1
	a1	˜b1	c1	d1
	˜a2	b2	c1	d1
	a3	b2	˜c3	d1
	a3	b2	˜c2	d1
	a1	b2	˜c3	d1
	a3	b2	c1	˜d2
	a3	b2	c1	d1
	a1	b2	c1	d1

[0149] As is evident, the resultant test suite tests all valid combinations, does not violate any user constraints, and does not have any negative test case with more than one invalid value. Thus, a full suite of positive and negative test cases are produced while avoiding the problem of input masking.

[0150] As can be seen from the foregoing detailed description, there is provided a method and system for black-box testing of software using positive and negative test cases with N-way combinations of parameter values that solves the problem of input masking. The method and system are largely automatic, and provide test case generation of statistically valuable positive and negative test cases from a domain description (model). The suite of test cases is generated while taking into account constraint limitations of the domain, to produce a test suite having a relatively small number of test cases, and thereby significant productivity gains while eliminating problems due to input masking. The method and system thus provide significant advantages and benefits needed in contemporary computing.

[0151] While the invention is susceptible to various modifications and alternative constructions, certain illustrated embodiments thereof are shown in the drawings and have been described above in detail. It should be understood, however, that there is no intention to limit the invention to the specific forms disclosed, but on the contrary, the intention is to cover all modifications, alternative constructions, and equivalents falling within the spirit and scope of the invention.