Method for verifying the design of a microprocessor

[0002] This application is a continuation-in-part of U.S. patent application Ser. No. 09/578,743 filed May 25, 2000 entitled Coverage-Based Test Generation for Microprocessor Verification.

[0003] 1. Field of the Present Invention

[0004] The present invention generally relates to the field of integrated circuit functional verification and more particularly to a verification system and method that evaluate randomly generated tests based on the amount of functional coverage achieved relative to a given test specification prior to performing actual simulation.

[0005] 2. History of Related Art

[0006] As the complexity of microprocessors and other complex integrated circuits has consistently increased over the years, the process of verifying each new design has accounted for an increasingly large percentage of the total resources required to design and manufacture a particular integrated circuit. Indeed, the verification of complex microprocessors with multiprocessing capability is now estimated to consume more time, labor, and other resources than the actual design of the integrated circuit.

[0007] Typically, functional verification is accomplished by generating a large number of test cases (test programs) developed manually by designers and verification engineers as well as with the help of various random and specific test generators and running these test programs on a simulator that attempts to mimic the operation of the microprocessor or other device. As the number of transistors, functions, registers, and other facilities in the integrated circuit have increased, conventional verification methods have responded by simply increasing the number of tests that are simulated. Unfortunately, generating a seemingly infinite number of tests is an inefficient and unreliable method of verifying the functionality of all components in the processor.

[0008] Typically, random test generators will generate a large number of test cases that are redundant from the perspective of the number of microprocessor states that are tested. In other words, random test generators tend to generate a large number of test cases that exercise the same or similar classes or closely related classes of functionality of the microprocessor despite cosmetic differences in the test cases. In the majority of verification activities, neither the custom (deterministic) test cases nor the randomly generated ones adequately address the interdependencies and interactions between system level components and functions.

[0009] In the early days of microprocessor development, inefficiencies in functional verification were tolerated because the size of the test space (measured, for example, by the number of states the microprocessor may assume) was sufficiently small. In addition, early microprocessors typically had fewer functional units than modem microprocessors, and the interactions between the few components and functions were well understood and controlled. The increasing number of functional units in microprocessors is significant from a verification perspective because interaction between functional units can no longer be ignored or only loosely verified by conventional verification methodologies.

[0010] The general purpose use of modem integrated circuits makes it impossible to predict and plan for the type of software applications that will run on them and thus the state and interdependence that will be exercised in the field are rather large and generally non-deterministic. Roughly speaking, the test space of a microprocessor is approximately equal to 2ⁿwhere n represents the number of latches (state storage devices) within the microprocessor. From this approximation, it will be appreciated that the test space of microprocessors increases exponentially as the number of latches is increased.

[0011] The conventional approach to functional verification, in which increased complexity in a device is verified by simply increasing the number of tests that are simulated, is rapidly becoming unfeasible. The simulation process itself is resource intensive. In addition, because the input to a simulator in a conventional verification process is simply a large number of deterministic tests or randomly generated tests, the output of the simulation must be painstakingly evaluated to determine whether a particular simulation was successful in testing the intended functionality of the device.

[0012] It would, therefore, be highly desirable to implement a verification system in which the test specification produced by a test generator is evaluated in terms of the functional coverage value-added prior to simulation in an effort to reduce the number of tests that are actually simulated and to ensure that the tests that are simulated are likely to achieve additional verification of the device. Such a capability will result in more focused verification, shorter verification cycle and more efficient utilization of costly simulation cycles. In addition, it will provide a mechanism for more meaningful measurement and monitoring of the verification process, resulting in better scheduling and coordination with other design and manufacturing tasks.

[0013] As The problems identified are addressed by an integrated circuit functional verification method and system as disclosed herein. The method includes generating a test description comprising a set of test cases. The functional coverage achieved by the test description is then determined. The functional coverage achieved is then compared against previously achieved functional coverage and the test description is modified prior to simulation if the test description does not offer a delta coverage value add.

[0014] Rather than loosely coupled randomly generated test cases, this invention provides the architecture and method for coverage-directed pseudo random test generation for measurable targeted functional verification. In one embodiment, generating the test description comprises generating a test specification and providing the test specification to a test generator suitable for generating the test description. In one embodiment, the test description comprises a generic (i.e., generic format) test description and the generic test description is formatted according to a project test format specification if the coverage achieved by the test description satisfies the test specification.

[0015] In one embodiment, the functional coverage achieved by the test description is displayed in a graphical format. The test description is preferably added to a regression test database if the coverage achieved by the test description satisfies the test specification. Determining the functional coverage achieved by a test description may include estimating the coverage achieved based upon the test description, the test specification, a project specification comprising a high level model of the integrated circuit and coverage knowledge base.

[0016] The system according to the present invention preferably comprises a test generator, a coverage estimator, and a test coverage analyzer. The test generator is suitable for receiving a test specification and generating a test description responsive thereto. The coverage estimator is adapted to generate a test coverage metric indicative of the functional coverage achieved by the test description. The coverage estimator includes a coverage analyzer which performs high level simulation of the test description to generate a coverage signature.

[0017] In one embodiment, the system further includes a test specification optimizer configured to receive a test specification and adapted to produce an optimized test specification based on inputs from the coverage estimator and a project specification comprising a high level model of the integrated circuit. The optimized test specification is preferably adapted to produce a test description that achieves functional coverage of the integrated circuit that is equal to or greater than the functional coverage achieved by the original test description.

[0018] In one embodiment, the test generator generates a generic test description and the system includes a formatter adapted to modify the generic test description based on project specification information. In one embodiment, the coverage estimator utilizes a rule base indicative of a mapping between a test specification and a coverage model. The rule base is utilized to map the generic test description to a set of coverage states.

[0019] Other objects and advantages of the invention will become apparent upon reading the following detailed description and upon reference to the accompanying drawings in which:

[0020] FIG. 1 is a block diagram of a data processing system suitable for implementing the verification system of the present invention;

[0021] FIG. 2 is a block diagram emphasizing the major architectural components of a verification system according to one embodiment of the present invention;

[0022] FIG. 3 is a block diagram illustrating additional detail of a knowledge based coverage estimator of the verification system of FIG. 2;

[0023] FIGS. 4A-4B is a illustrative depiction of a display produced by a verification system of FIG. 2;

[0024] FIG. 5 is a block diagram of selected elements of an embodiment of the coverage analyzer of FIG. 2;

[0025] FIG. 6 is a flow diagram depicting a coverage analysis process according to one embodiment of the invention; and

[0026] FIG. 7 illustrates exemplary coverage matrices produced by the analyzer of FIG. 5.

[0027] While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail. It should be understood, however, that the drawings and detailed description presented herein are not intended to limit the invention to the particular embodiment disclosed, but on the contrary, the intention is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the present invention as defined by the appended claims.

[0028] The present invention contemplates various methods and processes for generating test cases (test programs) for functional verification of an integrated circuit such as a microprocessor. Portions of the invention may be implemented as a set of instructions recorded on a storage medium and suitable for execution by a computer or data processing system or a network of such systems. The storage medium may comprise a system memory of the data processing system, a hard disk, floppy disk, CD ROM, magnetic tape, or other appropriate storage device.

[0029] Turning now to FIG. 1, a block diagram of selected elements of a data processing system 100 suitable for use with the present invention is presented. The depicted embodiment of system 100 includes one or more processors 102a. . . 102n(generically or collectively referred to herein as processor(s) 102) coupled via a system bus 104. The processors 102 may comprise any of a variety of commercially distributed processors including, as examples, PowerPC® processors from IBM Corporation, x86 family processors from Intel Corporation, or 68000 family processors from Motorola.

[0030] A system memory 106, typically implemented as an array of dynamic RAM's, is accessible to processors 102 via system bus 104. A first bridge 108 of system 100 provides an interface between system bus 104 and a first peripheral or I/O bus 110. A wide variety of I/O devices may be coupled to first I/O bus 110 including hard disk controllers, audio adapters, and high speed network controllers for embodiments in which system 100 comprises one of multiple interconnected systems in a computer network.

[0031] First I/O bus 110 is preferably compliant with any of a variety of high performance industry standard I/O bus architectures including the PCI, MCA, AGP, or EISA bus architectures. In the implementation of system 100 shown in FIG. 1, a graphics adapter 112 and video controller 114 are coupled to first I/O bus 110. The depicted embodiment of FIG. 1, further includes a second bridge 118 that provides an interface between first I/O bus 110 and a second I/O bus 129 thereby providing a path between second I/O bus 120 and system bus 104. Second I/O bus 120 is preferably compliant with various industry standard bus architectures including the ISA and PCI bus architectures. In one configuration, first I/O bus 110 is a PCI bus while second bus 120 is an ISA bus.

[0032] In the depicted embodiment, a non-volatile memory device (NVM) 122 is coupled to second I/O bus 120. NVM 122 is preferably configured with a set of computer instructions executable by processors 102. NVM 122 is preferably implemented as a flash memory device desirable for its combination of non-volatility and programmability. In the preferred embodiment, the set of computer instructions contained in NVM 122 includes a boot code sequence suitable for transitioning computer system 100 from an idle state to a functional state following a system reset. The boot code sequence typically includes code suitable for loading the operating system software and may further include the system's basic input/output system (BIOS).

[0033] BIOS is utilized in conjunction with certain operating systems such as the Windows® operating system from Microsoft and the OS/2® operating system from IBM Corporation and includes low level microcode that controls the I/O device hardware such as the disk drives of system 100. Detailed BIOS information may be found in Croucher, Que's BIOS Companion(MacMillan 1998). Additional information regarding the OS/2 operating system is available in OS/2 Version2.1 Facts & Features(Order No. G326-0169-04) from IBM Corporation.

[0034] In alternative embodiments, system 100 may be implemented in conjunction with non-BIOS based operating systems such as JavaOS and other suitable network based operating systems. Regardless of software implementation, system 100 further includes conventional input devices such as a keyboard 130 and mouse or other suitable pointing device 128 coupled to host bus 104 (via I/O busses 110 and 120) through keyboard adapter 124 and mouse adapter 126 respectively.

[0035] Turning now to FIG. 2, a block diagram emphasizing the major architectural components of a system 200 suitable for generating test programs to verify the functional design of an integrated circuit is presented. Although, the present disclosure assumes a microprocessor as the integrated circuit that is to be verified, it will be appreciated that other integrated circuits such as an application specific integrated circuits (ASICs), digital signal processors (DSPs), analog-to-digital converters (A/Ds), and digital-to-analog (D/As) may be verified with the present invention as well.

[0036] Conceptually, system 200 contemplates a method of generating test cases that includes a coverage analysis process whereby a given test case is analyzed for the functional coverage that it provides prior to running the test case on a simulator. If a particular test case generated by a test generator does not provide adequate additional functional coverage of the microprocessor or other integrated circuit under consideration, the test case is not run on a simulator. In this manner, the present invention contemplates optimizing test cases prior to simulation or measuring actual architectural verification value-added prior to simulation in an effort to address the increasing cost and time required to verify microprocessors and other complex integrated circuits. If a test generated by a test generator does not provide additional functional coverage of the microprocessor being verified, system 200 refines or discards the test in favor of tests that provide additional coverage.

[0037] By focusing the test generation process on the coverage provided by the test, system 200 is distinguishable from verification by simulation methodologies (cycle-based and deterministic methodologies) in which it is presumed that the greater the number of random or deterministic test cases that are generated, the greater the functional coverage achieved (statistical coverage) or the greater the number of design specifications exercised, the greater the coverage (specification-based coverage), or the greater the number of resources exercised, the greater the coverage (program-based coverage). System 200 provides a coverage-based test generation by systematically directing and monitoring the generation and specification optimization process to cover all functional features of the architecture and micro-architecture.

[0038] As depicted in FIG. 2, system 200 includes a test specification generator 202 that provides input to a test specification optimizer 204. The test specification optimizer 204 converts a test specification received from test specification generator 202 to an optimized test specification that is forwarded to a pseudo random test generator 206. The test specification generator 202 receives inputs from traditional test generation sources such as, for example, inputs from a verification engineer, inputs from existing tests, inputs from a high level test specification, and inputs regarding the coverage requirement specification as well as a microarchitecture knowledge base 203 which provides all basic and known microarchitecture features to be covered (included) in test specifications.

[0039] Typically, test specifications are geared to a specific integrated circuit function, unit, set of instruction, bus protocol, memory coherency, control flow, data flow, etc. As an example, a test specification written by a verification engineer may specify that the engineer wants to verify that we can read/write very large numbers from/to every register in the microprocessor. This initial test specification description is output from test specification generator 202 and received by test specification optimizer 204, which optimizes the test specification description based, in part, upon project specification information (instruction formats, instruction names, register names and types, etc.) and test knowledge (range, value, and format of large numbers for the design) 205.

[0040] Test specification optimizer 204 is configured with test knowledge base 205 regarding test generator 206 such that the optimized test specification generated by test specification optimizer 204 is in a format suitable for use with a particular test generator 206. Including knowledge regarding test generator 206 into test specification optimizer 204 beneficially removes a significant burden on the verification engineer. More specifically, in the absence of test specification optimizer 204 in conjunction with test generator 206, the verification engineer must typically spend a significant amount of time and energy learning the details of a specific test generator. As soon as a well understood test generator is modified or replaced, all the information learned by the verification engineer concerning the previous test generator becomes obsolete. Delegating the task of optimizing test specifications for specific test generators to test specification optimizer 204 enables the verification engineer to focus on the task of defining test specifications that will provide new and useful information regarding the functionality of the microprocessor. The test generator 206 as described here can be classified as a pseudo random test generator which can be directed (navigated via biasings and control parameters) and influenced in how and what test programs are generated.

[0041] To illustrate the operation of test specification optimizer 204, imagine that test specification generator 202 generates, in response to inputs received from a verification engineer, existing test specifications, and coverage requirements, a test specification indicating that all registers in the microprocessor are to be tested for functionality with respect to floating point numbers. Based on project specific information indicated by reference numeral 208 in FIG. 2, test specification optimizer is configured to produce an optimized test specification to the test generator 206. In the example of a test specification to verify that all registers are functional with floating point numbers, the test specification optimizer 204 produces a test specification specific to a particular architecture, storage bus protocol, cache implementation, register set, instruction set, etc.

[0042] Thus, if the project specification 208 indicates that the integrated circuit being verified includes 32 registers, for example, the optimized test specification provided to test generator 206 may indicate that the user wants to verify the floating point number functionality of registers 0 through 31. (Throughout this disclosure, “user” refers to a verification or design engineer that utilizes system 200 to verify the functionality of a specific device. The various software components that comprise system 200 described herein including microarchitecture knowledge base 203, project specification 208, test knowledge base 205, coverage knowledge base 216, and test optimizer formatter 214 are typically developed by a tools group that supports the design verification).

[0043] In addition, the project specification 208 may indicate value ranges and types that a specific integrated circuit treats as a floating point number such that the optimized test specification provided to test generator 206 incorporates this additional information. If, for example, a particular integrated circuit treats numbers that are greater than 10¹⁰as very large numbers, the optimized test specification may indicate that the user wants to verify the functionality of register 0 through 31 with numbers greater than 10¹⁰.

[0044] The test generator 206 receives the output of test specification optimizer 204 and generates a test description. In the preferred embodiment, the test description generated by test generator 206 comprises a set of test cases (identified as generic test description 210 in FIG. 2).

[0045] In conventional verification systems, test generators typically generate test cases that are specific to a particular verification and simulation environment. Removing the environment specific information and test format from test generator 206 increases the portability of test generator 206. In other words, it is unnecessary to continually modify test generator 206 for each new verification and simulation environment. In addition, only one set of tools and functions need to be developed for processing and analyzing output of the generator, regardless of the specifics of the integrated circuit being verified. Furthermore, removing the specific test format requirements from the generation process, allows reuse of test specification optimizer 204 and coverage estimator 212 (described below) across various architectures, designs and design environments.

[0046] Test specification optimizer 204 also receives feedback from a coverage estimator 212. The coverage estimator 212, as described in greater detail below, analyzes test description 210 against an overall design coverage model and identifies various indicia of functional coverage with respect to the integrated circuit. Coverage estimator 212 provides feedback to test specification optimizer 204 regarding the potential coverage achievable by the test description 210.

[0047] Coverage knowledge base 216 maintains information regarding the functional coverage achieved by previously generated test descriptions and includes a set of rules that coverage estimator 212 applies against a test specification produced by test specification optimizer 204 as well as against a generic test description 210 generated from a specific test specification. The coverage knowledge base rules generate a set of actions and explanations that are used by test specification optimizer 204 to further optimize the test specification based on accumulated similar tests and knowledge gained from previous designs and previous tests generated for the current design. In addition, coverage knowledge base 216 includes all coverage criterion such as events to be covered, transitions to be observed, composite scenarios to be exercised, bus and cache activities that have to be generated by tests and micro-architecture features to be exercised.

[0048] Various inputs to coverage knowledge base 216 (such as the events, states, and scenarios depicted in FIG. 2) may be input by a verification engineer. Other inputs to knowledge base 216 may be automatically generated. The depicted embodiment illustrates the automated extraction of coverage attributes from a repository 219 of design information. Repository 219 may include a design description in a hardware description language (HDL) such as VHDL or Verilog® (a registered trademark of Cadence Design Systems, Inc.).

[0049] Thus, test specification optimizer 204 and coverage estimator 212 function as a model-based system where a dynamic model of a test specification is optimized based on a coverage information. Because there is no simulation result available to the coverage estimator, it functions based on a coverage model of the design. Inputs from coverage estimator 212 are converted to new test specification attribute values. These new values are then checked against design/architecture data base information in project specification 208 for validity and conformance.

[0050] Test specification optimizer 204 may generate confidence factors assigned to each test specification attribute. These confidence factors can trigger validation and optimization methods when their value falls below an acceptable threshold for that attribute. If a test specification or test specification attribute is modified as a result of inputs from coverage estimator 212, a “when-changed” method is automatically activated in one embodiment of the present invention. The when-changed method checks the new value or attribute against the test knowledge base 205 to assure that the specification is attainable by the test generator.

[0051] Test specification optimizer 204, as indicated previously also consults the project specification data base 208 to insure that specifications are consistent with architecture and design attributes. Attributes such as the instruction set, register set, operand sizes, operand registers, addressing modes, cache line sizes, cache configuration, level of sharing of cache lines between processors, and other suitable attributes are all taken into consideration to optimize the test specification provided to test generator 206.

[0052] Thus, it will be appreciated by those familiar with functional verification having the benefit of this disclosure that the method and system for generating generic test description 210 including feedback from coverage estimator 212 are utilized in system 200 to refine and focus test descriptions produced by test generator 206 prior to any actual simulation. Although test generator 206 may generate a large number of tests, it is possible that one or more of the generated tests actually provides no new coverage with respect to the test specification under consideration or does not provide the coverage outline in the test specification. Imagine, for example, a test specification designed to determine whether a register R1 can be read and written. Test generator 206 may generate first and second test specifications as follows:

[0053] Because the instructions in the second test specification vary from the instructions in the first test specification, one may assume that additional coverage is being provided by simulating the second test specification. Moreover, in a cycle-based verification scheme, one may assume that the second test case is actually more valuable because it will exercise more simulation cycles. In other words, because the complexity of the second test specification is greater than the complexity of the first test specification, conventional cycle-based verification methods might prefer the second test specification to the first test specification. From the perspective of exercising register R1 for read and write functionality, however, the second test specification does not exercise the stated objective even though it exercises functions not covered in test specification 1.

[0054] Utilizing the coverage estimator 212 in conjunction with test specification optimizer 204, system 200 will evaluate the second test specification and conclude that, with respect to the test specification under consideration, the second test specification does not provide additional coverage and is therefore unnecessary to simulate. By reducing and focusing the tests that are actually simulated, system 200 contemplates a significant reduction in simulation resources. Resources required to simulate tests generated with a conventional test generator have escalated out-of-control for state of the art microprocessors. The same applies for deterministic test generation methods, where there is no clear and measurable indication that enough tests have been generated or that all functional unit interactions have been covered.

[0055] The conventional approach to verifying ever increasingly complex microprocessors has been to generate an ever increasingly large number of test cases and to optimize the performance of the simulator. Unfortunately, the assumption that a greater number of test cases produces a greater assurance that the underlying microprocessor is functional is often incorrect. While a test generator may generate an essentially infinite number of test cases, many of these test cases may produce little or no new coverage of the design. In the case of deterministic test cases, it is not possible for the human test writer to consider all possible inter/intra actions among all the components of a given design. In addition, simulating the test cases and evaluating the simulation results is a resource intensive process that can overwhelm the designer's ability to develop an integrated circuit in a timely and cost effective manner. By reducing the number of test cases that are simulated and focusing those test cases to provide additional coverage, system 200 offers the prospect of gaining control over the verification process to reduce both the time required to adequately verify a given design but also to reduce the simulation and verification resources necessary, and increase the quality and effectiveness of the verification.

[0056] Another important benefit of the proposed system is the reusability of the various components of system 200. If there are any changes in a design architecture or its specific implementation, only the corresponding information in micro-architecture knowledge base 203, project specific information 208, coverage knowledge base 216, and design information repository 219 needs to be updated. All other major components of system 200 (i.e. test specification generator 202, test specification optimizer 204, test generator 206, coverage estimator 212, matrix 217, coverage analyzer 221, regression database 222, etc.) remain intact. Tests generated from then on will be consistent with the new design and there is no need for rewriting test specifications, which is a costly requirement in deterministic verification methodologies, as well as existing random test generation methodologies.

[0057] Turning now to FIG. 3, additional detail of coverage estimator 212 and its corresponding knowledge base 216 is illustrated. As depicted in FIG. 3, estimator 212 includes a coverage model 302 that is extracted from the design specification. As its name implies, coverage model 302 is a software representation of the instructions, state sequence, transitions, etc., that the device under consideration might encounter. Thus, coverage model 302 identifies the set of potential coverage states. Estimator 212 further includes a rule base 304 that defines the subset of potential coverage states that a given test specification will cover and a rule base 308 that enables estimator 212 to assign confidence factors to instructions, states, etc., covered by a test description. Coverage model 302, rule base 304, and rule base 308, are typically developed and maintained by a verification support group.

[0058] As depicted in FIG. 3, coverage estimator 212 receives a generic test description 210 generated by test generator 206 as its input. Coverage estimator 212 analyzes generic test description 210 against coverage model 302 and rule base 304 to map the test specification to potential coverage states as indicated in block 306 of FIG. 3. The potential coverage states generated by block 306 are then analyzed in conjunction with rule base 308 to determine any full states, partial states, transitions, transition sequences, collisions, bus transactions, cache operations, and so forth that are covered by generic test description 210.

[0059] From this information, coverage estimator 212 generates a test coverage model indicated in block 312 that includes a set of confidence factors. The test coverage model 312 is indicative of the states, transitions, events, protocols, and so forth that are likely to be exercised by test description 210. This information is routed to a test coverage value added analyzer 314 of estimator 212 that analyzes the potential test coverage for the test description 210 against test coverage information stored in coverage knowledge base 216. Coverage knowledge base 216 includes state information, transition information, protocol information corresponding to the coverage already achieved with previously generated tests. Test coverage value added analyzer 314 is configured to determine whether the test coverage model 312 corresponding to the current test description 210 provides any additional coverage to the coverage indicated in the coverage knowledge base 216. If the test coverage value added analyzer 314 determines that test description 210 achieves additional functional coverage of the integrated circuit, coverage knowledge base 216 is updated with the incremental coverage information corresponding to test description 210. If test coverage value added analyzer 314 determines that test description 210 does not provide any additional coverage over the coverage indicated in coverage knowledge base 216, test 210 is discarded and test specification optimizer 204 is invoked to refine the corresponding test specification and send to generator. Alternatively, the test description may be marked as duplicate, redundant, or similar to previous test descriptions and stored in regression database 222. In this manner, coverage estimator 212 is adapted to analyze generic test description 210 with respect to the test specification indicated by the verification engineer and optimized by test specification optimizer 204 and to determine what, if any, new states or transitions are potentially exercised by the test description 210.

[0060] In the traditional verification process, it is not known what specific states, transitions, etc., will be covered by the random test generator. In system 200, however, coverage estimator 212 serves as an expert to test specification optimizer 204 as well as the test generator 206. Imagine, for example, a test specification indicates that the verification engineer wants to verify the functionality of the register R1 with respect to very large numbers. A conventional random test generator typically includes a random number generator that provides the operand and a register value generator that determines the specific register in the test case. Frequently, the output of the random test generator is not known until after simulation. It is possible that the random test generator may generate an instruction that adds the number 25 to register R6 in response to a specification that indicates a desire to verify the functionality of register R1 with very large numbers. Since the test case is not consistent with the test specification, simulation of this test case does not test the desired conditions. By providing information to test specification optimizer 204 about what constitutes a large number and what registers are desired to be covered in the test specification, coverage estimator 212 is able to analyze test description 210 to determine if it is potentially capable of achieving the desired coverage for the test specification. Note that in current test generation environments, the user has to specify the range and scope of large numbers to be used for each test as well as specific registers to be tested. This makes the test useless if the design changes impact registers or data or address ranges.

[0061] Returning now to FIG. 2, if coverage estimator 212 determines that generic test description 210 provides additional coverage information, the coverage information from coverage estimator 212 corresponding to test description 210 is analyzed to determine if the generic test description 210 achieves adequate coverage of the test specification produced by test specification optimizer 204. If the test description 210 does not provide sufficient coverage of the test specification, test specification optimizer 204 performs additional processing until a suitable test specification is generated that results in a test description 210 providing the desired test coverage or a predetermined maximum number of trials is reached. When, ultimately, a test description 210 achieves sufficient coverage of the defined test, the test description 210 is forwarded to a project specific test optimizer and formatter 214.

[0062] The project specific optimizer and formatter 214 modifies the generic test description 210 according to product specification information 208 and project specific requirement and format. As an example, a generic ADD instruction comprising a portion of generic test description 210 will be formatted in project specific optimizer and formatter 214 according to the instruction set provided by project specification information 208. The output of project specific test optimizer and formatter 214 is a test description 220 that is suitable for input to a specific simulator. Preferably, the simulation results are provided to a coverage analyzer 221 that determines covered states, transitions, events, and scenarios that were covered when test description 220 was simulated. The coverage information determined by coverage analyzer 221 is fed back to coverage Knowledge base 216 where it can by used by coverage estimator 212 in optimizing subsequent test descriptions.

[0063] If the generic test description 210 achieves the desired test coverage with respect to the test specification, the generic test description and its corresponding test specification are added to the coverage regression knowledge base 216. By continually updating coverage knowledge base 216 with each test description 210 and its corresponding test specification, subsequent generic test descriptions can be compared against the most current test coverage information.

[0064] In one embodiment, the system 200 of FIG. 2 includes code to generate a graphical representation or matrix 217 of the test coverage likely to be achieved by a given generic test description 210 as well as the overall functional coverage achieved thus far. Turning now to FIGS. 4A and 4B, a representative illustration of one embodiment of matrix 217 is depicted as the coverage matrix 400. Coverage matrix 400 provides a graphical representation of the functional coverage achieved by a given test description 210 (or a group of test descriptions).

[0065] Matrix 400 depicts composite and simple coverage attributes that are likely to be achieved when a particular test description is simulated or a regression bucket is simulated. The system user can rapidly identify coverage attributes (i.e., state, transition, operation, and other interactions) that are likely to be exercised by inspecting coverage matrix 400. Coverage matrix 400 is preferably generated from and updated by the information stored in the coverage knowledge base 216 and coverage estimator 212.

[0066] The shaded areas 402a, 402b, 402c, etc., represent the confidence that a pair of coverage attributes will be exercised simultaneously. Shaded area 402a, located at the intersection of attribute 2 along the x axis and attribute 1 of the y axis, represents the confidence that attribute 1 (the y-attribute or dependent attribute) will occur while attribute 2 (the x-attribute or dominant attribute) is being exercised during simulation of a given test description or regression bucket. The notation (2&1) is used to identify the intersection of attribute 2 as the dominant attribute and attribute 1 as the dependent attribute.

[0067] Shaded area 402d(1&2), on the other hand, indicates the confidence that attribute 2 (the dependent attribute) will occur while attribute 1 (the dominant attribute) is being exercised. A graphical representation of the notation (A&B) is depicted in where the lower and longer vector represents the occurrence of the dominant attribute (attribute A) while the upper and shorter vector represents the occurrence of the dependent attribute (attribute B). Thus, the notation (A&B) signifies the occurrence of attribute B during the occurrence of attribute A.

[0068] The confidence level that a particular combination of attributes covered by a test specification is indicated by the percentage of the box corresponding to the combination of attributes that is shaded. A box that is mostly shaded indicates a relatively high confidence that the corresponding attribute combination will occur whereas a box that is less shaded indicates a relatively low confidence. In one embodiment, a user can click on a shaded area to see the exact numeric confidence factor as well as a list of tests that comprise that confidence.

[0069] In addition to shaded areas 402, the depicted embodiment of matrix 400 includes confidence arrows 404a, 404b, 404c, etc. (collectively or generically referred to a confidence arrow(s) 404) that indicate the confidence that a particular attribute is exercised relative to other attributes. In one embodiment, the positioning of a confidence arrow 404 within a box of matrix 400 indicates the relative timing between the attribute under consideration and one or more other attributes.

[0070] A confidence arrow such as confidence arrow 404apositioned in the center of its box (2&1), for example, may indicate the confidence that an attribute (attribute 3 in this case) is exercised while attribute 1 and attribute 2 are being exercised. Confidence arrow 404b, positioned at the left side of its corresponding box (4&4) may indicate the confidence that the attribute under consideration (attribute 3) is exercised prior to another attribute (attribute 4 in this case). Confidence arrow 404c, positioned at the right side of its box (5&4) may indicate the confidence that the attribute under consideration (attribute 3) is exercised after a pair of attributes (attributes 5 and 4) are exercised. Confidence arrow 404d, located at the boundary between two boxes (2&4) and (3&4) may indicate the confidence that an attribute (attribute 5) is exercised after a first attribute or first pair of attributes (attributes 2 and 4 in this case) but prior to a second attribute or second pair of attributes (attributes 3 and 4 in this case).

[0071] In addition, the depicted embodiment of coverage matrix 400 includes a set of y-attribute cumulative bar graphs 406a, 406b, 406c, etc., (y bar graph(s) 406) and a set of x-attribute cumulative bar graphs 408a, 408b, 408c, etc., (x bar graph(s) 408). Each y bar graphs 406 indicates the confidence that the corresponding y-attribute will occur as a dependent attribute (regardless of the dominant attribute) whereas each x bar graphs 408 indicates the confidence that the corresponding x-attribute will occur as a dominant attribute (regardless of the dependent attribute).

[0072] Y bar graph 406a, for example, represents the confidence that attribute 1 will occur as a dependent attribute (i.e., attribute 1 will occur during the time that a dominant attribute is occurring). X bar graph 408arepresents the confidence that attribute 1 will occur as a dominant attribute (i.e., that a dependent attribute will occur while attribute 1 is occurring). In this manner, each y bar graph 406 represents the normalized sum of the confidence levels in the corresponding row of matrix 400 (excluding the “diagonal” boxes, namely, (1&1), (2&2), (3&3), etc.) while each x bar graph 408 represents the normalized sum of the confidence levels in the corresponding column of matrix 400 (excluding the diagonal boxes).

[0073] Thus, y bar graph 406arepresents the normalized sum of the confidence factors indicated by the shading 402aand 402ewhile x bar graph 408arepresents the sum of the confidence factors 402b, 402d, and 402f. The symmetrical boxes such as box 402grepresent the confidence that the corresponding attribute (attribute 3 in this case) will be simulated irrespective of the other attributes. The graphical representation of confidence levels presented by coverage matrix 400 enables a system user to determine rapidly the likely scope of coverage for a given test description, as well as the overall coverage achieved.

[0074] Thus, test specification optimizer 204 receives a set of desired test specifications in the form of the original test specification and the achieved test specification. In response, test specification optimizer 204 generates a new set of test specifications such that the test coverage is increased or maintained while the difference between the desired and achieved test specification is decreased. One embodiment of the invention contemplates a delta specification matrix that is utilized by coverage estimator 212 to determine whether the coverage achieved by a given test specification is adequate. In one embodiment, the delta specification matrix includes a list of events and associated min/max specification differences that are acceptable. If the test coverage achieved by a given test description is within the tolerance specified in the delta specification matrix, the test description is then considered acceptable for the associated event.

[0075] The preceding portion of the disclosure is generally concerned with the process of estimating the potential functional verification valued-added of test cases prior to simulation or measuring actual architectural verification valued-added prior to simulation in an effort to avoid the time and cost associated with simulating test cases that add little or nothing to the functional verification of a particular design. This process may generally be identified as coverage driven generation (CDG). In CDG, estimates or actual measurements of the architectural and micro-architectural functional coverage achievable with a given test are used to adjust and redirect test generators to improve and optimize the functional coverage obtained by the generated test cases. In this sense, CDG deals primarily with test generation and is usually concerned with and supports a specific test generator.

[0076] The following portion of the disclosure is generally concerned with the analysis of the actual functional coverage valued-added achieved by those test cases that have passed the simulation process. In this portion, referred to herein as Coverage Driven Verification (CDV), a system and method for optimizing the overall verification processes is provided. CDG is concerned with issues such as how to tie a particular test generator (such as the GenesysPro or FPGen generators from IBM Corporation) to a particular coverage analysis tool (such as IBM's Meteor or Abacus tools). CDV expands on CDG by providing an architecture and methodology for monitoring, directing, and adjusting the various components of the verification process and not just a particular test generator. CDV thus encompasses not just a particular test generator, but all of the major components in the verification process including verification engineers, test plans, test specifications, test generators, test optimizers, and verification management frameworks.

[0077] Central to CDV is an algorithm for decomposing and analyzing the functional coverage results achieved by a particular simulation or series of simulations. Based on this analysis, the CDV engine disclosed herein is configured to compose verification directives based on coverage states determined to be missing by the CDV analysis. In one embodiment, the CDV algorithm uses a library or knowledge base of verification directives, which are basic verification tasks, to generate verification task descriptions. These task descriptions are then passed to verification engineers who are developing deterministic tests or to test generators such as GPro and FPGen. Portions of the invention may be implemented as computer executable instructions (software) stored on a computer readable medium. The medium may include the system memory of the computer on which the invention is installed, a floppy diskette, hard drive, CD ROM, DVD, magnetic tape, or other suitable medium.

[0078] Referring now to FIG. 5, a block diagram illustrating selected elements of an embodiment of coverage analyzer 221 is presented. In the depicted embodiment, coverage analyzer 221 includes a verification directive generator 526 that receives input from a coverage analysis and coverage data base interface (Coverage Interface) 524 and drives a verification and test generation interface (Verification Interface) 528. Coverage Interface 524, as described in greater detail below, provides a front-end interface between directive generator 526 and the functional coverage environment, which includes functional coverage data collection and tabulation tools, coverage models, and the functional coverage databases that they generate. Verification interface 528 provides a back-end interface between directive generator 526 and the functional verification environment, which includes test plans, test generators, test optimizers, test specifications, and verification engineers.

[0079] Functional simulation indicated by reference numeral 522 produces simulation results and traces for tests, simulations dumps (also known as VCD traces), hardware description language (HDL) monitor traces, as well as debug information for tests that fail the simulation. This information is provided to the functional coverage framework 523. The results of simulation 522 may include intermediate and final results. The intermediate results may include the value of resources and variables after each operation whereas the final results may include values for all registers, caches, memory, and system status flags at the completion of the test.

[0080] The output of simulation 522 is provided to functional coverage framework 523 for functional coverage data extraction and tabulation. This extraction and tabulation process is directed by a set of coverage models and coverage filters developed for and specific to each framework 523. The coverage models and filters parse corresponding simulation files for the conditions, states, scenarios, and properties in which the model is interested or that the model needs to filter or count. Some models may analyze the tests directly to gather information on content, structure, and the intermediate attributes of test components such as instructions, registers, data types, dependencies, and so forth. Direct analysis of test is also performed after simulation to determine whether the test passed simulation or failed it.

[0081] Coverage framework 523 typically generates detailed raw coverage data and various statistical and summary reports on the functional coverage achieved as well as on functional scenarios not yet observed (coverage holes). Coverage framework 523 can also provide various association lists indicating the coverage results produced by each test or monitor. Such association lists are generated from incremental coverage results. For each test or monitor, one can determine what new functional coverage was added and what components of the test or monitor were responsible.

[0082] Referring now to FIG. 6 in conjunction with FIG. 5, a flow diagram of a method 600 suitable for analyzing verification coverage of an integrated circuit design according to one embodiment of the present invention is presented. Initially, the output of verification framework 523, including coverage reports, coverage databases, coverage models, and coverage monitors (collectively referred to as coverage information), is analyzed (blocks 602, 604, 606, and 608) by a coverage interface 524. In one embodiment, the analysis of the coverage information includes classifying and decomposing the coverage information into a set of basic coverage tasks (BCT's) according to information contained in a coverage decomposition and classification knowledge base 525. Coverage decomposition and classification knowledge base 525 includes the information, rules, and heuristics that describe the verification environment under consideration (i.e., for a specific set of test generators being used, styles of test being developed, and generation biases being utilized) and how each coverage task can be replaced or represented by a set of equivalent but simple and generic coverage tasks. For example, a coverage task “Check_A” may be decomposed into the set of generic coverage tasks:

[0083] ADD Instruction is executed

[0084] Register FPSCR bit 10 eq. 1

[0085] No exception

[0086] The coverage tasks “ADD Instruction is executed” and “Register FPSCR bit 10 is equal to 1” and “There is no exception” are basic coverage tasks that define or comprise the Check_A coverage task.

[0087] Coverage decomposition and classification knowledge base 525 is generated by coverage engineers and verification engineers. The heuristics and rules are the expertise coverage engineers and verification engineers have gained with the current design and its predecessors. Coverage engineers also generate some of the rules by interviewing the architect and designers who can explain the sequence of micro-architectural actions which take place for each specific instruction to be executed, system condition to reach (e.g., overflow, system reset, rounding, etc.) or a system behavior to be simulated. Another part of the information in knowledge base 525 is extracted form the association lists generated by coverage framework 523.

[0088] The set of BCT's generated by coverage interface 524 includes covered BCT's and non-covered BCT's also referred to as coverage holes or, simply, holes. Coverage interface 524 provides the list of covered BCT's and BCT holes to verification directive generator 526, and to BVT-BCT matrix generator 530.

[0089] Verification directive generator 526 then extracts test cases corresponding to each BCT from a test generation environment 521 as well as the corresponding intermediate and final test results produced by simulation 522. Test generation environment 521 represents test plans, test generators, test optimizers, test specifications, and verification engineers described in greater detail above in the coverage driven generation portion of the disclosure. Test generation environment 521 produces deterministic tests, random tests, pseudo-random tests, and highly specialized and targeted tests. In addition, test generation environment 521 may produce HDL monitors such as Verilog and VHDL monitors, and Value Change Dump (VCD) monitors. Collectively, the output of test generation environment 521 comprises the set of tests and validation components that are generated and simulated for functional verification of an integrated circuit such as a microprocessor. These tests may be generated by design automation tools or by verification engineers, or by a combination of both.

[0090] Verification directive generator 526 then analyzes, classifies, and decomposes (blocks 610 and 612) the extracted tests and test results into a set of Basic Verification Tasks (BVT's) based upon a verification task decomposition and classification knowledge base 527.

[0091] Verification task decomposition and classification knowledge base 527 is comprised of a set of rules, methods, and heuristics for expanding a test program (test case) to a set of basic and generic operations (lowest denomination of operations supported and understood by the system being verified). Such rules and heuristics are usually specific to the test format utilized in the verification environment (test program language, syntax, and format), as well as the integrated circuit or system being verified.

[0092] A basic verification task (BVT) can be either a single basic operation such as “store data address” where data is any valid numerical data and address is a memory address within the range supported by the system, or a composite operation such as:

[0093] The rules, methods, and heuristics comprising this knowledge base are provided by verification engineers, coverage engineers, and architects. In addition, some of the methods can be automatically generated by comparing the high level description of the test specification to the corresponding test programs generated by test generators or by analyzing the test specification tables and matrices in the test plan against the specific test generated or developed to address the requirements of a specific section in the test plan.

[0094] The BVT set describes each test as a collection of dependent and independent basic processor operations and resources representing the architectural and micro-architectural attributes of the test.

[0095] Following the generation of BCT sets and BVT sets, each BVT set is analyzed (block 614) by BVT-BCT matrix generator 530 against its corresponding BCT set to determine any correlation between them. If, for example, a single BVT (or a set of BVT's or a logical combination of BVT's) always produces the same BCT (or the same set of BCT's or the same logical combinations of BCT's), then their exact associations can be described as one of the following:

[0096] if (BVT_n), then p(BCT_m)=100 (exact one-to-one association)

[0097] if (BVT_n), then p(BCT_m1|BCT_m2|BCT_m3)=100 (exact one-to-many association)

[0098] if (BVT_n), then p(BCT_m1& BCT_m2)=100 (exact one-to-many association)

[0099] if (BVT_n1|BVT_n2|BVT_n3), then p(BCT_m)=100 (exact many-to-one association)

[0100] if (BVT_n1&(BVT_n2|BVT_n3)), then p(BCT_m)=100 (exact many-to-one association)

[0101] if (BVT_n1|BVT_n3), then p(BCT_m1|BCT_m2BCT_m3)=100 (exact many-to-many)

[0102] if (BVT_b1&BVT_n2&BVT_n3), then p(BCT_m1&(BCT_m2|BCT_m3))=100 (exact many-to-many)

[0103] In addition to the cases in which there is an exact correlation (i.e., p=100) between BVT(s) and BCT(S), there may be cases in which the association does not occur all of time. This may be due to some change in the value of parameters that were not observed or due to some random attributes of the processor or the test generators. If there are sufficient samples for these cases, a probability of occurrence for each scenario may be generated. These probabilities represent a confidence factor that, given similar circumstances, a particular BVT or BVTs would produce the same BCT or BCTs. These probabilistic relationship may be expressed as follows:

[0104] if (BVT_n), then p(BCT_m)=p1(probabilistic one-to-one association)

[0105] if (BVT_n), then pBCT_m1|BCT_m2|BCT_m3)=p2(probabistic one-to-many association)

[0106] if (BVT_n), then p(BCT_m1&(BCT_m2|BCT_m3))=p2(probabilistic one-to-many association)

[0107] if (BVT_n1|BVT_n2|BVT_n3), then p(BCT_m)=p3(probabilistic many-to-one association)

[0108] if ((BVT_n1|BVT_n2)&BVT₃), then p(BCT_m)=p4(probabilistic many-to-one association)

[0109] if (BVT_n1|BVT_{n2), then p(BCTm1}|BCT_m2|BCT_m3)=p5(probabilistic many-to-many)

[0110] if (BVT_n1&BVT_n2), then p((BCT_m1|BCT_m2)&BCT_m3)=p6(probabilistic many-to-many)

[0111] The result of the exact and probabilistic BVT-BCT relationships and their corresponding values are then converted to associative matrices by BVT-BCT matrix generator 530. These matrices could be used for visual presentation of data as depicted in FIG. 7 or visual investigation of the relationships but the main purpose of these matrices is for mathematical manipulation and automatic search. In addition, these matrices can be represented by computer programs in the form of associative arrays and hash tables, where given one term or a set of terms, (e.g., BVT, or BVT1 & BVT3), a corresponding term and its associated probabilities can be searched for and identified. For example, given term BVT1, the search would return BCT1, 90% (from the one-to-one matrix) and BCT1|BCT2, 70% (from one-to-many matrix).

[0112] These matrices, as described in FIGS. 5 and 6 are used to compose new verification directives corresponding to coverage holes. Verification directive generator 526 does the reverse of what was described previously to decompose test programs to basic and generic tasks. Verification directive generator 526 generates (blocks 616, 618, and 620) new verification tasks by combining BVTs that are relevant to coverage tasks that have not been covered yet.

[0113] Verification directive generator 526 then compiles a list of all BCTs corresponding to all coverage holes identified by coverage framework 523 and coverage scenarios described in the functional coverage plan that have not yet been observed. This list of simple (e.g., BCT_{m1, BCTm2}, BCT_m3) and composite (e.g., (BCT_m1&BCT_m3), (BCT_m2&BCT_m4|BCT_m6) coverage tasks may then be analyzed against all BVT-BCT matrices to identify all exact, partial, and probabilistic BVT matches. These BVTs can then be classified into functional categories according to verification task decomposition and classification database 527. This functional categorization facilitates integration with test plans, which are typically classified according to functional units. In addition, test generators usually provide biasing controls and macros that are geared towards a specific functional objective. Therefore, a functional classification facilitates the mapping of BVTs to specific verification processes . Composite BVTs are also analyzed against verification task decomposition and classification database 527 to identify all simpler and different forms with which a particular BVT could be represented. This equivalency form generation may be useful in mapping complex verification scenarios to verification processes in the target environment.

[0114] Verification interface 528 then converts all outstanding verification directives to verification tasks, processes, goals, metrics, and monitors that are suitable to test generation environment 521 and attainable by the resources and components of the environment. The mapping of verification directives to specific verification tasks and scenarios is facilitated by information in a test generator and verification environment database 529.

[0115] Test generator and verification environment database 529 includes commands, variables, and their value ranges, test generator macros and their options, environment variables and their corresponding values, test program language syntax and macros and any other information that is necessary to compile test specifications for the test generators, initialize test generators, initialize the verification environment, provide verification engineers test specifications that they could use to generate or develop specific test or information that can be added to the test plan.

[0116] Database 529 is generated (populated) from a variety of sources including test generator user guides and technical documents, verification engineers expertise and know-how of a tool or specific test program language, automatic analysis of test cases generated by test generators, and by using knowledge acquisition tools and techniques to extract information from tool vendors, tool support personnel, verification engineers, coverage engineers, and verification support personnel familiar with verification and simulation environment.

[0117] The verification directives that are generated may then be provided to the appropriate verification processes, resources, tools, and components. The test generation, test simulation, coverage analysis and verification directive generation steps may be repeated until either all of the functional coverage goals are achieved or until no further improvement is detected.

[0118] It will be apparent to those skilled in the art having the benefit of this disclosure that the present invention contemplates a pseudo random directed test generation system suitable for analyzing test cases from a coverage perspective after simulation in an effort to improve the efficiency of the test generation and verification process. It is understood that the form of the invention shown and described in the detailed description and the drawings are to be taken merely as presently preferred examples. It is intended that the following claims be interpreted broadly to embrace all the variations of the preferred embodiments disclosed.