System and Method for Software Immunization Based on Static and Dynamic Analysis

[0001] 1. Technical Field

[0002] The present disclosure relates to software analysis and more specifically to software code defect detection and correction.

[0003] 2. Introduction

[0004] As part of the development process of software, the software code is analyzed to find and repair defects. One such approach used by developers is static program analysis. Static program analysis uses many checkers to discover a very large number of programming issues. Static analysis, however, is known to identify many portions of code that might initially appear erroneous but in fact are defect free. These are commonly known as false-positives. As each potential defect requires significant manual effort to process, understand, and correct, software engineers often spend a lot of time searching for and analyzing false positives that are not actual problems while other errors and flaws can remain. Because of this false-positive issue, the static analysis approach is seldom put to its full usage by many software development teams.

[0005] Furthermore, if one defect is found and corrected, more of the same are likely to exist in the same codebase or in other code from the same author. The remaining defects, although possibly similar to those previously found and corrected will also need to be located and corrected using the same process. Thus, fixing the same problems reoccurring in different places in the code can unnecessarily require the same amount of effort as fixing it the first time, namely requiring some amount of human developer attention.

[0006] Accordingly, there is a perceived need for a software analysis tool that more appropriately focuses human efforts on potential defects identified in a static analysis process as being an actual defect. By prioritizing potential defects, a developer can spend more attention on identified defects that have a higher likelihood of being real (i.e., not a false-positive). Further, there is also a perceived need for a software analysis tool that can automatically apply fixes to other portions of the code identified as having a similar defect as one previously confirmed and corrected. Such a solution has the potential of reducing the time spent by a developer in repairing software code defects.

[0007] Additional features and advantages of the disclosure will be set forth in the description which follows, and in part will be obvious from the description, or can be learned by practice of the herein disclosed principles. The features and advantages of the disclosure can be realized and obtained by means of the instruments and combinations particularly pointed out in the appended claims. These and other features of the disclosure will become more fully apparent from the following description and appended claims, or can be learned by the practice of the principles set forth herein.

[0008] Disclosed are systems, methods, and non-transitory computer-readable storage media for identifying, prioritizing, and repairing potential code defects. Herein disclosed are embodiments for identifying potential software code defects, testing the potential defects, and ranking each defect based on the initial identification and subsequent test outcome.

[0009] Static analysis can identify potential code defects. The results generated by the static checkers can be processed through dynamic analysis testing to minimize the number of false-positives. Then the identified potential code defects can be ranked according to their likelihood of being an actual defect. Thus, a human software developer can first examine and correct the defects that have been determined to have the highest likelihood of being an actual defect.

[0010] Another object is to automatically correct or propose corrections to subsequently identified defects that are similar to one or more defects that had been previously confirmed and corrected. Such a feature has the potential of minimizing the time spent by a human developer analyzing and correcting similar defects that have a likelihood of occurring throughout the code again and again. This approach can be especially useful for correcting idiosyncratic repetitive programming flaws originating from specific developers, for example.

[0019] Various embodiments of the disclosure are discussed in detail below. While specific implementations are discussed, it should be understood that this is done for illustration purposes only. A person skilled in the relevant art will recognize that other components and configurations can be used without parting from the spirit and scope of the disclosure.

[0020] The present disclosure addresses the need in the art for a software analysis tool that helps a developer to effectively identify software code defects. A system, method and non-transitory computer-readable media are disclosed which identify potential software code defects with a minimal number of false-positives. A brief introductory description of a basic general purpose system or computing device in FIG. 1 which can be employed to practice the concepts is disclosed herein. A more detailed description of the software analysis tool will then follow. These variations shall be discussed herein as the various embodiments are set forth. The disclosure now turns to FIG. 1.

[0021] With reference to FIG. 1, an exemplary system 100 includes a general-purpose computing device 100, including a processing unit (CPU or processor) 120 and a system bus 110 that couples various system components including the system memory 130 such as read only memory (ROM) 140 and random access memory (RAM) 150 to the processor 120. The system 100 can include a cache of high-speed memory 122 connected directly with, in close proximity to, or integrated as part of the processor 120. The system 100 copies data from the memory 130 and/or the storage device 160 to the cache 122 for quick access by the processor 120. In this way, the cache 122 provides a performance boost that avoids processor 120 delays while waiting for data. These and other modules can control or be configured to control the processor 120 to perform various actions. Other system memory 130 can be available for use as well. The memory 130 can include multiple different types of memory with different performance characteristics. It can be appreciated that the disclosure can operate on a computing device 100 with more than one processor 120 or on a group or cluster of computing devices networked together to provide greater processing capability. The processor 120 can include any general purpose processor and a hardware module or software module, such as module 1 162, module 2 164, and module 3 166 stored in storage device 160, configured to control the processor 120 as well as a special-purpose processor where software instructions are incorporated into the actual processor design. The processor 120 can essentially be a completely self-contained computing system, containing multiple cores or processors, a bus, memory controller, cache, etc. A multi-core processor can be symmetric or asymmetric.

[0022] The system bus 110 can be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. A basic input/output (BIOS) stored in ROM 140 or the like, can provide the basic routine that helps to transfer information between elements within the computing device 100, such as during start-up. The computing device 100 further includes storage devices 160 such as a hard disk drive, a magnetic disk drive, an optical disk drive, tape drive or the like. The storage device 160 can include software modules 162, 164, 166 for controlling the processor 120. Other hardware or software modules are contemplated. The storage device 160 can be connected to the system bus 110 by a drive interface. The drives and the associated computer readable storage media provide nonvolatile storage of computer readable instructions, data structures, program modules and other data for the computing device 100. In one aspect, a hardware module that performs a particular function includes the software component stored in a non-transitory computer-readable medium in connection with the necessary hardware components, such as the processor 120, bus 110, display 170, and so forth, to carry out the function. The basic components are known to those of skill in the art and appropriate variations are contemplated depending on the type of device, such as whether the device 100 is a small, handheld computing device, a desktop computer, or a computer server.

[0023] Although the exemplary embodiment described herein employs the hard disk 160, it should be appreciated by those skilled in the art that other types of computer readable media which can store data that are accessible by a computer, such as magnetic cassettes, flash memory cards, digital versatile disks, cartridges, random access memories (RAMs) 150, read only memory (ROM) 140, a cable or wireless signal containing a bit stream and the like, can also be used in the exemplary operating environment. Non-transitory computer-readable storage media expressly exclude media such as energy, carrier signals, electromagnetic waves, and signals per se.

[0024] To enable user interaction with the computing device 100, an input device 190 represents any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, keyboard, mouse, motion input, speech and so forth. An output device 170 can also be one or more of a number of output mechanisms known to those of skill in the art. In some instances, multimodal systems enable a user to provide multiple types of input to communicate with the computing device 100. The communications interface 180 generally governs and manages the user input and system output. There is no restriction on operating on any particular hardware arrangement and therefore the basic features here can easily be substituted for improved hardware or firmware arrangements as they are developed.

[0025] For clarity of explanation, the illustrative system embodiment is presented as including individual functional blocks labeled as a “processor” or processor 120. The functions these blocks represent can be provided through the use of either shared or dedicated hardware, including, but not limited to, hardware capable of executing software and hardware, such as a processor 120, that is purpose-built to operate as an equivalent to software executing on a general purpose processor. For example the functions of one or more processors presented in FIG. 1 can be provided by a single shared processor or multiple processors. (Use of the term “processor” should not be construed to refer exclusively to hardware capable of executing software.) Illustrative embodiments can include microprocessor and/or digital signal processor (DSP) hardware, read-only memory (ROM) 140 for storing software performing the operations discussed below, and random access memory (RAM) 150 for storing results. Very large scale integration (VLSI) hardware embodiments, as well as custom VLSI circuitry in combination with a general purpose DSP circuit, can also be provided.

[0026] The logical operations of the various embodiments are implemented as: (1) a sequence of computer implemented steps, operations, or procedures running on a programmable circuit within a general use computer, (2) a sequence of computer implemented steps, operations, or procedures running on a specific-use programmable circuit; and/or (3) interconnected machine modules or program engines within the programmable circuits. The system 100 shown in FIG. 1 can practice all or part of the recited methods, can be a part of the recited systems, and/or can operate according to instructions in the recited non-transitory computer-readable storage media. Such logical operations can be implemented as modules configured to control the processor 120 to perform particular functions according to the programming of the module. For example, FIG. 1 illustrates three modules Mod1 162, Mod2 164 and Mod3 166 which are modules configured to control the processor 120. These modules can be stored on the storage device 160 and loaded into RAM 150 or memory 130 at runtime or can be stored as would be known in the art in other computer-readable memory locations.

[0027] Having disclosed some basic system components and concepts, the disclosure now turns to an exemplary method embodiment shown in FIG. 2. For the sake of clarity, the method is discussed in terms of an exemplary system 100 as shown in FIG. 1 configured to practice the method. The steps outlined herein are exemplary and can be implemented in any combination thereof, including combinations that exclude, add, or modify certain steps.

[0028] The present system and method is particularly useful for assisting in the identification and repair of one or more software code defects. One exemplary process in performing these tasks is shown in FIG. 2. In FIG. 2, the process 200 begins with the identification of a potential source code defect 210. The identification step 210 can be performed similar to that of a static analysis process. Static analysis is generally known in the art to analyze the code without actually executing a program that is built from the code. The static analysis process can apply a series of “checkers” to the source code and identify a wide range of potential errors. One checker example is a memory leak checker, which compares the number of times memory is assigned or allocated to the number of times memory is released to determine if all allocated memory has been properly reclaimed. Another example of a static checker is the checking of unit matching. The number of static checkers that exist is quite large and continues to grow. For example, Coverity and Klocwork each includes hundreds of checkers. Another checker example, described below for illustrative purposes, is a check of data flow consistency “def-used” pairs. Step 210 can run one or more of these checkers known in the field of practice but can also be updated from time to time with additional checkers as they are developed. The particular combination of checkers can be selected based on the codebase, authors, desired flaws to correct, available processing time, and so forth.

[0029] Static analysis detects potential issues in a program without pointing out exact failures that the potential issues can cause. Consequently, the static analysis process, although very good at finding a large quantity of actual source code defects, can also identify a large number of false-positives. Thus, the additional steps in process 200 can be used to separate out the actual defects from the false-positives and/or prioritize the detected source code defects so that a developer can focus on more urgent or serious flaws first.

[0030] If one or more potential defects are identified in step 210, they can optionally be compared to, or treated similarly as, other known defects in optional step 211, which will be discussed in more detail below.

[0031] If an identified potential defect was not recognized as known by optional step 211, it can be assigned an initial default potential value (“DPV”) in step 212. The assignment of the DPV can be performed using a heuristic-based approach, for example, categorizing the defects into “severe,” “major,” “minor,” and “correct” and assigning scores of 3, 2, 1, and 0, respectively. While integers are discussed herein, the defects can be categorized anywhere along a scale of 0 to 5, for example, with any integer or floating point value. One example of heuristics is the rareness of a defect. For example, as discussed in more detail in the process example below, a unique occurrence of assigning a value to itself can indicate a higher defect potential because it could most likely be an unintended developer error.

[0032] The assignment step 212 can also preferably include the generation of one or more data flow graphs (“DFG”) of the program under analysis (this step is not shown in FIG. 2). A program DFG represents all possible execution data flows, in which each node is a sequence of non-branching statements (basic block) and each directional edge represents the direction of the flow. The DFGs of the program segments in FIG. 3 are shown in FIGS. 4 and 5 and will be discussed in more detail below.

[0033] The testing step 213 can include generating and running tests of the sections of code that contain the identified potential defect to trigger failures. The testing performed can be similar to the testing performed by a dynamic analyzer. For example, the testing step can execute some or all of the code containing the identified potential defect on a real or virtual processor. When the code is executed, a number of test inputs can be used in an attempt to produce interesting behavior. In some preferred embodiments, a sufficient group of test inputs can be used to ensure that the tester has adequately observed a sufficient portion of the code's possible behaviors. For example, input values can include a smallest value, a random number mid-range value, and the largest value of the input range.

[0034] For each failure identified in testing step 213, the process 200 increases the DPV in step 214, thus indicating that the potential code defect is more likely to be real. For each pass in testing step 213, the process 200 decreases the DPV in step 215, thus indicating that the defect is less likely to be real. The potential code defect can then be ranked among the other found potential code defects by their respective DPV in step 216. Depending on whether the potential defect is above or below a predetermined threshold DPV value, step 217 determines whether the potential defect is sent to warning report 218 (below) or to error report 219 (above). The potential code defects in error report 219 are those with the highest DPV value and can therefore have the highest likelihood of being actual code defects.

[0035] A human developer, in step 220, can review the ranked potential defects reported in 219 and confirm the actual defects. Upon confirming a potential defect to be an actual defect, the human developer can then correct the defect, as indicated by step 221.

[0036] Optionally, the repair step 221 can also include gathering the information relating to confirmed defects and the subsequent corrections made and supply that information to the identification step 211. Having this information, step 211 can then compare subsequently-identified potential defects to confirmed defects. When step 211 finds a potential defect to be sufficiently similar to a confirmed defect, that defect can be deemed an actual defect and automatically repaired in step 221 with little or no human interaction. Consequently, a human developer can avoid having to review and repair similar errors occurring elsewhere in the code.

[0037] Having generally summarized some of the steps of process 200, the disclosure now turns to the detailed operation of process 200 by applying it to an example section of source code 300 shown in FIG. 3.

[0038] The source code 300 shown in FIG. 3 is a Java program. When process 200 is applied to source code 300, the two checkers in identification step 210 identify two potential source code defects: as self-assign, 310, and a possible-negative return, 320. Even though these errors seem obvious to most software developers, they are real defects and can be found in real industrial released code. Further, such errors are not so easy to detect when they are hidden in millions of lines of code and among hundreds and thousands of issues identified by the static analysis results. Therefore, this example is a good representation of a typical program segment on which developers can use this process to find potential defects.

[0039] The static checker in step 210 that found defect 310 uses traditional data flow analysis technique, “def-used” pairs, to discover defects. The term “def” means definition of a variable, either being initialized or assigned a value. The term “used” means a variable is used either in calculation (c-use) or in predicates (p-use). For each definition, e.g. “a=b”, the checker compares the variable to be defined (“a”) and the variable to be used (“b”) to make sure they are different variables; otherwise a “self-assigned” defect will be discovered. For example, highlighted statement 310, “this.id=id” has a “def” of “this.id” and “use” of “id.” Even though they are syntactically different, the two variables are in fact the same one since the local variable was likely erroneously typed in as “i” rather than “id.” The correct method header should have been “public setID(int id) {”. In this case, the “def-use” technique in step 210 identifies the self-assign error.

[0040] Another checker in step 210 searches for improper usage of functions from math library and identifies the potential defect identified in 320. It is more of a Java Math lab issue than the program issue. The checker discovers that the random number could be the value of Integer.MIN_VALUE, which can trigger the “Math.abs( )” function to return a negative number. First of all, the probability of the random number generator generating the minimal integer value is extremely small. Secondly, negative numbers might not be acceptable for most usage of “abs” function, but it is acceptable in this case because the impact of returning a value slightly smaller than “min” is negligible. Therefore, for most developers, this defect would be considered a false alarm or a trivial defect not worth fixing.

[0041] When analyzing this very small program segment 300, the checkers return a 50% false positive rate. As the number of lines of program code grows, the number of detected potential defects would also grow substantially. It is the remainder of the process 200 that filters out the defects having a high defect potential from these false-positives.

[0042] Assuming that the identified code defects are not recognized as “known” in step 211, process 200 generates a DFG of the program defects in step 212 and then runs heuristics to prioritize defects and optionally mark them on the optionally-created DFG.

[0043] A program DFG represents all possible execution data flows, in which each node is a sequence of non-branching statements (basic block) and each directional edge represents the direction of the flow. The DFGs of the program segments in FIG. 3 are shown in FIG. 4 and FIG. 5.

[0044] FIG. 4 and FIG. 5 are the respective DFGs of the methods 310 and 320 in the class of FIG. 3. They show basic blocks, BBx, of the program. In FIG. 4, BB1 (Basic Block 1, line 2 of the program) uses the variable “id” to assign a value to variable “this.id”. In this example, the self-assignment issue was considered to be a “severe” error and assigned DPV value of “3” by step 212. The assignment was based on the heuristics that the occasionally occurring errors are most likely to be real because intentional or unintentional coding habits usually occur multiple times.

[0045] In FIG. 5, the defect identified at (BB4) was assigned a DPV value of “2” by step 212. This assignment was based on another heuristic that the out-of-range values can cause major problems, but not severe ones.

[0046] Once the DPV value of each DFG node is assigned in step 212, the code sections are executed in step 213 with various inputs to generate test data. Three values for each test input variable are selected: two boundary conditions and one random value within the boundary. For the testing of the simple program segments in FIG. 3, the input values to method 310 are the smallest value, a random number, and the largest value of the input range. FIG. 6 shows the three test cases generated to reach node BB1 of FIG. 4.

[0047] In the case of method 310, all three test cases failed. Note that even though the test case code and the input data are generated automatically, the testing oracle, i.e. the “assert” statement in the test case can be filled in manually. In this case, the assertion should be “id.getID==input”. With the three failed tests and the original DPV of “3,” step 215 increases the value to “6” for BB1.

[0048] For the validation of defect 320, automatic test generator again gives three values to each input parameter. Since it only has two input parameters, “min” and “max,” it is a total of “9” combinatory for the two variables, each with three levels of values. Automatic execution of these nine test cases turns out to be positive, i.e. all nine tests passed. Thus, step 214 decreases the DPV initially assigned to BB4 from “2” to “−7.” Since “0” is to be the lowest DPV value for correct code, the DPV of BB4 is now “0” after the testing step.

[0049] The last task of the automatic tester component is to rank the DPV values, step 216, and use a threshold, step 217, to determine true failures. Assuming that the threshold is three, the BB1 is then reported to error report 219. No defects are found in BB4 and all other basic blocks.

[0050] The error report 219 is presented to a human developer for further confirmation 220 and/or repair of the defect 221. Once the code is fixed through on-line patching or off-line code changes, the process 200 can rerun the checker and the test cases to validate that the defect is truly removed. The confirmed defect, test, and repair information can be then sent back to the static analyzer at step 211 where it can stay with the code to prevent similar errors or errors of the same types from occurring again. By doing so, the static analyzer can “learn” which checkers have previously found confirmed defects and assign higher DPVs to subsequent code identified by those checkers. Subsequently identified defects that are sufficiently similar to the confirmed defects can also be automatically repaired. In this example, all subsequently identified “self-assign” type of errors should be corrected and avoided.

[0051] FIG. 7 illustrates an example architecture of an automatic testing tool suite. The static checkers, preventers, and automatic testers can be implemented as part of a single automatic tester in an exemplary tool suite. Many existing static analysis tools are available either commercially or in the open-source space. These static analysis tools are often complimentary, i.e. providing different and yet complimentary sets of checkers. Therefore, the example architecture creates an adaptor 702 to link any static analysis results into the tool suite automatic testing. Because the preventer includes a partial manual step of bug fixing, it does not require a separate tool implementation, but is embedded in some features of the tool suite.

[0052] One feature of the tool suite is an automatic tester including five components in a pipeline or other suitable architecture as shown in FIG. 7. The adaptor 702 generates static analysis results for potential defects. The potential defects are fed to the DPV-Calculator 704 with heuristics to generate basic blocks with DPV. The basic blocks are fed to a TestGen 706 to generate test cases given in the source language, which are fed to a DynamicAnalyzer 708. The DynamicAnalyzer 708 runs and records testing results. The testing results are fed to a ReportMaker 710 which generates a confirmed defect report with their checkers and test cases.

[0053] The adaptor 702 first converts the static analysis results into a standard potential-defect report format and sends the results to the DPV-Calculator 704, which processes the defects in the following four steps. First, the DPB-Calculator 704 generates DFG of the programs. Second, the DPB-Calculator 704 uses heuristics to calculate DPV value of each defect. Third, the DPB-Calculator 704 marks the defects on their corresponding basic blocks of DFG with their DPV values. Fourth, the DPB-Calculator 704 finally sends the list of all basic blocks with DPV values to the next component, TestGen 706. TestGen 706 generates test cases to reach the basic blocks with the highest DPV value. For some systems with a sufficient number of existing tests, TestGen 706 may not be necessary. Instead the code coverage information of the existing tests can be used to select those that can reach blocks with high DPV values. The DynamicAnalyzer 708 takes in the results from testing, uses them to calculate a new DPV value for each basic block. The DynamicAnalyzer 708 can also calculate the testing code coverage. Then the ReportMaker 710 generates a report of basic blocks with the DPV values passing a certain threshold along with their corresponding checkers and test cases, so that the preventer can readily use the results in the software build environment.

[0054] Embodiments within the scope of the present disclosure can also include tangible and/or non-transitory computer-readable storage media for carrying or having computer-executable instructions or data structures stored thereon. Such non-transitory computer-readable storage media can be any available media that can be accessed by a general purpose or special purpose computer, including the functional design of any special purpose processor as discussed above. By way of example, and not limitation, such non-transitory computer-readable media can include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to carry or store desired program code means in the form of computer-executable instructions, data structures, or processor chip design. When information is transferred or provided over a network or another communications connection (either hardwired, wireless, or combination thereof) to a computer, the computer properly views the connection as a computer-readable medium. Thus, any such connection is properly termed a computer-readable medium. Combinations of the above should also be included within the scope of the computer-readable media.

[0055] Computer-executable instructions include, for example, instructions and data which cause a general purpose computer, special purpose computer, or special purpose processing device to perform a certain function or group of functions. Computer-executable instructions also include program modules that are executed by computers in stand-alone or network environments. Generally, program modules include routines, programs, components, data structures, objects, and the functions inherent in the design of special-purpose processors, etc. that perform particular tasks or implement particular abstract data types. Computer-executable instructions, associated data structures, and program modules represent examples of the program code means for executing steps of the methods disclosed herein. The particular sequence of such executable instructions or associated data structures represents examples of corresponding acts for implementing the functions described in such steps.

[0056] Those of skill in the art will appreciate that other embodiments of the disclosure can be practiced in network computing environments with many types of computer system configurations, including personal computers, hand-held devices, multi-processor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, and the like. Embodiments can also be practiced in distributed computing environments where tasks are performed by local and remote processing devices that are linked (either by hardwired links, wireless links, or by a combination thereof) through a communications network. In a distributed computing environment, program modules can be located in both local and remote memory storage devices.

[0057] The various embodiments described above are provided by way of illustration only and should not be construed to limit the scope of the disclosure. Those skilled in the art will readily recognize various modifications and changes that can be made to the principles described herein without following the example embodiments and applications illustrated and described herein, and without departing from the spirit and scope of the disclosure.